Almost exactly a year ago today a study was released in the journal PLOS "The Economics of Reproducibility in Preclinical Research" which states $28 billion is spent in the United States each year on biomedical research that can’t be reproduced by other researchers. This is one of the most provocative analyses that have been presented in recent years, among several other studies that have had similar conclusions. In response, the National Institutes of Health (NIH) issued new criteria for grant reviews aimed at bolstering the reproducibility of NIH-funded research.
Many stakeholders in healthcare are still expressing rightful concern regarding the integrity of published scientific research coming from the biomedical community. The question being asked is, can we trust published data, and can it serve as the foundation for new medical breakthroughs? A report released in 2012 stated as little as 11% of the most important published biomedical research was able to be reproduced by the industry's top scientists, which caused great alarm in the scientific community. However, it must be understood that a quagmire has been created by a pervasive business practice, which is at the heart of the reproducibility crisis and costs billions of dollars annually, while hurting all biomedical research unilaterally. It is the relabeling of scientific products which eliminates traceability for the future reproduction of scientific work, and is a business practice consistently used by most vendors. Even though this business practice is at the heart of the dilemma, it is rarely mentioned as a factor by many of the industry's key opinion leaders.
Real-Life Examples from the Lab
These next three examples demonstrate how frustrating and expensive it is to do research with this damaging business practice in effect. The first example comes from a casual conversation at the Experimental Biology Conference this past April. A researcher at a prestigious research institution wanted to test all available products to detect a bio-marker believed to be involved in tumor suppression. He was able to find 15 options online, and he had the funding to purchase them all at an average price of $300 each. When he compared the data, many of the results from products purchased from different vendors were identical. In fact, there were only 5 unique results from the entire set of products screened. This experienced researcher had been duped into retesting 2 out of 3 products, or $3,000 worth of unnecessary purchases, not to mention lost time and sample. The following 2 examples were presented in the Nature article,"Reproducibility crisis: Blame it on the antibodies".
A highly respected pathologist, Dr. David Rimm at Yale University, offers a more costly example of how market forces thwarted his research. Rimm had successfully developed a test for detecting the effectiveness of treatments for melanoma patients. His testrelied on antibodies that he acquired from commercial sources in order to detect biological markers in the patients' samples. After years of data collected and millions of dollars in funding invested, the Rimm lab ordered a fresh set of antibodies from the same vendors that were previously selected to scale-up the melanoma tests for commercial release, but the new batches produced inconsistent results from the original batches. Rimm was forced to give up his work on the melanoma diagnostic assay and effectively start from scratch after years of work. This also had a dire effect on a countless number of patients who suffered from not having this test available.
This final example regards Ioannis Prassas, a proteomics researcher at Mount Sinai Hospital in Toronto, Canada. He and his colleagues had been chasing a protein called CUZD1, which they thought could be used to test whether someone has pancreatic cancer. They bought a protein-detection kit and wasted two years, $500,000 and thousands of patient samples before they realized that the antibody in the kit was recognizing a different cancer protein, CA125, and did not bind to CUZD1 at all.
The last example appears to be an issue of nonspecific binding at first glance. Yet, because of the practice of relabeling, this same protein-detection kit is also being sold by other vendors under their brand names making it infeasible to remove this kit completely from the marketplace.
Today's researchers are becoming increasingly aware that the industry supplying them with the products they need cannot be trusted, because the products might not work as described by the vendors. What most people don't fully realize is there is an extremely active behind the scenes buying and selling of these products between vendors and manufacturers, with the vendors almost always relabeling the products under their own label. This means what researchers are buying from one company is often times manufactured by another, but researchers are blind to this arrangement. The reality is the marketplace has fewer manufacturers than vendors, vendors who are relabeling and reselling the same products under different brand names.
Tim Bernard, head of the biotechnology consultancy Pivotal Scientific in Upper Heyford, UK says that the 2 million antibodies on the market probably represent 250,000–500,000 unique ‘core’ antibodies.
If what Tim Bernard stated in the Nature article is true, then every antibody manufactured will be relabeled an average of 4 to 8 times. This means that the average antibody is available on at least 4 to 8 different vendors' catalogs, which appears to researchers as unique products offered by each vendor. In some cases, the same antibody can be offered in a dozen or more catalogs under different brand names. In the case of the first example presented above, the researcher buys all available products on the market to find out that only a few are actually unique.
This fact is hit upon in another Nature article, "Finding the right antibody for the job." If a graduate student tries and tries again to reproduce a result from a paper that applied an antibody-based assay, but the student keeps failing, he or she does not tend to doubt the paper's result, says Mike Browning, president of the antibody company PhosphoSolutions, the student “doubts himself or herself.” But consideration of the many variables that can lead to failed antibody assays can be a more productive approach to troubleshooting. “You've got to look at these things or else you're going to be chasing the same antibody that doesn't work through 12 different [vendors' catalogs].”
Perspectives from the Manufacturers
Yet, there is still another disturbing aspect to this relabeling racket, the back and forth trading of biological products between the vendors. The following is a real account I heard from an anonymous manufacturer: a researcher was using the manufacturer’s product in her project and needed to order more, but the manufacturer did not have enough left in stock. The manufacturer had to procure more product from a vendor in order to fulfill the request. Because of the prolific reselling occurring in the industry, it could have easily been the same product that was originally sold by this same manufacturer many years earlier. The products (which are temperature sensitive) might have traveled over many oceans, between many custom holds, and several countries before making their way to the researchers' hands, creating a new set of problems. This could expose the products to freeze-thaw cycles or heat that would adversely affect the quality of the products and the reproducibility of results. Vendors do this because this business practice has helped to create some of the industry's largest companies.
For example, Abcam produces its own antibodies and also has around 300 suppliers. “At Abcam we've got experience with both, making sure we're doing things properly ourselves and making sure our suppliers are doing things properly,” says Danielle Miller, who manages the company's operations. That is 300 suppliers and hundreds of thousands of products coming from all over the world into Abcam's facilities to be relabeled and resold under its brand. That means researchers do not know which one of the suppliers they are actually using in their experiments. This makes it nearly impossible for researchers to effectively screen through unique content available in the market in order to find the best choice for a particular experimental model. The goal of most companies thus far has been to grow the largest catalog under any pretense.
I learned during a meeting with the CEO of a reputable manufacturer, who must also remain anonymous, that his company had been making a significant amount of revenue selling antibodies to one of the largest vendors in the industry; antibodies to later be relabeled and sold to researchers at an ample markup. He knew his company's antibodies were good from the consistent restocking orders placed by this same vendor. One day he received a call from this large vendor and valuable customer, saying his prices needed to be lowered by 50% or they would take their business elsewhere. At 50% less, the manufacturer could not be profitable, so he had to decline. The vendor stopped buying from the manufacturer only because there was another supplier providing what the vendor considered "bio-equivalent" at a price point of 50% less. This switch had nothing to do with quality, but the bottom-line. The researchers who were using these products were not notified that there was a switch of suppliers nor were any researchers who published using these products.
Most vendors stress supplier relationship management as equally as customer relationship management, which in this industry, the suppliers are the manufacturers and the customers are the researchers. However, most researchers do not realize how much the vendors rely so heavily on a network of manufacturers to create their offering. The problem is that all this hyperactive backdoor dealing leaves the researchers in the lurch.
Swimming in a Sea of Subterfuge: The Shell Game
Several external efforts have been launched to deal with the reproducibility crisis, including Global Biological Standards Institute (GBSI), Antibody Registry, and the Research Resource Identifiers (RRID). GBSI approaches the problem by establishing a practice of including standards as part of every experiment that will insure proper and specific levels of detection, which can be accurately measured repeatedly and consistently between labs. Alternatively, the Antibody Registry and RRID are a combined effort to assign a unique identifier to every antibody and reagent used in published research. The hope is that this will allow researchers everywhere to procure the same reagents that are listed in scientific journals, in order to successfully reproduce the results.
The problem with GBSI, Antibody Registry, and RRID is that one pervasive business practice almost completely neutralizes their efforts. The profuse amount of relabeled content on the market means that there is little value of putting a unique identifier on a product that exists on dozens of other catalogs. Diminishing the value of these efforts even further is the abundance of supply-chain substitutions. Vendors switch out suppliers, but maintain the same catalog numbers (lot or batch identifiers are the only things changed). Certainly there is value in establishing a practice of standards in research, but with an inconsistent source of materials, adding standards to a protocol will not stop what happened to Dr. Rimm and a multitude of other investigators like him. This is like trying to put labels on the cups in a shell game. You think you will be able to identify where the ball is hidden, but in a game with 2 million shells frenetically travelling the globe, it hardly suffices. The efforts of GBSI, Antibody Register, and the RRID are only treating the symptoms of the disease, but not the cause.
The following excerpt from an article written in F1000 by Dr. Jan Voskuil, CSO of Everest Biotech, further illustrates the futility of trying to put unique identifiers on every product sold by every vendor or even hoping a widespread use of standards in research will curb the reproducibility crisis:
"The vast majority of vendors do not manufacture all the antibodies on the catalogue, and most of their antibodies have been obtained from a wide variety of different manufacturers from all over the world under OEM agreement (Other External Manufacturer). Such an agreement usually has a clause to forbid the supplier from publishing which of their products are sold by their OEM vendor. The vendors keep up the appearance that they themselves are the primary source of all their antibodies. This enables them to keep QC data on the product sheet that were generated many years ago thus keeping the sales going, while the actual antibody that generated these data may have sold out and has been replaced by successive other batches (from different animals) and the current batch on sale may no longer be able to generate such data at all.
Even monoclonal antibodies suffer from batch-to-batch variations, but not to such severe extent as some types of polyclonal antibodies. Nonetheless, certain hybridoma clone numbers are still being used for decades while, just like with cell lines, hybridomas cannot be the same after so many passages. It is misleading to use QC data that were generated decades ago, unless the current batch has proven to still be capable of generating such data (in which case one might as well show the latest version of the data).
Vendors accrue data from their own customers or from their own QC department, thus making the OEM product look unique. This way the same antibody can show different QC data on different catalogues. And while batches run out and are being replaced by others, it can happen that a vendor still has some of the old batch in stock, while another vendor will keep the QC data obtained from the former batch on their product sheet. From this moment on customers start to buy products that are no longer necessarily reflected by their product sheet.
Vendors do not only obtain their antibodies from the original manufacturers. There is a network of vendors obtaining each other’s catalogue items. Consequently, the same antibody starts to occur several times in one catalogue: one time with the current QC data provided by the original manufacturer and one or more times with QC data obtained from the other vendor’s direct customers or QC department. Potentially, assay developers buy several antibodies from several vendors thinking they are buying different antibodies, yet a number of them originate from the same manufacturer’s catalogue number."
The article from Voskuil makes it clear that manufacturers are under a "gag order" to not identify themselves as the makers of the products they sell to the vendors. Manufacturers have to watch in silence as their products are cited in scientific journals under the brand names of vendors they sell to routinely. While this might not be the worst aspect of relabeling, Voskuil presents a glaring problem that the data used in product sheets provided by the vendors is highly inaccurate, and in many cases, misleads researchers. There are many other deleterious issues Voskuil has ascribed to the practice of relabeling, such as lack of visibility to the cell lines used to produce the antibodies which can mutate over the passage of time, the inability to know if products are unique or identical from one vendor to another based on their data sheets, and the same antibody appearing with different catalog numbers on the same vendor's catalog.
This last quote from the Labome article "Antibody Quality" succinctly states the real frustration researchers face daily: "Antibody users are encumbered to communicate with specific antibody providers to discover and discuss their quality control practices. Adding to the confusion is the practice of some antibody suppliers/distributors to rebrand antibody catalog numbers and to re-assign antibody clone names. The antibody rebranding, in combination with the low quality of some antibodies, generates one of the most perverse aspects of antibody business. That is, when an antibody fails, the antibody user tends to purchase a second antibody from a different supplier, while not knowing the second antibody is a re-branded version of the failed one."
The practice of relabeling has caused, and is still causing, real despair in the research community. It is already difficult to produce groundbreaking research due to the complexity of biology itself. There is an innate challenge working with biological systems. Coupling that with an industry whose "dark side" is shuffling products from vendor to vendor, relabeling along the way, makes new scientific discoveries seem near impossible. Simultaneously, biomedical research is facing budget cuts that limit the work that can be done.
Article written by Micheal Simson (CEO of One World Lab)