- In genetic studies, one potential cause of spurious associations is differences between cases and controls in ethnicity, a situation termed population stratification.
- Was measurement of the genetic variants unbiased and accurate?
- Methods for determining DNA sequence variation are not perfect and may have some measurement error.
- Do the genotype proportions observe Hardy-Weinberg equilibrium?
- Have the investigators adjusted their inferences for multiple comparisons?
I have several others to add to this list, but HUGENet covers most of them. What is HUGENet? It is the Human Genome Epidemiology Network and it is a "global collaboration of individuals & organizations committed to the assessment of the impact of human genome variation on population health & how genetic information can be used to improve health & prevent disease."
In essence this voluntary set of collaborators evaluates epidemiologically, NOT CLINICALLY, but epidemiologically whether a GWAS or other Genome study is valid. PLOS reviews thet workings of HUGENet in a nice article.
This is an important network to have.
In addition, EGAPP (Evaluation of Genomic Applications in Practice and Prevention) evaluates the validity and applicability of these results if they are attempted to be turned into clinical practice. This too is a consortium of physicians and scientists evaluating such tools. Genetics in Medicine has a nice article about the methods of EGAPP too.
It is important to note that these are not "in house" services. Why do I say that? Well it is a little cloudy if a company such as Navigenics or deCode is telling you that their tests are clinically valid......Why? Well, they are selling the tests. Doesn't that make you stop and think?
How does Navigenics review studies for clinical applicability? They have posted on it. In essence they require at least 250 cases and controls and have a limited requirement for independent replications.....unlike HUGENet.
Thus the quandary with "in house" statistical analysis for scientific validity OR clinical utility.
The Sherpa Says: If you want to know the skinny on any of these studies, you need look no further than EGAPP or HUGENet....rather than trying to make sense of it through your 23andME account or tursting deCode or Navigenics to provide "unbiased" evaluations.....
3 comments:
Hi Steve,
Does your "physician friend" not know the difference between a genetic association study and a microarray analysis of gene expression? The abstract you link to discusses the latter, not the former.
Still, let's assume that's just a typo. In the bad old days of candidate gene association studies, your friend's estimate of 9/10 being wrong was probably not far from the truth; these were generally small studies with many uncorrected sources of bias and error. The same is not true for the majority of modern genome-wide association studies, which use large sample sizes, apply conservative corrections for genotyping error, population structure and multiple testing, and typically only report findings independently replicated in a separate cohort.
If your "physician friend" had followed the literature over the last two years, he'd be aware that the majority of genome-wide significant associations are indeed successfully replicated. If you are in fact using this 9/10 number to guide your review of genome-wide association studies you are well off the mark.
All three major personal genomics companies (and particularly 23andMe) employ reasonably transparent reporting of associations, allowing customers to make up their own minds about the strength of a specific finding; no offense, but they certainly seem to have a more accurate picture of the error rates in modern GWAS than you do.
Personal genomics customers should by all means check out HuGENet (and the primary literature) for independent validation and reassurance, but they should be aware that (1) the false positive rate in modern genome-wide association studies is very low, despite your exaggerated claims to the contrary, and (2) by and large, personal genomics providers report these associations accurately and fairly.
I have a few issues with your blog today, especially the article you refer to in support of your premise (Nature Genetics 2/09: "Repeatability of published microarray gene expression analyses.":
First, this article does not discuss 'association studies' but instead expression studies and the two are entirely different. Second, the primary conclusion is stated as "The main reason for failure to reproduce was data unavailability, and discrepancies were mostly due to incomplete data annotation or specification of data processing and analysis." So, even if this were about association studies (which it's not), the author isn't saying the findings of the studies are inaccurate, only that the way the data is presented makes them difficult to independently analyze.
Epidemiological evaluation is certainly worthwhile, but saying 100% epidemiological & 0% clinical is paramount to everything else is a extremely one-sided and in no way accurate (in my opinion).
What constitutes 'replication' hasn't been definitively defined yet and certainly one way is not 100% correct and another way is 100% wrong... I am sure there are good parts and bad parts to each person's methodology for categorizing something as replicated and validated.
@Daniel,
He is a physician at MSKCC. The abstract is a link that was provided to me from him. Maybe I misuderstood him. But I don't think so. GWAS do use large sample sizes and also are often not replicated......I will be more than happy to forward you all the non replication stuff on some of the MI markers. Any casual observer of this data will tell you that. Feel like asking Wylie Burke? She'll give you a similar stat on the failure to replicate..... IN SEPARATE PAPERS, not the same research group. Even if they are different cohorts. This can be due to chip used, sample selected etc. etc. etc. I think that if you are using the replication in the publication as the only proof you need of replicationto be valid, you my friend are fooling yourself.
I am curious, how is a nonscientist to make up their own mind about how good a study is? That's like asking an accountant "how his cath went?"
People just can't make up their mind effectively and ultimately rely on what they consider an "excellent" and often unbiased resource.....and HUGENet is just that....unlike companies who make profit from selling you tests by convincing you that they are useful...
-Steve
p.s. how about 9p21.3? Not replicated in the dutch, but in many other populations.....and That is one of the strongest things we have and it STILL has failure to replicate studies...
Post a Comment