Algorithms and Validation

A friend of mine asked me "If the framingham risk assessment fails to take family history into account, then why do we use it to guide anti-cholesterol therapy?" My answer was "It is scientifically validated."

In medicine we like to do things based on evidence. It is true that we do many things that do not have solid evidence behind them. But we always try to acquire data and then make a rational conclusion, leading to a treatment. When it comes to risk prognostication, validation studies are extremely helpful. And often keep us from getting sued.

So what did the Framingham do to be validated?

The Framingham Heart Study and the Framingham Offspring Study were the first epidemiological studies that prospectively collected population based data on the association between risk factors and the occurrence of fatal and non-fatal coronary and other cardiovascular events in a systematic and sustained fashion. It has been dissected for it's validity over the years.

So can we use the Framingham for everyone? Well, in Europe they tried. Several articles like this one show that the risk tool must be validated in the population you plan to use it for. The Framingham doesn't work so well on the Dutch. However when modified by the REGICOR, Spain's NHLBI, it seemed to perform well for the Spanish. The same with the Chinese modification.

You may now be asking what the heck does this have to do with Personalized Medicine. My answer....Everything. You see part of personalized medicine is prediction. That's why Helix Health of Connecticut trademarked "Prediction, Prevention, Privacy" These are the pillars of genomic medicine.

How can you predict the likelihood of Alzheimers in 5 years? Well, there are some corporate genomic companies doing it without having ever submitted articles for peer reviewed publication. They have "Trade Secret" algorithms that calculate risk. What the hell? How can you trust the accuracy of an algorithm without validation?

This is why we advise against using the Gail model for breast cancer risk. It can only work for certain populations, absolutely not for African Americans. There are new attempts at this type of risk stratification, and several attempts to defend the Gail model. But what has evolved is even more important. New algorithms...that were put to the test and peer reviewed.

Which brings me to my last point. What good is a tool if you don't know how to use it, or who it works for? A recent post on Wingedpig points this out. Confusion as to the tools. But what I wonder is what tools they used to create the tools. Would they publish their algorithm? Should they have to? Should other companies who will foray into medicine have to? As for SNPedia...a great resource, but the results are just like a wikipedia....not exactly peer reviewed.

The Sherpa Says:

The votes are in. I am surprised of the results. 23 and Me is the big winner. Why? Well, they specifically state that they do not intend their tools to be used for medicine. Yet all the posts I read have authors who mistakenly are using it as a medical tool. (See the genealogists post "when will they learn") I would have thought the readers would have chosen Navigenics. Navigenics WILL use their tool as a medical device! So I have to ask them. Where are your data on algorithms? Where did you publish and validate them? Which algorithms are you using? I guess we will find out soon enough. 2008 is right around the corner.

The SaluGen response

I wanted to highlight what goes on in my comments section daily. I know that these comments don't often show up on the blog and you frequently have to click them to investigate.

I recently posted on something I feel that is not ready for prime time medicine. This doesn't mean you can't spend a fortune on it to make you feel good. People can even buy and drink tons of alcohol to feel good. Not particularly useful for your health...But it can make you feel good.

From one of my respected readers,

Dear Steve -


This is just not a fair statement. I really enjoy your blog and have commented previously (see link - https://www.blogger.com/comment.g?blogID=6173393362223742012&postID=2267329423294071851).


But to suggest that Salugen is "snake oil company" and that SpaGen is some scheme is just untrue. I would be happy to provide you with greater detail on our Company and on one of our products, SpaGen.


So you know, Salugen is a personalized health and wellness company that leverages our thirty years of research in genetics published in over 350 scientific publications to deliver DNA-customized nutritional solutions based upon patent protected genetic tests.

Obviously a blog is not the best vehicle for disseminating comprehensive information, but let me share with you a few quick facts:

1) The SpaGen survey assessment which looks at health, diet, lifestyle, and Rx interactions/depletions related to nutrients has passed due diligence with the nation's largest PBM which is planning on using it for their own employee base.

2) The SpaGen DNA test is performed in a high-complexity, CLIA-certified laboratory which has been around for over 12 years. Hardly a fly-by-night snake oil company.

3) Our Chief Scientific Officer, Dr. Kenneth Blum, leads our research and development effort and studies to support what we do have been published in JAMA, Nature, Science, Lancet, etc. (hardly discredited rags)

4) Our nutritional supplements are manufactured in facilities that are either licensed by the FDA for pharmaceutical manufacturing or exceed those standards.

So, I would ask that before you rush to judgement about what we do, please take the time to be as thorough as you are in other areas of your blog. I would be happy to elucidate more on how we genotype and customize nutrients.

For example, based upon the MTHFR C677T mutation involved in the metabolism of folate, ultimately converting it into 5-methyl tetrahydrolate folate which is an enzyme that converts homocysteine into methionine, there is a published, randomized, double-blinded placebo-controlled study to demonstrate that by giving a tailored amount of folate (3x the recommended dietary intake) that you can reduce the risk for the nation's largest cause of death by 2.4 times. Not to shabby, eh?


Talk to you soon,
Brian Meshkin(from Salugen)




The Sherpa Said:
August 1, 2007 9:19 PM
Steve Murphy MD said...
Brian,

I ask that you forward me your researchers' abstracts and data on supplementation and disease prevention as well as their published data on genetic polymorphisms and their role in nutritional inadequacy. I would also love to see the journal citations of replication studies carried out by your team. I will happily review these and make an evidence based commentary after I receive these. I don't make rash statements unless I have done some due diligence. Nutrigenomics is in its infancy. We are not talking about the perfect being the enemy of the good. We still don't even know who the enemy is. As for the homcysteine, there was an article in the NEJM which actually showed increased risk with viatmin supplementation to lower homocysteine.

Take Care and Thank You

-Steve

p.s.

The jury is still out on Homocysteine lowering and reduction of cardiovascular risks. Even Nutritional Journals acknowledge that.

WikiPedia Meets Genetics

I just received an email from one of my readers today. Trip said " am med student at Univ of KY, interested in medical genetics and have been reading your blog.........I am recommending http://www.snpedia.com/ for a blog post on the gene sherpa" Well Trip....You Asked for it, You got it..... As they say on that old Toyota commercial....

First I would like to mention that my friend Bertalan over at ScienceRoll commented on this Yesterday. He did an excellent job. Also SNPedia has their own blog although there are only 2 posts so far.....

So Single Nucleotide Polymorphisms (SNPs) are little genetic changes, much like letters in a word. There is some data out there which shows taht when readign a senetnce letters in the middle of a word do not alter the readers undertsanding. This could be the case for a SNP, it may result in no change in the protein function. Or it could be the case that a SNP may change the word altogether.

Even crazier is when a SNP isn't even in the coding region of a protein. This may affect a protein as well. When we make mRNA there is a process called splicing. This splicing could be altered by a SNP located in an intron (noncoding region of a gene) or it could be located in an another gene and affect the protein by epistasis........

Listen, this is all confusing. Much like SNPs are...... It reminds me of other "genetic markers" like HLA haplotypes. No one knows what role HLA B27 has in ankylosing spondylitis....it is just linked to an increased likelihood of having the disease.

So what about SNPedia. This is a catchy idea. There exist several databases out there including OMIM. However, the more databases, the better. If we can cross reference these for validity it certainly would be nice.

In reviewing SNPedia I performed searches on several SNPs including in TCF7L2 and CCR5. The database has listed some but not all of the associations within each of these "genes" in fact CCR5 is not only an HIV associated gene, it is also implicated in abdominal aneurysms.

The Sherpa Says:

Any database is only as good as the data in the base. I feel that opening it up to public contribution through wiki is a great idea. However, we must assure the public that SNPedia will be monitored by a knowledgeable set of curators.