Current medical knowledge is to a large extent based on results from scientific studies. Traditionally, these results are published in peer-reviewed medical journals.
Before being accepted, a scientific paper has to go through critical assessment by expert reviewers who will assess the paper’s suitability for publication. The peer review process is intended to guarantee standards of quality and provide credibility. The highest ranked medical journals only accept a small fraction of papers submitted to them for publication.
Clinical medicine relies on the scientific literature. For a clinical cardiologist like myself, this is a key issue. The procedures I decide to perform, and the therapy I recommend is, and should be, based on scientific evidence. For the clinician, evidence based medicine is the holy grail.
But what if scientific studies are flawed? What if evidence based medicine relies on erroneous data? Then, obviously, clinical medicine is broken.
Unfortunately, the scientific community is not free from dishonesty and greed. Scientific fraud is hard to deal with.
Faked data exists and is often difficult to expose. However, we should be able to rely on high quality medical journals when it comes to wrong use of statistics, erroneous calculations and wrong conclusions. These journals should guarantee that papers plagued with such problems are not accepted for publication. But, are they up to the task?
The Statins and the Elderly Saga
One of the most important questions facing clinical cardiology today is when to use statin drugs for individuals who have not been diagnosed with cardiovascular disease (CVD). Clinical trials have shown that these drugs lower mortality and reduce the risk of future cardiovascular events among people with CVD. However, in those without established CVD, the magnitude of effect is less clear end it is uncertain when the benefits of therapy outweigh the risks.
Age itself is independently associated with the risk of CVD, and risk factors such as blood pressure, lipid disorders and diabetes are common among the elderly. However, limited clinical research is available addressing statin treatment among healthy people above 65 years old.
Four months ago I read with interest a paper by Gianlugi Savarese and colleagues, published in the Journal of the American College of Cardiology (JACC) presenting a meta-analysis of the benefits of statins in elderly subjects without established CVD. The authors concluded that “their meta-analysis provided the first time evidence that the benefits of statins on major cardiovascular events extend to people above 65 years old”.
In the paper, the authors came to the conclusion that statins significantly reduce the incidence of myocardial infarction (MI) and stroke, but do not significantly prolong survival in the short term.
Their numbers show that 83 patients have to be treated with statins to prevent one case of MI and 142 patients have to be treated with statins to prevent one stroke, for a mean follow-up of 3.5 years. However, they did not present these numbers in their paper. Instead they claimed that 24 patients needed to be treated for 1 year to prevent one MI and that 42 patients needed to be treated for 1 year to prevent one stroke.
I guess anybody with some statistical knowledge will see that the Number Needed to Treat (NNT) for one year should be a higher number than the NNT for 3.5 years. If one is performing a clinical trial in order to test an effect of a drug, a higher number of patents is needed if the study is planned to run for 1 year than if it is supposed to run for 3.5 years. There will be fewer events in 1 year than in 3.5 years, therefore the NNT for 1 year is a higher number than the NNT for 3.5 years. If you’re still doubtful, read my earlier blog post on the issue.
My Letter to the Editor of JACC
After discovering the error in the paper by Savarese and colleagues I wrote a letter to the editor of JACC which was recently published. I pointed out that the authors appeared to have made an erroneous calculation when reporting the NNT for a period of one year. By using data from their paper I had calculated that the NNT for one year to prevent one MI and one stroke was approximately ten times higher than reported in the paper, given that NNT is constant over time. In other words, the statin effect was exaggerated by a factor of ten.
I also suggested that the most appropriate approach would have been to report the NNT for the mean follow-up of 3.5 years, instead of calculating the NNT for one year.
A correction was published by the authors in JACC on March 25. It’s not very substantial:
The authors report the number needed to treat (NNT) for the entire mean follow-up of studies was 83 and 142 for myocardial infarction and stroke, respectively. The authors apologize for this error.
Strangely, not a word about the erroneous calculation. The wrong NNT numbers per year are left uncorrected.
Two of the authors of the paper, Gianlugi Savarese and Pasquale Perrone-Filardi, responded to my letter. Their response was published in JACC together with my letter. They agreed that it was more appropriate to report the NNT for the mean follow-up of 3.5 years than presenting the NNT for one year. Furthermore they write:
As previous authors did (citation) using the same formula adopted in our meta-analysis, our aim was to calculate the NNT per year dividing the overall NNT calculated for the entire trial duration by the length of the follow up. We agree with dr Sigurdsson that this may represent an oversimplification, since this calculation assumes that the effect of the treatment (relative risk reduction) is constant over time and that events occur at a constant rate over time.
Oversimplification is not the right word. Simply put, this is a completely wrong approach. But to my surprise, Savarese and colleagues don’t seem to realize or understand it. In fact dividing, when you should multiply will provide numbers that are very far from the truth.
Repeating An Error Won’t Make it Right
Of course I was curious to see the paper cited by Savarese and Perrone-Filardi in their response to my letter. It turns out that it’s a paper published in Circulation 2008; “Lipid Management to Reduce Cardiovascular Risk: A New Strategy is Required“, written by H. Roberto Superko and Spencer King III. Dr. King is a world-famous senior cardiologist, a pioneer in cardiac catheterization and coronary angiography.
These two renowned cardiologists address the NNT from a number of statins trials in primary and secondary prevention. Interestingly, they also calculate the NNT per year. The results are published in Table 2. The table shows that the NNT per year is always a higher number than the NNT for the whole study period (which is always longer than one year). For example the NNT to prevent one MI in the famous 4S (SSSS) trial was 11.7 for the whole study period, but the NNT per year of the study was 63.2. The NNT for the WOSCOP trial was 44.2 for the whole study period, but 216.6 per year of the study. In fact, this all looks very reasonable and correct.
But the strange thing is that in the paper’s text, Superko and King use a different approach which is in complete disagreement with the table. They write:
… such as the Scandinavian Simvastatin Survival Study (SSSS), which achieved an NNT of 11.7 and an NNT per year of 2.2
And they do this again and again, as if they never saw the table in their own paper. So, Superko and King are dividing the overall NNT by the length of follow-up in order to find the NNT per year. If they continue to do this they will find that the NNT per six months in the 4S-trial was 1.1. This would mean that only one patient had to be treated for six months to prevent one event. Completely absurd.
I wonder, do these renowned scientific authors not understand what they’re talking about, or is this just a slight oversight. Whatever it is, it’s serious and unprofessional. How can a respected peer-reviewed journal such as Circulation publish such rubbish? And, five years later Savarese and colleagues decide it’s time to repeat the error. And now it’s accepted by JACC, another highly respected medical journal.
Surprisingly, Savarese and Perrone-Filardi don’t acknowledge their error. Instead, they cite the old paper where the same error was made, and believe that’ll make it right.
Furthermore, despite the real NNT being a tenfold higher number than the one they reported (meaning the drug effect is ten times less), they have no intention to reconsider the main conclusion of their study. Unfortunately, their mistake was not picked up by the peer reviewers or the editors of JACC.
I must admit I’m deeply disappointed. The medical community expects much more responsibility from the editorial boards of these medical journals. If the medical literature is full of such errors, our knowledge is worthless? Maybe, in this particular context, lying with numbers, whether it’s done on purpose or not, could be called statinistics instead of statistics. Statinistics could be the new word for badly treated statistics.
Thank you for pointing these errors out Axel. The response to them was unconscionable.
Not obvious always but following the money often leads to the real reason behind lies, obfuscations and statistical gimmicks.
unconscionable response yes, but surprising no. Thanks for the post Doc!
Thank you for the post Axel and keep up the good work
I am awed by your ability to see what these guys were up to and where they went wrong. It is very sad that these men were able to do this and it was not checked and double checked as it should be with such important findings.
Thanks for this post. For people like me, who are starting out, this is a great example of how not to accept everything you see in the big journals!
Claiming ignorance of the law does not excuse one for a misdeed; one is chastised by the courts. Claiming that one was only playing with a gun does not excuse one for a killing; one is prosecuted in court. Claiming to be an authority, claiming to have done research, reliance on others to check the data lead to misdemeanors in the published medical world. People’s lives are at stake, and people suffer harm or death because of these misdemeanors, yet when does one hear of criminal procedures being instituted against those who publish or produce the misinformation? It would seem that in this field, the sad joke that “doctors merely bury their mistakes” is more widely applicable than to just doctors in the “health and medical” world. I’m not talking about malpractice cases brought by private persons, but criminal court proceedings by the State. Makes you wonder, does it not-?
It’s a bit concerning that such things still happen.
But let’s play the game the other way around. The real life exposure to statins might be much longer than that of used in randomized trials. Typically, randomized trials end at about 3-6 years.
If you lengthen the period of observation, say from 5 years to 25 years, NNTs tend to become smaller and smaller nearing 3 or 10 or whatever. Therefore, NNTs for statins in prevention of hard end points such as heart attacks, should not be interpreted as absolute number for any given time period but time-dependent relative estimates. What do you think of this Doc?
That’s a good point Reijo. However, a clinical trial can only answer a certain question or hypothesis. If a trial with a mean follow-up of 5 years gives you a specific NNT number for that period of time, it really tells you nothing more. In this situation we can’t conclude that the NNT for 25 years will be a 5 times lower number. There’s no guarantee that the NNT is constant over time. Possible adverse effects of treatment might kick in after 10-15 years. People get older and kidney function deteriorates. This might cause problems and affect the efficacy and risks of treatment. So translating the results of a clinical trial to another situation, whether it’s a longer time period, another patient population or something else, is always controversial.
Another issue with NNT numbers is that in almost all chronic non-communicable diseases NNTs are high whatever the treatment. Most treatments what cardiologists use as gold standards in prevention of second heart attack have NNTs no lower than those of statins’, ie. Aspirin, beta blockers, thrombolysis, clopidrogrel and ACE-inhibitors, and even PCIs. This is because endpoints acrue slowly in clinical trials, patients have multiple treatments beyond the one tested and you cannot reverse atherosclerosis and calcification in a larger scale.
NNT numbers are easy targets for critics. For many interventions, what many doctors/dietitians/low carb proponents endorse, there is no NNTs available. For example, there is no NNT available for low carb diets in prevention of heart attacks because there is not a single randomized trial on low carb diet with heart attacks as an endpoint. For Mediterranean diet there is.
It follows that interventions that are never exposed to large randomized trials, like low carb or paleo diets, escape NNT scrutinity and criticism. I hope to see NNT figures for paleo and low carb diets one day, so that we can then make fair comparisons in prevention of heart attacks.
This site offers nice data on NNTs in cardiology and more https://www.thennt.com/home-nnt/#nntcardiology
Makes me want to reconsider the way I look at long term end point trials
https://www.telegraph.co.uk/health/healthnews/10832580/Statins-may-have-no-harmful-side-effects-as-controversial-paper-withdrawn.html
Claims that statins cause muscle pain, tiredness and diabetes contained in a paper by a Harvard medical expert may be wrong and have been withdrawn after errors were found in the research.
The British Medical Journal (BMJ) said it was setting up a panel of experts to decide whether it should completely retract two articles which claimed the drugs – taken by millions of Britons – caused harmful side effects and did not cut death rates.
The papers prompted a row, with one expert calling them “misleading”.
The authors, Dr John Abramson from Harvard Medical School and UK cardiologist Dr Aseem Malhotra, have already withdrawn statements from the articles after some figures they cited were found to be incorrect, the BMJ said.
It admitted the errors had not been picked up by editors or experts who peer-reviewed them before publication in October.
Excellent points Reijo,
one might think that the statin-skeptics are fervent promoters of the work of Dean Ornish: 40% reduction in LDL cholesterol in 12-weeks on an average patient, no drugs involved. But, no. The logic seems to be that the that statins are bad and low-carb diet causing increase in LDL cholesterol is good.
Good points Reijo, but unfortunately the site you cited is in fact misleading in many ways. Remember my earlier criticism when you first cited it in connection with Lyon Diet Heart/Mediterranean diet & statin therapy?
Re your last paragraph: YOU’RE deeply disappointed, Dr. S?
Imagine how “deeply disappointed” (and confused!) mere patients are when observing these pissing matches (excuse my French) between respected cardiologists, academics and researchers? What’s a poor dull-witted patient like myself to do with these conflicting “expert” opinions?
PS to Charles Grashow re the Abrahamson/Malholtra BMJ study on statin side effects: read Dr. David Newman’s comment at https://www.bmj.com/content/348/bmj.g3306?tab=responses
Thanks for the link Carolyn. I think the BMJ should be applauded for the “Recent Rapid Responses” enabling individuals to respond with electronic letters to he editor… and I understand it hurts experiencing the “secondary role” of the patient himself in the whole discussion.
Dr Newman is correct in pointing out that the reporting of adverse effect in RCTs is not as rigorous as it should be. However, if you read a few comments more you’ll see that studies like this
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2432025/
are being cited as evidence of the harms although the mere study design renders them inaccurate. And what about managing the milder side effects such as myalgia?
Or the fact that – just like Reijo pointed out – virtually all medication carries some sort of risk of side effects but some some “strange” reason it’s precisely the effective cholesterol-lowering medication that gets the heat, time after time … Hmm?
Simple maths if the NNT for 3 years is 36, the NNT for 1 year is 108 not 12. Taken to the extreme if the NNT was 36 over 3 years you would only need to treat 1 person for a month to prevent an event. Well done for pointing this out I wonder how many other places in the literature this simple error appears
I’m suprised that people are turning this discussion into whether statins are beneficial or not. Come on, WAKE UP! This should be discussion of honesty of the people who practise the science. I would call it cancer of science. Very often people endorse science and say that it’s create because it´s objective. And indeed as method science is great but people tend to forget that the people performing the science are not necessarily objective at all.
I really would like to see the time when for example Mie and RichardOrnishForLife could really give some critisim to this too.
Thanks Someone. This is exactly the point. It’s not about whether statins are good or bad. It’s about the responsibility of scientists and medical journals.
In fact the whole thing is quite embarrassing for the authors of the paper, and for JACC, and their reviewing process. But if you try to wipe it under the rug, you may get away with it – and in fact I think they will.
Someone: if you’re missing me being critical in this particular case, have no fear. I totally agree with what Axel wrote, even though I saw no particular need to underline it. Making mistakes is one thing – everybody does that, to some extent – but not acknowledging one’s mistakes properly is a far more serious problem.
Mie: glad to hear that…
DEAR Axel,
This is not a comment on the ethics or otherwise, as I do not qite understand what the Statistical maths invoved here. But agree, if your maths are correct,ethics show up poorly in not admitting it.
My point is the change in FDA rules recently on Statins use, and is it because of any infirmity in such reports.
I (79yo)was recently told by my Doctors in India, as I am an Indian,there are two aspects of Stains in the recent
FDA rules on use of statins (a) an 80 mg dose for reducing LDL C and (b) a 10 mg to prevent clots causing MI.Above 75 yo, only 10 mg recommended . And (I am a layman) why arereferences being made to LDL C. I was told the new ATP3 of AHA, has made any reference to LDL C unneccessary to be recommended by Cardiologist. If hsCRP is below 1.0, by Lipoprotein testing,the older Lipid Profile are not of much importance.
VEPA
I wonder, do these renowned scientific authors don’t understand what they’re talking about, or is this just a slight oversight.
There are third and fourth options that are probably more likely than ignorance and absent-mindedness – cognitive dissonance and outright fraud.
Researchers fall in love their own ideas and rarely change them regardless of what the data say. This phenomenom is what led Max Planck to say “Science advances one funeral at a time”. Bad science doesn’t change because it’s bad science, it goes away because its advocates die out allowing new ideas a chance to gain a foothold. This is commonplace. And the expectation that peer review by reviewers who most likely believe the same dogma or whose careers/reputations hinge on the status quo remaining the status quo is bound for disappointment.
And I doubt I have to explain fraud – much more fun to give an example. Of the 9 members of the group that determined NCEP’s latest guidelines that decided cholesterol targets needed to be lowered even further, 8 of them have taken money (honoraria, grants, speaking fees) from pharmaceutical companies. I find it difficult too believe there wasn’t bias involved in that decision, whether it was purposeful or not doesn’t matter.
Science today is as honest as a third world dictator. I thinks it’s nearly impossible for interested parties to truly know a best course of action from published literature. I recently read an article claiming that 50% of published is incorrect – I think the author was an optimist.
You writre in the section,
The Statins and the Elderly Saga
“Clinical trials have shown that these drugs lower mortality and reduce
the risk of future cardiovascular events among people with CVD.”
Do statins really lower mortality? Evidence?
Kent
There are a few large placebo controlled trials that have shown that statins lower mortality in patients with established cardiovascular disease. The 4S trial was the first https://www.ncbi.nlm.nih.gov/pubmed/7968073?access_num=7968073&link_type=MED&dopt=Abstract
One of my hobbies is correcting errors in the medical literature, and I have been amazed at the cognitive deficits I have encountered at the highest levels of medical publishing. The inability to comprehend logic or basic mathematics is astounding, and getting them to admit an error or fix it is like pulling teeth. This case, involving NNT, is typically egregious. Errors such as this should be punished by public flogging, and the NNT should be small if the proper implements are used.
Well said. Couldn’t agree more.
Thank you. I had a letter published in the BMJ Open in reply to the anti-statin phony study, and the author was forced to admit I was right – statins are good for most seniors. I will get the link for you.