In the spring of 2014, a debate on the use of statins (cholesterol-lowering drugs) in healthy individuals, and the risks of side effects associated with such therapy reached the headlines of many news media.
One of the points raised was that doctors can inflict harm to their patients by exaggerating potential adverse effects of statin drugs. This might lead to a patient’s unwillingness to accept treatment that is of potential benefit.
What followed was a heated discussion about the responsibility of medical journals and their peer review process, how erroneous statements should be corrected and when scientific papers should be retracted.
The BMJ Story
On March 21, 2014 Dr. Rory Collins, a Professor of Medicine and Epidemiology at the University of Oxford was quoted in the Guardian saying that:
Doctors worrying about the safety of cholesterol-reducing statins are creating a misleading level of uncertainty that could lead to at the loss of lives…
In fact, Professor Collins had alerted the editorial board of The BMJ to an error in two articles (1,2) published in the journal addressing the side effects of statins. Both articles stated that side effects of these drugs occur in 18-20% of people, which turned out to be inaccurate.
The editorial board of The BMJ and the authors of the two papers reacted by officially withdrawing their statements.
On May 15th, 2014, The BMJ published an editorial written by Fiona Godlee, editor in chief.
The editorial aimed “to alert readers, the media and the public to the withdrawal of these statements so that patients who could benefit from statins are not wrongly deterred from starting or continuing treatment because of exaggerated concerns over side effects.”
In her article Dr Godlee also raised the question whether a correction was enough in this case. She pointed out that Professor Collins didn’t think so. In fact, he requested the retraction of both articles.
Dr. Godlee wrote that
Guidelines of the international Committee on Publication Ethics state that journals should consider retracting a publication if there is clear evidence that the findings are unreliable, either as a result of misconduct or honest error
Dr Godlee decided to pass a decision on whether to retract the papers to an independent panel.
Later the panel came to the conclusion that retraction of the two papers was not necessary:
On the basis of the statistical reviews and their reading of the background papers, the panel concluded that the calculations in the paper had been largely sound
The JACC Story
At the beginning of January 2014 I alerted the editorial board of the Journal of the American College of Cardiology (JACC) to an error in a paper addressing the efficacy of statin drugs in elderly subjects without established cardiovascular disease.
The authors had calculated that 24 patients needed to be treated for one year to prevent one myocardial infarction (heart attack) and that 42 patients needed to be treated for one year to prevent one stroke.
On closer inspection, it was obvious that they had made an incorrect calculation. In fact, the correct values for the numbers needed to treat (NNT) are approximately tenfold higher. Thus, the paper had exaggerated the effect of statin therapy by a factor of ten.
In a March issue of the JACC the following correction was published:
The authors report the number needed to treat (NNT) for the entire mean follow-up of studies was 83 and 142 for myocardial infarction and stroke, respectively.
The authors apologize for this error.
So instead of correcting their error, the authors decided to publish the NNT’s for the whole study period. Although this was probably what they should have done from the beginning, providing these numbers is not a correction, it’s just additional information.
In their response letter, they explain that the NNT’s for one year published in their original article were achieved by dividing the overall NNT calculated for the entire trial duration by the length of the follow-up.
However, they never acknowledged that they made an incorrect calculation nor that they should have multiplied, but not divided.
Furthermore, neither the authors nor the JACC’s editorial board bothered to correct the initially published NNT’s for one year.
Anybody reading their paper for years to come will find these wrong numbers which exaggerate the effect of statin therapy by a factor of ten.
Recently, in a formal letter, I asked the editorial board of JACC to address the issue again. The following statement concluded my letter:
It would be in the best interest of the JACC and its readers if the authors acknowledged their error, published the correct NNT’s per year and made an apology. By doing that, further misunderstanding and misinterpretations may be avoided.
A few weeks later I received the decision letter from the editorial board of the JACC. This is how they handled the issue:
Dear Dr. Sigurdsson:
After publishing your first letter, the authors’ response and an erratum to the paper, we do not feel that it is necessary to publish further dialogue on this point. We suggest you contact the authors of the original paper.
Thank you for your interest in the journal.
Is it about The BMJ vs. The JACC…
In my opinion, there are a lot of similarities between these two stories. Both deal with erroneous data and how such issues should be dealt with by medical journals relying on peer reviewing.
However, despite the similarities, the different methods used by these two medical journals are striking.
The editorial board of The BMJ seems to have acted both seriously and responsibly. Statements were withdrawn, and corrections were highlighted in an editorial. The decision on whether to retract the two papers was passed to an independent panel.
In the case of the JACC, there was no official response from the editorial board. No apologies either.
Inaccurate and misleading “corrections” from the authors were accepted. The fact that an incorrect calculation was made was never acknowledged.
Why and how such a simple but serious error escaped JACC’s peer review process is not explained.
Is it possible that the way the editorial boards of these two medical journals were approached could explain how they handled their issues?
In the case of The BMJ, a highly renowned professor and public figure reacted by visiting the BMJ and taking up the matter in the UK media. He finally put his concerns in a letter to the editor in chief but declined several requests to send a rapid response or letter for publication.
In comparison, Dr. Sigurdsson (of course a complete nobody compared with professor Collins), kept a much lower profile by using more traditional channels, but unfortunately to a lesser effect.
The power of medical journals is immense and so is their responsibility.
They decide if, how and when results of scientific studies are published.
Their peer review process is supposed to guarantee high standards of quality and provide credibility.
Of course, all human processes are subject to error. But admitting a mistake and dealing with it in a responsible manner is a sign of strength.
So while I have to congratulate the BMJ for their professionalism, I can only hope that the editorial board of the JACC will improve its methods.
A first step might be to present ‘errata’ corrections in a manner that everyone accessing these publications in the future will be aware of the very important informational errors that have been discovered.