Is this cheating?

You would imagine that when the UK Medical Research Council (MRC) funds a multi-million pound trial of a drug to treat Alzheimers disease, and publishes the results in the New England Journal of Medicine, the world’s highest impact medical journal, that things would be done properly.  But take a look at the DOMINO trial report last month(NEJM 2012: 366: 893-903) which claims to show that both drugs improve memory and function compared with placebo. Here is a link to the full text although you will need a subscription

The trial was registered here with a planned sample size of 800 (200 per factorial group donezipil+placebo, memantine+placebo, D+M, placebo). But only 295 participants appear in the published trial. The authors give two explanations. The first makes no sense and the second comes near to an admission of cheating.

Their first explanation is that they reduced the sample size to 430 on the basis that half way through they noticed that the standard deviations (SD) of the two primary outcome scores were smaller than predicted. These were the Standardised Mini-Mental State Examination (SMMSE) which ranges from 0-30, where 30 is the best score, and the Bristol Activities of Daily Living Scale (BADLS), range 0-60, where 0 is good. A smaller SD means that a smaller trial would have the same power to detect the pre-defined minimum clinically important difference (MCID) in these outcome scores. Namely 1.4 units on the SMSME scale and 3.5 on the BADL.

But how had the authors got those MCID numbers? They interviewed experts, who said that numbers of 1.4 and 3.5 had no real meaning. So instead they asked the experts to express the MCID in terms of fractions of the score SD.  On this basis the MCID was set at 0.4 SD (Howard et al 2010 you’ll need a subsciption for the full text). But reducing the sample size to detect a difference expressed in terms of SD, on the basis that the SD is smaller than expected, is not just circular. It’s bonkers!

But they did not even recruit the new target of 430. They gave up at 295 because “the disadvantages of a delay in reporting the results outweighed the benefits of increasing the power”.  How could anyone decide this without knowing the results? Someone peeked at the data, saw that they could squeeze out a positive result, and said “let’s stop”. That’s cheating.

If this had  been a commercial trial, the NEJM would have devoted a special issue to the iniquities of big pharma!  But somehow, we imagine government salaried doctors are honest seekers after truth. I doubt it. Twelve of the authors report receiving payments of various sorts from pharmaceutical companies. More seriously, many are NHS consultants whose chances of a getting a clinical excellence award, worth anything up to £50K per annum will improve with this paper!

Even if treatment really does give a 1.9 point better SMMSE, and a 3 point better BADL, what does that mean?

Here is the crucial figure.

Note that only part of the scales are shown and the bars indicate standard errors rather than deviations – the raw data are all over the place – and remember the SMMSE ranges from 0-30 and the BADL from 0-60.  All four groups deteriorated over the one-year follow-up. 78 (not shown on the figure but clear from another table) could not even do a score, 39 of them because they were dead, so only 217 completed the primary outcome at one year.  And what did those who took the drugs get? One point on the 30 point SMSSE scale and 3 on the 60 point BADL scale!  Talk about straining to deliver a mouse!

This matters because the press reports this as a breakthrough, pressure groups jump up and down, the National Institute of Clinical Excellence (NICE) approves NHS funding, and we all end up wasting billions on these ridiculous drugs. No wonder the lowly carers in residential homes for the elderly are underpaid.

Jim Thornton