Too good to be true?

It sounds crazy, but this trial of 12 weekly percutaneous tibial nerve stimulations (near the ankle!) to treat overactive bladder (click here, or here Peters et al) looks promising; randomised, double-blind, 110 participants per group, sham needling in controls, and analysed by intention to treat. The result a statistically significant increase in the primary endpoint of “moderately or markedly improved symptoms at 13 weeks” (55% v 21% P<0.001).

The study recruited between Sept 2008 and Jan 2009, and was registered here on October 9, 2008.  The primary endpoint is defined identically in October 2008 as in the paper. The planned sample size of 214 (later changed to actual recruitment of 220) was entered on the registration site on 24 June 2009 (i.e. after recruitment was completed). This would be a problem if the results were being inspected as the trial progressed, but hopefully that didn’t happen.

More seriously 23 centres recruited and randomised in a 1:1 ratio using “a random block design stratified by investigation site”, and somehow achieved exactly equal numbers, 110 per group. How likely is that?  Did the researchers really randomise as they claimed? Ripe-tomato.org has seen this before (click here).

But wait. The same group of researchers published another trial comparing PTNS with tolterodine, a couple of years earlier (click here, full text here) NCT00448175 registered here.  The sample size was prespecificed at 100 (50 per group) on the trial registry in 2007 and recruitment ocurred from  June 2006 to September 2008. So far so good.

The registered primary outcome was frequency of voids at 12 weeks. The result 9.8 ± 3.0 for PTNS, 9.9 ± 3.8 for tolteridine, i.e no differece. The reported primary was a subjective assessment of overactive bladder symptoms, which was nominally statistically significantly reduced in the PTSD group (79.5% improved or cured with PTNS v 54.8% with tolterodine. p = 0.01). Altering the primary outcome like this is not convincing. This trial result is negative.

But again 11 centres recruited and randomised in a 1:1 ratio using “a random block design stratified by investigation site” and somehow again they achieved exactly equal numbers, this time 50 per group.  This is an even less likely achievement, not only exactly equal numbers per group but ending exactly on the button.  They recruited those 100 patients over 27 months so it required very good communication to get all 11 centres to stop as soon as the hundredth participant joined.  I’ve been involved as researcher, data monitor or steering committee member in about 20 trials, and I’ve never seen it done.  Peters and colleagues did it twice!

Other minor worries. For some reason the researchers did not cite the trial registration in either paper. I wonder why not. In both papers many of their statistics tested differences between before and after treatment, in the same trial group!  Readers are misled by many low p values with no applicability to the randomised comparisons.

At best these triallists are naive. At worst they cleaned up their data to get nice even sample sizes, and in the process may have got misleading results.

Jim Thornton