Did an under-powered poorly-analysed trial give a misleading result, would the results have been generalisable anyway, and did the parents understand what they were consenting their sons for?

The trial, funded by the Bill and Melinda Gates Foundation, was conducted in 2013 in Zimbabwe, “traditionally a non circumcising country” by researchers from Harare and London (2015 report here or Zimbabwe circ trial, registration here or PACTR Registry zimbabwe circ trial).

Participants –  male infants from a Harare polyclinic.

Intervention – circumcision with a new Accucirc device (left above).

Control – circumcision using the standard Mogen clamp (right above).

Primary Outcome – the number of “moderate and severe” adverse events (AEs). The registry plan to include “minor” AEs as well was silently changed in the paper.

Statistics

Using envelope randomisation in a 2:1 ratio, Accucirc: Mogen, the planned sample size of 100:50 had:

“80% power to detect noninferiority, based on a 2-sided 95% confidence interval (CI) approach, a 2% risk of AE in the Mogen clamp arm, and a noninferiority margin of 6% failure between the 2 arms. A noninferiority margin of 6% was chosen because this was deemed the maximum difference in safety that would be acceptable in terms of public health”.

A six percent absolute difference in moderate or severe adverse events is large for the “minimum clinically important difference” (MCID) for a trial of surgical devices. It implies that parents would choose the new device if the trial could reassure them that it had no more than six percent additional complications!  Did anyone ask them?

The trial ran more smoothly than almost any other in history.

“One hundred fifty male infants aged 6–54 days were circumcised between January and June 2013. All were circumcised according to their allocated intervention (n = 100 AccuCirc; n = 50 Mogen clamp). All participants attended the 3 scheduled follow-up visits on days 2, 7, and 14.”

The result was two moderate or severe adverse events in the Accucirc group and none with the Mogen clamp. One baby suffered excess skin removal which took four months to heal, and another inadequate skin removal requiring further surgery. The authors conclude:

“2.0% higher in the AccuCirc arm compared with the Mogen Clamp arm (95% CI: −0.7 to 4.7). As the 95% CI excludes the noninferiority margin of 6%, the result provides evidence of noninferiority of AccuCirc compared with the Mogen clamp.”

This is wrong. The normal approximation to the binomial’s 95% upper bound for 2/100 events is indeed 4.7%. But that formula is unreliable for small numbers. In that situation the exact confidence interval, for which the 95% upper bound for 2/100 events is 7%, is preferable.

This means that even ignoring the implausible zero envelope loss, the perfect compliance and the 100% follow-up, the correct statistical test alters the results from positive to negative; Accucirc may have too high an excess of adverse events for it to be acceptable.

The following wasn’t in their analysis plan, but if the authors had analysed their trial as a conventional superiority one, the estimated relative risk (RR) would have been infinite because of zero events with the Mogen clamp. If they had followed convention and added 0.5 events to the Mogen group to allow a relative risk to be estimated they would have RR 2.5, (95% CI 0.12 – 52).  They would be 95% certain that the true effect lay somewhere between Accucirc having 10 times fewer, or 50 times more, adverse events than the Mogen clamp.

Generalisability

The authors ran the trial to a higher standard than they could implement in practice.

“All infants received vitamin K to minimize bleeding; vitamin K should be routinely administered at birth but was out of stock nationally at the time of the trial and therefore had to be imported specifically for that purpose.”

i.e. the research was conducted with special drug safety cover unavailable in the rest of Zimbabwe. How could they therefore extrapolate from the first sentence of their conclusion: “We safely circumcised 150 infants in a randomized trial of AccuCirc versus Mogen clamp for EIMC in Zimbabwe.” to the second: “The AccuCirc device has the potential to facilitate widespread scale-up of safe EIMC in sub-Saharan Africa.”

Consent

For a trial in Zimbabwe “traditionally a non circumcising country”, the researchers had to both persuade the parents to let their baby undergo circumcision, and then gain informed consent to him participating in the research trial comparing the standard with a new “experimental” method.

“Sensitization on EIMC [Early Infant Male Circumcision] and participant recruitment took place at the antenatal clinic and after delivery in the maternity ward. Educational materials (posters and pamphlets) and demand creation activities (road shows, dramas, group and interpersonal discussions) were used to educate and sensitize the community about the trial.”

Ignore for a moment the ethics of “sensitizing” non-circumcising communities to, and running “demand creation activities” for, neonatal circumcision. If the researchers themselves confuse “sensitization and demand creation” for circumcision, with “sensitisation and demand creation” for the trial, how likely is it that parents of potential participants were clear about the difference?

“To enrol 150 babies in the comparative trial, we approached 1151 parents of newborn male infants, corresponding to a 13% uptake of EIMC. A total 984 (85%) parents declined for their son to participate.” […] “A further 17 male infants were excluded after assessing their eligibility for inclusion (Fig. 1).”

This reads as if all parents of eligible participants consented to the trial.

However, Fig 1 (the trial flow diagram above) shows that only three babies were ineligible for medical reasons, leaving 14 parents who definitely did understand the difference between agreeing to the circumcision and consenting to the trial. We know this because they agreed to the former but declined the latter.

This still leaves the consent rate for the research trial as 150/164 or 91%! That is high for any randomised trial, extraordinarily high for a surgical trial, and suggests that some parents had, like the researchers, indeed muddled consent for circumcision with consent to randomisation?

Elsewhere, while trying to explain the low overall circumcision uptake, the authors accidentally acknowledge just such a muddle, although they argue that it caused parents to decline their son being circumcised at all because of reluctance to join the research.

“Zimbabwean parents were informed that the trial was comparing 2 EIMC devices. Parents may therefore, have felt this indicated that the devices were “experimental”; this thought may have exacerbated their fear of harm.”

Subsequent non-randomised rollout of Accucirc by the same authors, in the same clinics, immediately after the trial ended (click here for details) had an even lower total uptake of 500/4617 = 11%,  so the authors’ claim is unlikely. Parents weren’t put off circumcision because they feared randomisation; they disliked it just as much when randomisation was not on offer.

No-one can prove at this distance what parents really understood. But it looks like many parents allowed their sons to join this badly designed and analysed trial thinking that they were consenting to the circumcision. This after nine months of pro-circumcision propaganda – forgive me! – for which nine months of “sensitizastion” had “created a demand”.

The study was approved by the Medical Research Council of Zimbabwe (click here), and the ethics committees of University College London (click here) and the London School of Hygiene and Tropical Medicine (click here).

Jim Thornton