Did you really think that?

AsPredicted.org (click here), created in 2015 by the University of Pennsylvania, Wharton School’s “Credibility Lab” (click here), provides a simple way for researchers to make post hoc analyses clear by registering experimental details prospectively, and creating a short, time-stamped, pre-registration document to share and read.  By publicly registering their study designs, clinical triallists have done something similar for years. AsPredicted.org makes it easy for others to do the same.

The reason is that a registered experiment, conducted and analysed as planned, is far more credible than one which cannot prove that it didn’t stop when the result was positive or select the most favourable outcomes to report.

A suggestion

Experts and opinion leaders could raise their credibility in a similar way; by registering their opinions, and indicating in advance whether they would alter in response to new data. There would be advantages both for the expert, for the general public and for researchers.

Imagine, for example that an expert judged that an experiment had a design flaw. If the expert revealed their objection only after the results appeared, perhaps in a direction not favoured by the expert, readers might suspect the expert was finding fault because he did not like the result. By committing to believe the results of a well-designed experiment the expert is indicating that his mind is open to new evidence. His reputation would rise.

The general public would have an objective way to judge who to believe; those who were prepared to commit (or not) to new evidence in advance.

Researchers would also gain, if the practice became popular, because expert criticism of experimental design would sometimes allow them to alter the experiment before it was too late.

I’ve made a start.  Here are my first five predictions.

PRISM (click here), CSTICH (click here), EPPPIC (click here), PHOENIX (click here), PITCHES (click here)

I will explain more about each one over the coming days.

Jim Thornton