Skip to content
Advertisements

The Embedding Formative Assessment (EFA) trial

July 31, 2018

Quality randomised education trials are possible, but they deserve better reporting

    

Embedding formative assessment in schools “probably” raises GCSE scores, but only by a little bit

Good teachers continually check their pupils’ grasp of a topic so that their teaching can be tailored to progress. Not got it, try again. Got it, move on. This sort of formative assessment also helps pupils direct their own study appropriately. It may consist of anything from asking an individual student a question or marking a piece of homework, to class wide mock exams or quizzes.

Most teachers know this is a good thing and try to do it regularly, but education expert Dylan Wiliam (click here) is a super enthusiast; he’s written books and study guides to help teachers do it better (click here) and travels the world extolling its importance.

Last month the Education Endowment Foundation (EEF) (click here), an outfit which among other things tries to put education on a firm evidence base, published the results of a randomised trial of Wiliam’s programme. They claim it works:

“Students whose teachers were trained in this approach made two months more progress than a similar group of pupils whose teachers did not receive the intervention. The findings have a very high level of security as it was a large and well-run trial, which means we can be confident in the results.”

But the EEF has not always covered itself in glory with its trial reports. A couple of years ago it drew howls of derision, from me and others, for falsely claiming that a negative trial of teaching philosophy to primary pupils improved their mathematical skills (click here). So let’s take a close look.

Appropriately for a teacher/classroom-based intervention, it was a cluster trial with schools as the unit of randomisation. The main report (click here) is turgid and repetitive, but I’ve read it, so you don’t have to. Here’s what happened.

Population – 140 UK secondary schools during the 2015/16 & 16/17 academic years.

Intervention – Each school implemented Dylan Wiliam’s Embedding Formative Assessment (EFA) programme. They got his EFA pack, a day’s training from the man himself, and ongoing support from the Schools, Students and Teachers (SSAT) network.

Control  – Each school got a one-off payment of £300, but otherwise carried on with ‘business as usual’, with no restrictions on how they took forward formative assessment.

Outcome – The pupils’ “Attainment 8 GCSE scores” calculated from their top eight GCSEs, each graded from 1-9, with maths and English counting double. Max score = 90. Pupils who took fewer than eight subjects scored zero for each unfilled slot.

Planned sample size – 120 schools, 60 per group, with 100 students per cluster (school), and assuming an intra-cluster correlation of 0.2, was judged to have 80% power at the 0.05 significance level to show an improvement of 0.2 standard deviations in the mean “Attainment 8 GCSE scores”. This 0.2 SD effect size was judged by the EEF advisory panels “to be an acceptable level of improvement from a policy perspective to roll out the intervention more widely”. In a medical trial this would be the minimum clinically important difference (MCID). I guess we could call it the minimum educationally worthwhile difference (MEWD).

The trial recruited 140 schools, an additional 20 to allow for attrition. However, although twelve intervention schools eventually gave up on the programme for one reason or another, excluding them from the intervention arm would have biased the results, so the final analysis was of all 140 randomised schools, analysed by “intention to treat”.

The trial wasn’t registered but there’s a protocol from 2016 (click here), and an undated statistical analysis plan (click here). I couldn’t find any major outcome switching, or other risks to the trial integrity. A lot of effort was made to achieve and measure fidelity to the programme, but it turned out that many intervention schools adapted it in unanticipated ways. A planned subgroup analysis of high fidelity schools was eventually judged impracticable, so the trial was a pragmatic test of the effect of implementing EFA in the real world. Schools were randomised within blocks, based on similar GCSE scores and proportions of pupils eligible for free school meals, so the trial groups ended up balanced on these factors.

There is a CONSORT flow diagram, and table 4 shows the trial groups were indeed well balanced at baseline.

       

Results

The trial was negative. The mean intervention group score was only 0.1 standard deviation higher, a difference that could have occurred by chance (P=0.088) using both the conventional and the pre-specified level of statistical significance (p<0.05). However, the 95% confidence interval around the effect size went from -0.01 to 0.21, so they had just failed to rule out their pre-specified minimum worthwhile effect of a gain of 0.2 standard deviations (table 5).  For those who are unfamiliar with education trials expressing their results this way, the second right column, labelled Hedges g, is the difference in mean scores between the trial groups, measured as proportion of a standard deviation, together with it’s 95% confidence interval. The right hand column is the P value, the probability that a difference as large or larger than the observed one would have occurred by chance if the treatment had no effect. ln summary a negative trial but, as it turned out, also a slightly under-powered one.

       

Sub-analyses, among children eligible for free school meals, and those scoring in the lower tercile (Table 6) showed the same non-significant 0.1 SD higher scores as in the overall sample, and there was negligible effect in the upper tercile pupils, or on English and maths scores (table 7).  TEEP is the Teacher Effectiveness Enhancement Programme, another intervention applied in some schools. The analysis of non TEEP schools was exploratory.

So how come EEF claims it works?

It appears that the report authors (who do not include Dylan Wiliam) quietly decided, after the results were in, that a level of 10% significance was OK, and that an improvement of only 0.1 standard deviation would be worthwhile after all. Hey presto! The result is positive.

Cheeky eh?  Imagine big pharma getting an effect size of half what they had pre-specified as the minimum clinically worthwhile difference, and a P value of 0.088. Imagine if they then announced not only that the smaller effect size was worthwhile after all, but that P <0.1 was what they had been aiming for all along. Doctors would be sceptical.

Or perhaps not. Imagine if the drug manufacturer was working in a difficult field where there was hardly any evidence that anything worked, where all previous trials had been tiny, fatally flawed, or worse, and that the P=0.088 trial had been otherwise well conducted.  Doctors, and even regulators, might well decide that, at least for the moment, we should use that drug.

The same applies to Dylan Wiliam’s Embedding Formative Assessment. This is one of very few large trials in education. One of even fewer with a proper protocol, a predefined endpoint and analysis plan, and most important of all an endpoint, GCSE exam results, that matters to parents and pupils. Sure, the benefit was smaller than the authors had hoped for, and the results didn’t quite reach conventional levels of statistical significance, but there’s still a smaller than 10% probability that the difference observed occurred by chance. The effect is also plausible.

If I was a head teacher looking to raise my school’s GCSE scores, I’d seriously consider buying Dylan Wiliam’s Embedding Formative Assessment programme. But I wish the EEF would report their trials more honestly.

Jim Thornton

Advertisements

City canoeing in Hamburg

July 19, 2018

Not the Elbe – the Alster

The 56km Alster river joins the Elbe in the centre of Hamburg, and the Canoe Club (click here) is so old that the German word kanu didn’t even exist when it was founded in 1905.

They own a lovely old building with all facilities, a camping lawn, and private river frontage right in the middle of the city.

  

For a modest fee members of other canoe clubs can camp and use the facilities. A great base for an urban paddle. We did two, each about 6km there and back

Trip 1. Upstream to Fuhlsbüttel dam and power station

Distances measured from the canoe club. Right and left banks labelled as for a downstream paddler.

200m – Meenk bridge

    

300m – Railway bridge (1941).  Beach bar right upstream

    

500 m – Deelböge bridge. As you pass under it you feel you’re entering open country.

  

Branches left and right mark the original river course. The straightened new channel varies in width.

800m – Metzger bridge. Still very rural. An estate of lovely old (im)mobile chalets left bank

     

1.3 km – Damm bridge 1918. Braband Bistro upstream right bank.

        

1.7 km – Hindenburg bridge 1920 Skaggerak canal leaves left

      

2.2 km – Senglemann bridge 1919 (widened 2001).

   

3 km – Hasenberg bridge 1913.  The river widens in front of a park on the left bank, allowing the tourist boats to turn.

     

Fuhlsbüttel hydroelectric power station (110KW) (2000), the navigation limit for larger boats. Fish bypass and canoe rollers.

Hamburg airport is a few hundred yards west, and Ohlsdorf, the world’s 4th largest cemetery an equal distance east. Helmut Schmidt, West German chancellor, James Last, big band leader, and 1.5 million others are buried here.

Trip 2. The Isebek canal

Ise beck, a tiny stream running into the Alster was canalised way back, became polluted and nearly got paved over in the 1960s. Various sewage treatment plants and storm water basins keep it reasonably clean today, but the resulting flow is too low for the water to be adequately oxygenated for most fish. In 1988 oxygenation pipes were installed, and they seem to be working well. No pong in June 2018. More here.

From the canoe club turn right. Downstream on the Alster

50m Fahrhaus footbridge. Weighed down with love padlocks

100m Winterhuder bridge followed by riverside restaurant left

         

Again side channels follow the river’s original course.

300m – U3 railway bridge. Tourist boats fatten their customers.

   

500m – U1 railway bridge

    

800m – Goene bridge

1 km – The next bridge leads to Alster lake. Too windy for me. Turn right under Heilwig bridge and enter Isebek canal.

        

The canal is lively with boat clubs and and bars.

1.3 km – Ise bridge

     

1.5 km – Railway bridge

        

1.6 km – Eppendorf bridge

         

2 km – Kloster bridge

    

2.25 km – Grindel bridge

About 50 metres after this bridge the canal narrows. Thick bushes hide the urban surroundings.

2.5 km – Manstein bridge

2.6 km – Geben footbridge

   

2.8 km – Bundestrasse bridge.  Excellent landing stage and access.

3 km – The canal ends in a stagnant dead end. No attractive landing.

   

But overall a fine urban paddle.

Jim Thornton

 

 

 

 

 

 

The Ronne river (Rönne å) in Sweden

July 1, 2018

  

Canoeable from Klippan power station to the sea and, I’m told, also a bit above the top Klippan dam. We paddled from the E4 motorway bridge. Gentle touring, no rapids, but beware fallen trees. A tourist boat, the Laxen, cruises the scenic loops around Angelholm, home of Koenigsegg sports cars and an ice cream factory. The locals love their bridges – I counted 16 between the E20 and the sea – although Tullport, the latest, is causing them trouble.

The best map is Terrankartan 1:50,000, no 507 Helsingborg. We stayed at the famous Starby B&B, (click here) run by the legendary Urban Lennartson; don’t miss the fermented fish and model Holy Land!

0 km – Klippan dam. Take the road signed to KMT AB Entr & Service. Barrier 0.5 km up the track, so longish carry. The track is good and the launch spot easy. The 0.7 MW power station (1950), one of eight run by Skanska Energi, is small for a hydroelectric power station, but approximately equivalent in power output to the largest wind turbine ever built.

      

  

1 km – A13 bridge. No easy access.

1.5 km – Railway bridge

8 km – Tranarp road bridge. No easy access.

8.5 km – E4 motorway bridge. Excellent access from the Starby road from which a track runs under the flyover to the river.  We started here.

   

10.5 km – Flimsy wire footbridge

12 km – Starby left. No easy access but the church is visible over the trees.

   

14 km – Starby bridge. Access left. Parking.

    

17 km – Neuhof bridge. Hoja left.  Access right. Parking

       

20 km – F6 E20 motorway bridge

20.5 km – Angelhom ring road (kungsgardsleden) bridge

22 km – footbridge

23 km – Angavangs bridge

23.25 km – Kristian II bridge

23.5 km – footbridge

23.75 km – Tegel bridge

24.5 km – Carl XV bridge (2005). Site of a much older bridge (click here)

    

24.6 km – Tullport (Customs port) bridge. Built in 2017 and already closed for repair. Not only is it sloping 4 degrees but the beams were shifting and rusting within months of opening. Good looking, but surely over-engineered for a footbridge.

      

Lilton hotel right

   

25 km – Jarnvags bridge. Railway station left.

   

25.25 km – footbridge. Followed by some lovely riverside apartments.

   

25.5 km – New bridge and more handsome flats

     

26 km – footbridge. Followed by Angelholm park right.

26.5 km – Roddock canoe club (click here) left.

  

28 km – Two adjacent railway bridges

29 km – Footbridge followed by marina right. Access and parking.

  

Jim Thornton

 

 

Extremely preterm baby dies from natural causes

June 19, 2018

Cord prolapse during breech birth

But the head became detached after death, or as The Sun put it last week: “Doctor decapitated baby in botched birth” (click here), and the story flew round the world (click here). The obstetrician was criticised by a public enquiry, but found fit to continue practice. The mob called for blood, and even level-headed lawyers raised their eyebrows.

              

What happened? I only know what was in the papers.

The baby was 25 weeks, i.e. 15 weeks early. Average weight at this gestation would be about 600 grams (1.5 lbs). Not so long ago, this would have been classed as a miscarriage, albeit a late one, and the mother would never have gone near a maternity ward. But over the last couple of generations, advances in neonatal care have pushed the limits of viability ever lower. In the best centres survival rates at 25 weeks are now about 60%, although about half the survivors will have some form of disability (click here). In this pregnancy the membranes had ruptured and the heart beat was slowing due to cord prolapse, so this baby’s chances were probably somewhat worse. 

Nevertheless the tribunal criticised Dr Laxman’s decision not to do a Caesarean. In fact they went further and said it “was negligent and fell below the standards ordinarily to be expected”. Not surprisingly the lawyers raised their eyebrows.

But unless the tribunal members had additional unreported information, this is an odd ruling. Caesarean is not mandatory for preterm breech births. The National Institute for Clinical Excellence (NICE) considered the matter at length (click here) and, even after 26 weeks, limited their recommendation to (my italics) “consider Caesarean […] between 26+0 and 36+6 weeks of pregnancy with breech presentation.” Below 26 weeks they decided not to make any recommendation.

NICE’s reasoning was sound. At term Caesarean is a bit safer for a breech baby. But preterm is less clear. Caesarean might be less traumatic, but it also brings forward the birth; preterm labour is difficult to diagnose, and may stop for days or even weeks. Unless you are certain the baby will deliver rapidly, the risk of Caesarean increasing prematurity often trumps the hypothetical benefit of reduced birth trauma. The uncertainties are such that the NHS has recently commissioned research to see if a randomised trial is feasible.

But what about after the cord prolapse? The baby will certainly die if not delivered within a few minutes.  But it’s not obvious that Casearean even then is the best option. For a very preterm baby in advanced labour, Caesarean is also traumatic, and the baby may deliver quickly vaginally. I know of two cases (not in my hospital!) where doctors did an emergency preterm Caesarean and found an empty uterus – the baby had delivered by the normal route under the drapes while the surgeon cleaned and incised the skin! 

NICE also considered whether at these limits of viability it is worth monitoring the fetal heart at all. Slow heart rate detects cord prolapse but may also push obstetricians into unwise Caesarean against their better judgment. It’s difficult to refuse a maternal request even at 25 weeks.

This is a controversial area, and again NICE’s experts wisely fudged the issue (again my italics): “Involve a senior obstetrician in discussions about whether and how to monitor the fetal heart rate for women who are between 23+0 and 25+6 weeks.” In my experience most UK experts advise against monitoring before 26 weeks, but even those who recommend it should also agree that Dr Laxman, who was a senior obstetrician, was working right at the limit of viability where even the best experts say there are no hard and fast rules. 

So why did the tribunal rule that not performing a Caesarean was negligent? I can’t read their minds, and perhaps they had further information, but I’ve sat on similar panels. It’s often politically difficult to say that an emotive tragedy was unavoidable. Easier to be wise after the event and blame someone. Dr Laxman, was in the frame because of what happened next. 

The baby didn’t deliver quickly. The body slipped through the incompletely dilated cervix and the head got trapped. Any obstetrician knows this is a horrible problem. For a larger baby the forceps can be applied to the trapped breech head, but not at 25 weeks and not with the cervix only 4 cm. Some experts advise incising the cervix but that’s a desperate, dangerous measure. Dr Laxman decided against. The baby died. 

Dr Laxman should have taken stock at this point. There’s no longer any clinical urgency, an oxytocin infusion could have been started and the baby would have soon delivered spontaneously. But emotions were high; the mother was awake, and perhaps the father was present too. They must have both been distressed. Assisting staff would also have been upset. Dr Laxman pressed on, urged the mother to push, and pulled again.

The head detached from the body. Extraordinary, but not unheard of, especially if a baby has been dead some time, although that situation did not apply here. Did anyone check whether the baby had some sort of connective tissue disorder? The newspapers don’t say. 

At this point other doctors got involved. They decided to perform a Caesarean to retrieve the head. Again we can criticise. Patience and an oxytocin drip, or slow removal through the vagina with the mother under a general anaesthetic and the father out of the room, would be my preferred option. But now I’m being wise after the event. Caesarean retrieval was reasonable.

In summary, a baby at the limit of viability died of natural causes, but a dreadful thing happened afterwards. The consultant involved, under pressure, pulled the head off. We don’t know whether she pulled too hard, or if some abnormality in the baby caused unusually weak ligaments. She had a previously unblemished record. No-one suggests it was anything other than an accident. It certainly wasn’t malicious. She did her best in a horrible situation. 

So what was gained by the public enquiry? Do the parents feel better? Will Dr Laxman, an ethnic minority doctor with a previously unblemished record – Dr Bawa Garba (click here) comes to mind – return to work? Baby decapitation headlines all over the world don’t make birth safer; they make sensitive doctors choose less demanding specialties, and the remainder defensively perform even more Caesareans. No lives are saved, and mothers are harmed. 

But newspaper readers had a pleasurable frisson of horror, and the mob had a chance to bay for blood. That’s something.

Jim Thornton

Canoeing the Loire – 2

May 31, 2018

Chateauneuf to Blois

Distances measured from Nevers (click here for Nevers to Chateauneuf). We camped at La Maltournee in Chateauneuf, and Val de Flux in Beaugency, but there are plenty of others.

No major obstacles. The rapid under George V bridge in Orleans is bouncy, and on a sunny day a tip would have an audience. But the rapids at Jargeau, Meung and Beaugency are all easy.

The weir at Saint Lauren-Nouant nuclear power station is not shootable. But it’s an easy safe portage.

No famous vineyards. The local AOCs are Orleans, Orleans Clery and Cheverny. AOC Cour Cheverny is special; white wine made in a few select villages from the ancient Romorantin grape found nowhere else.

132 km – Chateauneuf sur Loire bridge (1946) Town right. Campsite La Maltournee left bank 100 metres upstream of the bridge (click here). No local wine, but one of the few small breweries on the upper Loire, La Brasserie des Ecluses (click here) is a couple of miles up the road at Vitry aux Loges.

138 km – Jargeau bridge (1988)  Jargeau left. Saint Denis de l’Hotel right. Small rapid at site of an older bridge, downstream of the present one. Easily shot left or right.

Camping L’Isle Aux Moulins left after bridge (click here)

146 km – Checy right. Easy access.

    

Camping right (click here) (plan here plan_camping_municipal_checy) between the river and canal d’Orleans.

   

An adjacent campsite, also between the canal and the river, seems to have closed.

   

Alliance canoe rental opposite bank (click here)

148 km – Combleux right.  The canal d’Orleans connects by lock to the river here and runs along the right bank to Orleans.

   

      

The right bank is Orleans AOC (details here). A few locals are making an effort (click here).  Keep to the right side of the river through Orleans.

As Orleans approaches river traffic increases

152 km – Vierzon railway bridge (1947) followed by Pont Rene Thinat (1977)

          

Canal d’Orleans junction right. The Orleans terminus

    

This is the nearest the river gets to Orleans centre. i.e. not very near.

153 km -George V bridge (1763) Orleans centre right.  The rapid under the bridge deserves respect in an open canoe, but we got through without taking on water.

154 km – Marshal Joffre bridge (1958)

155.5 km – Europe bridge (2000)

Canoe club right. Good landing spot. Drains a bit smelly.

  

The campsite Gaston Marchand right about 200 metres upstream of motorway bridge (click here) seems to have closed (May 2018).

157 km – A71/E9 motorway bridge (1980)

158 km – Eglise Saint Mesmin right.  Les Forges right.

Camping Le Chateaux right, a few hundred yards after the church is an RV stop. No facilities.

159 KM – River Loiret joins left.  Easily missed. Carries little flow.

165 km – Saint Ay right. Camping la Fontaine de Rabelais (click here). Quiet when we paddled past.

       

The right bank AOC Orleans again. Left bank AOC Orleans-Clery (red cabernet franc). One of the best is Clos Saint Fiacre.  No vineyards along the bank. Most in or around the village of Mareau-aux-Prés.

orleans-clery

168 km – Meung-sur-Loire bridge (1948) Village right. Chateau Meung-sur-Loire (click here)

Shallows at site of old bridge/ford about 200 metres above the new suspension bridge. Shoot right.

Public swimming pool right bank just below the bridge.

174 km – Beaugency bridge (14th century)  Beaugency right. Camping Val de Flux left bank just upstream of the bridge. Click here for more. Rapid under the bridge. Shoot left or right.

         

Indoor and outdoor pool right bank just after the first island after the bridge.

179 km – Saint Lauren-Nouant des Eaux nuclear power station left.

Weir. Warning sign at 2km advises canoes to keep right. Ignore ambiguous sign right bank at 1km. Easy portage right just above the weir. Click here for more.

                 

181 km Camp friendship left.  Click here

186 km – Muides-sur-Loire bridge (1932). Camping Municipal Bellevue left bank below the bridge.  No website but it looks lovely. Easy landing left either before or after the bridge. The day we arrived a bric a brac sale was going on.

    

The left bank is now AOC Cheverney. Domaine Croc du Merle, farm and vineyard (click here) sells wine, cheese and other delicacies. But they don’t make or sell AOC Cour Cheverney. That is reserved for wine made in a few central villages from the traditional grape, Romorantin, found nowhere else.

       

188 km – Saint-dye-sur-Loire left

       

194 km – Cours-sur-Loire right.

      

196 Chateau de Menars right.

    

198 km – Disused railway bridge.

199 km – Camping left Val de Blois.  Activity centre, good access, campsite, canoe hire, pool, and this extraordinary structure. It is the Restaurant Le Lac de Loire. Some crazy architect decided the Loire needed this sort of brutalist concrete. But I rather like it.

         

We stopped here, so Blois itself will have to wait till next year.

Jim Thornton

Camping Val de Flux

May 29, 2018

Perfect base for Loire canoeing

I don’t usually post about individual campsites, but this spacious one just upstream of Beaugency bridge deserves the exception; an eleven out of ten.  Click here for the official site.

Good river access.

Super clean toilets and free hot showers.

The friendly barman speaks excellent English

The groundsman loves his job.

   

The goats praise God by being goats

Many other lovely touches

A few static tents

And loads of space, no unnecessary nonsense, and Beaugency over the bridge is lovely.

Jim Thornton

Saint Laurent des Eaux nuclear power station.

May 22, 2018

On the bank of the Loire

France’s third nuclear power plant opened in in 1969, with one Uranium Naturel Graphite Gaz (UNGG) gas-cooled reactor (similar to the UK’s Magnox reactors); a second was added in 1971. They were shut down in 1990 and 1992 respectively, but the housings remain, apart from the cooling towers, the largest buildings on the site, and the first things a canoeist sees.

In 1983 two 900MW pressurised water reactors (PWR), of similar design to the UK’s Sizewell B, came on stream and remain in service.  They are much smaller, and as trees have grown around the site, the concrete domes are inconspicuous from the river. In the left hand picture below they are visible in front of the right hand cooling tower.

The PWR reactors use water from the Loire to cool the steam after it has powered the turbines. Hence the cooling towers which condense most of the steam back to water for return to the river.

The cooling towers are visible from Beaugency. Two km above the power station a sign on the left bank warns of the forthcoming barrage. One km later an ambiguous sign on the right bank indicates that canoeists should keep to the right bank. Don’t misinterpret it as “land and start the portage here” or you’ll have a long carry.

The landing point is about 50 meters above the weir, marked by a proper portage sign, and easy to see and land at.

    

The portage itself is on a concrete path. Arrrgh! I’d forgotten my wheels.  With a sandy beach put in.

Here’s some views of the power station and the weir

   

  

A lovely spot for a picnic.

I love nuclear power.

Jim Thornton

%d bloggers like this: