Hands-on STEM project work leads to positive effect on students’ GCSE grades and future subject choices

Hands-on STEM project work leads to positive effect on students' GCSE grades and future subject choices

First quantitative evidence published by the British Science Association on the impact of extra-curricular STEM interventions

A Pro Bono Economics report, published today (Wednesday 27 January 2016) by the British Science Association (BSA), reveals that students who have taken a CREST Silver Award – a hands-on, extra-curricular STEM project – achieved half a grade higher on their best science GCSE result, compared to a matched control group.

The report, conducted by a group of volunteer economists, is the first independent review of its kind on the effect that undertaking practical, hands-on science projects can have on student attainment and subject choice. The findings include:

• Students who took Silver CREST achieved half a grade higher on their best science GCSE result compared to a statistically matched control group.

• Students who undertake a CREST Silver Award are 21% more likely to take a STEM AS Level. 82% of Silver CREST students took a STEM AS Level, compared to 68% of a statistically matched control group.

• Silver CREST students eligible for Free School Meals (FSM) saw a larger increase in their best science GCSE (two thirds of a grade) compared to a matched control group who were also eligible for FSM.

• Students who were eligible for Free School Meals and took part in a CREST Silver Award were 38% more likely to take a STEM subject at AS Level than the matched control group.

• Students who undertake Silver CREST have higher average GCSE grades compared to those who did not do a CREST Silver Award.

• The sample for this analysis included 2.4 million Key Stage 4 students (of whom 3,800 took CREST Silver) and 1.0 million Key Stage 5 students (of whom 2,300 took a Silver CREST Award).

• Half (50%) of students taking Silver CREST Awards were young women.

CREST Awards is the BSA's flagship education programme, which allows 11 to 19-year-olds to explore real-world science, technology, engineering and maths by curating a unique hands-on project. It is a practical science intervention, which seeks to broaden students' interest in science and encourage them to continue with STEM subjects.

There are four levels of Awards in the CREST programme; Discovery, Bronze, Silver and Gold, which each require increasing amounts of teacher and student time and mentor involvement. The analysis in this report, Quantifying CREST: what impact does the Silver CREST Award have on science scores and subject selection?, focuses on students in English state schools aged 14-16 who took part in CREST Silver Awards between 2010 and 2013.

David Willetts, Chair of the British Science Association, said:

“The BSA has been running the CREST Awards for almost 30 years. We have long-suspected that CREST is a great thing, of course – but we now have quantitative evidence to suggest that undertaking CREST Silver Award appears to have a positive impact on students' GCSE grades and the likelihood of them continuing on study STEM.

"That in itself is hugely significant. However, I am particularly interested in the finding that there is an even greater impact on students who are or have been eligible for free school meals."

In order to conduct the research for this report, data collected by the BSA on students starting Silver CREST Awards between 2010 and 2013, was linked to data in the National Pupil Database. This enabled the authors (Rosie Stock Jones, Tom Annable, Zoe Billingham and Cee MacDonald) to bring together information on CREST participants with pupil characteristics, attainment and subject selection data.

Propensity Score Matching was used to create a control group of students who did not take part in CREST, but had similar characteristics to those who did. Key Stage 2 (KS2) SATs results were used to control for prior attainment. Other control variables included gender, ethnicity, region of school, year of GCSEs, participation in triple award science, type of school, free school meal status in the six years prior to taking GCSEs and the Income Deprivation Affecting Children Index (IDACI).

Karen Hancock, Economist, Pro Bono Economics, and Ex-Chief Economist, Department for Education, said:

“There are always caveats with this type of analysis, but the results suggest that students participating in the Silver CREST Awards achieve about half a grade higher on their best science GCSE result on average compared with a statistically-similar control group. Silver CREST students were also 21% more likely to take a STEM AS level subject than control group students. We hope that the results of this report will support the British Science Association in making a case for encouraging greater uptake of the CREST Awards in schools."

The authors have made several recommendations for further work, including replicating this analysis through a Randomised Control Trial, broadening it to cover Discovery, Bronze and Gold Award types and conducting a cost benefit analysis for schools.

The full research report and additional data tables can be downloaded from this link or is available on request from the BSA press office.

[Ends]

For all press enquiries, copies of the report or interview requests, please contact:

Elspeth Houlding, PR Officer, British Science Association on 020 7019 4953

or Elspeth.Houlding@britishscienceassociation.org

Notes to Editors

Quantifying CREST: what impact does the Silver CREST Award have on science scores and subject selection? was published on Wednesday 27 January by the British Science Association and launched by the Rt Hon. the Lord David Willetts at a reception at the House of Lords.

The report was written by volunteer economists who were matched to the BSA research project by Pro Bono Economics.

To request a press pass for the reception, please contact Amelia Perry on amelia.perry@britishscienceassociaiton.org or call 020 7019 4968.

• 1.About the authors

Rosie Stock Jones works as an economist at the Cabinet Office for the central Analysis and Insight team. She currently provides analytical support to the Office for Civil Society.

Tom Annable is also an economist for the Cabinet Office Analysis and Insight team and has worked there since 2014. Before that he obtained a Masters in Economics from the University of Warwick, where he specialised in Labour Market Econometrics.

Zoë Billingham is an economist in the Treasury working on EU reform, having previously worked in the Deputy Prime Minister's Office on social mobility.

Cee MacDonald loves charts and chickpeas. She has worked as an economist for the Department of Work and Pensions and Cabinet Office and is currently based at Defra.

• 2.About the research

This analysis focuses on students in English state schools aged 14-16 who took part in CREST Silver Awards between 2010 and 2013. The report addresses three research questions:

• 1.What are the characteristics of students taking Silver CREST Awards?

• 2.Does participation in the Silver CREST Award programme have an impact on attainment in science subjects at GCSE level?

• 3.Does participation in the Silver CREST Award impact on the likelihood of taking a STEM AS level?

Methodology

In order to conduct this research, data collected by the BSA on students starting Silver CREST Awards between 2010 and 2013, was linked to data in the National Pupil Database. This enabled us to bring together information on CREST participants with pupil characteristics, attainment and subject selection data. The total sample sizes for this analysis were 2.4 million students at Key Stage 4 (KS4), of whom 3,800 undertook a Silver CREST Award and 1.0 million at Key Stage 5 (KS5), of whom 2,300 participated in Silver CREST. The number of Silver CREST students in the sample who had received Free School Meals was 380 at KS4 and 190 at KS5.

Propensity Score Matching was used to create a control group of students who did not take part in CREST, but had similar characteristics to those who did. Key Stage 2 (KS2) SATs results were used to control for prior attainment. There is a strong relationship between KS2 attainment and GCSE attainment. However, these exams were taken around five years prior to GCSEs and A level choice, and do not give a detailed picture of attainment. Other control variables included gender, ethnicity, region of school, year of GCSEs, participation in triple award science, type of school, free school meal status in the six years prior to taking GCSEs and the Income Deprivation Affecting Children Index (IDACI).

The highest science point score at GCSE was chosen as the outcome variable for research question two (impact on attainment). This is a pre-existing variable in the NPD and helped to remove difficulties comparing performance across differing numbers of science GCSEs taken. For research question three (STEM take-up at KS5), we created binary indicator for whether AS-level students had selected a STEM AS level subject. The main science, maths, technology and engineering subjects were considered qualifying choices. Analysis for this research question was restricted to non-vocational AS levels.

Results

In regards to the first research question, we found that students taking Silver CREST were broadly representative of the wider pupil population in terms of gender and ethnicity. However, the CREST students were substantially less likely to have been eligible for free school meals (10% versus 22%) or have special educational needs (7% versus 16%) than the wider pupil population. Further, CREST students were more likely to have achieved stronger results at KS2 across all subjects. Their higher attainment was also reflected in their results at GCSE; students who undertook Silver CREST had higher average GCSE grades than those who did not do CREST.

The second research question looked at the impact of Silver CREST Award participation on GCSE science results. We found that students who took CREST achieved half a grade higher on their best science GCSE result, compared to a statistically matched control group. We looked at a subset of students who had been eligible for free school meals (FSM) in the six years prior to their GCSEs. The CREST students eligible for FSM saw a larger increase in their best GCSE science score (two thirds of a grade) compared to a matched control group of other students who were also eligible for FSM. These results were statistically significant.

Finally, we looked at the subject choices students made in their AS levels. 82% of CREST students took a STEM AS level compared to 68% of a statistically matched control group. CREST students were therefore 21% (or 14 percentage points) more likely to take a STEM AS level than students in the control group. For students who had been eligible for free school meals this difference was larger (38% or 21 percentage points). These results were statistically significant.

Discussion and recommendations

To demonstrate causality i) the treatment must have occurred before the outcome, ii) there must be a demonstrable statistical link between the presence of the treatment and the outcome, and iii) there must be no alternative plausible explanation. This research satisfies the first condition for the majority of students (a small proportion are likely to have taken CREST after their GCSEs and AS level choices). The second condition is satisfied, but not the third. We cannot rule out the possibility that other unobserved variables are affecting GCSE results and AS level subject choice. For instance, the motivation and enthusiasm of students and teachers for science are likely to have an effect, yet we have not been able to control for these. There may also be selection bias into Silver CREST if only high achieving students are permitted to participate, or if the Award only appeals to very bright students. Further research would be required to determine whether CREST has a causal effect on GCSE science attainment and AS level subject choice.

We make several recommendations for further work, including replicating this analysis through a Randomised Control Trial, broadening it to cover Discovery, Bronze and Gold Award types and conducting a cost-benefit analysis for schools. Additionally, we make three broader recommendations: that charities ensure accurate and usable data collection, that young people consider taking part in project/inquiry-based learning such as CREST and finally that the BSA consider targeting CREST at students from low income families.

• 3.About the BSA

Established in 1831, the BSA is a registered charity that organises major initiatives across the UK, including British Science Week, the annual British Science Festival, regional and local events, the CREST Awards and other programmes for young people in schools and colleges. The BSA also organises specific activities for professional science communicators, including a specialist conference and training. For more information, please visit: www.britishscienceassociation.org

The British Science Association (BSA) believes that science should be part of – rather than set apart from – society and culture, and is owned by the wider community. Our programmes encourage people of all ages and backgrounds to engage with science, become ambassadors for science, and ultimately to be empowered to challenge and influence British science - whether they work in science or not.

• 4.About CREST

CREST Awards is a UK awards scheme that recognises success, and enables students to build their skills and demonstrate personal achievement in project work. It offers educators an easy-to-run framework for curriculum enhancement and is student-led, which means that young people take ownership of their projects and choose to undertake them in areas they enjoy or see as relevant. For more information, please visit: www.britishscienceassociation.org/CREST

• 5.About Pro Bono Economics

Pro Bono Economics (PBE) matches volunteer economists with charities who need their expertise. Set up in 2009 by Martin Brookes (Tomorrow's People) and Andy Haldane (Bank of England), PBE has been approached by over 200 charities and has more than 400 economist volunteers on its books. Its work to date has extended across a range of areas, from the costs of reoffending to the impact on society of adult innumeracy, and from the financial burden to the UK of eating disorders to economic analysis on homelessness and the housing crisis.

PBE is supported by high-profile economists, including the Chief Economist at the Bank of England, Andy Haldane, and the Chief Economist at HM Treasury, Dave Ramsden as trustees, and Kate Barker, Jim O'Neill, Robert Peston, Martin Wolf, Gus O'Donnell and Adair Turner as patrons. Spring 2016 will see Diane Coyle join the Board and the appointment of a new CEO.

PBE projects and reports can be found here: www.probonoeconomics.com/projects-reports

To apply as a volunteer or as a charity, follow the link here: www.probonoeconomics.com/applications