Background
Key themes
Overview of findings
Detailed findings
Final summary and recommendations
Next steps
 

Background

Following the controversies over the awarding of qualifications in Scotland in 2020, the Scottish Government commissioned Professor Mark Priestley of the University of Stirling to lead an independent review of the awarding process. Beginning in mid-August, an interim report was made to the Deputy First Minister on 15 September, with completion of the final report on 30 September.

The review has been compiled using the collection of primary data and review of secondary data. The NASUWT was heavily involved in panel discussions as part of data-gathering from key stakeholders, which also included young people and parents, teachers, senior school leaders, local authorities (LAs), the Scottish Qualifications Authority (SQA), and government officials.

This briefing encapsulates the issues that affected the 2020 qualifications, as well as a summary of its conclusions and recommendations, some of which have already been put into place by the Scottish Government.

Key themes

The issues identified are split into key themes. These are:

• estimation and local moderation;
• national moderation;
• appeals;
• equalities issues;
• communications;
• impact on young people and their families; and
• impact on teachers and lecturers.

Overview of findings

The report notes there were three principles which the SQA followed when developing the Alternative Certification Model (ACM). These were fairness to all learners, safe and secure certification of qualifications (while following the latest public health advice), and maintaining the integrity and credibility of the qualifications system, ensuring that standards are maintained over time, in the interests of learners.

There has been little criticism of the fundamentals of these principles. This said, many stakeholders have, during the review process, questioned whether in developing the ACM, an overemphasis on the third principle, maintaining the integrity and credibility of the qualifications system, has had a detrimental impact on any focus on the first, fairness to all learners. It should be noted that this concern has only become apparent with hindsight, rather than being raised at the time of the development of the ACM.

There is a general recognition that the SQA faced an unenviable task in developing an assessment process against a backdrop of challenging timescales, while operating within an unprecedented situation. It has also been widely accepted that there is no system that would be perfect and would mitigate every potential problem. This does not, however, overshadow the multiple issues that have been identified in this review.

Detailed findings

Estimation and local moderation

The evidence compiled for the review shows very clearly that the estimation process was taken seriously by schools and involved the highest levels of professional integrity, dedication and hard work. There is further recognition that this activity was completed away from their usual workplaces, with many practitioners encountering difficulties in evidencing their estimations. The review cites two difficulties:

1. different approaches to progression from subject to subject made a consistent approach across centres problematic; and
2. problems with physically accessing evidence, particularly coursework, due to it being on school premises and therefore unable to be retrieved, or it had already been sent to the SQA.

Local authority evidence presented to the review suggested that some centres over-estimated, not due to deliberate grade inflation, but rather a consequence of an inability to complete robust moderation for reasons including workload, a lack of LA capacity/expertise and a lack of evidence. Some over-estimation may also have been a result of a desire to assess how each individual would have performed had the examination gone ahead as planned. The review has found no evidence of accountability systems that have led to grade inflation.

Looking specifically at local moderation of the estimation process, the role of LAs appears to be crucial. This said, there are nearly 100 centres that are not within LA jurisdiction. The review has found evidence of highly variable approaches to local moderation which in some cases were excellent while other cases took a more minimal approach.

A final element in the review of the estimation process is the decision to cancel coursework. Many stakeholders have expressed that a greater effort could have been made to assess coursework, which would have allowed it to contribute to final grades and offer a more robust evidence base for estimation. However, the rapid review accepts that the SQA made a rational decision to cancel coursework, based on reasons of equity, logistics, and safety concerns.

National moderation

The moderation of centre estimates was a part of the ACM developed by the SQA. While the review does not question the need for some form of moderation of estimates, it does question the approach and assess its shortcomings. These largely arise from the fact the moderation was primarily based on a quantitative approach using historical attainment data for each centre. There was no engagement with centres and/or LAs to enable a qualitative element in the moderation, which would have allowed an understanding of cases where there was variance from historical attainment.

While the review agrees with the SQA’s position that it would not have been possible to engage in dialogue at a centre level, it does recognised that two approaches could have been made to mitigate some of the fallout from the national moderation process:

• analysis of data to identify anomalies, drawing on government and local government expertise in statistics; and
• dialogue with LAs to discuss and moderate in a qualitative sense (for example, engaging with the rationales for cohort variance collected by LAs).

Post-certification reviews and appeals

The original post-ertification review (PCR) process was technically appropriate, offering a review of individual candidate evidence. Furthermore, it was free of charge and the priority PCR process was designed to expedite the needs of students whose university offers were dependent on their grades.

However, during the process of awarding qualifications in Scotland for 2020, there were a number of issues that arose around the use of the PCR. The PCR was widely perceived as an appeals process, rather than an integral part of the awarding process. The SQA worsened this by not publishing details of the statistical moderation process and its likely implications. This therefore meant the SQA did not clearly communicate the integral role the PCR would play in the 2020 ACM.

Had there been clear messaging set out that, given the unique circumstances of 2020, not only would a large-scale appeals process be needed, it was in many ways unavoidable to address the issues of students being penalised unfairly.

Further to this, a secondary problem then arose once the decision had been made to honour centre estimates. The decision to exclude academic judgement from the revised appeals process has removed the option for students to access the appeals process where estimates were believed to be inaccurate. Instead, schools had to accept the right to appeal on the grounds of bias or discrimination in the original decision. Not only does this place schools at risk of further action such as litigation, it may also create conditions where it is not in the school’s interest to pursue appeals and, therefore, they are declined.

The review has seen significant levels of evidence that suggest the situation is severely damaging relations between schools and parents.

Equalities

The review has concluded that equalities issues were considered at various stages of the ACM’s development. These include discussions relating to bias in the estimation process as early as March, as well as the delivery of unconscious bias training to centres.

However, it seems that this initial focus on equalities work was on the area of bias in assessment, with less consideration given to how the moderation process itself might produce inequality. There was also little evidence found to suggest that equalities issues were systematically considered or built into the development of the ACM from the onset. Compounding this, the Equalities Impact Assessment (EIA) and Child Rights Impact Assessment (CRIA) were produced very late in the process.

This is particularly disappointing to the NASUWT, given that the Union pressed the SQA from the very start of the process to publish the details of any EIA. This was particularly in respect of the extent to which equalities issues were taken into effective consideration throughout the design and implementation of the moderation process for 2019/20.

The review has recognised and supported the NASUWT’s concern that is it difficult to understand how decisions were taken in the absence of any completed EIA.

Other issues over equality have been identified as part of the review process, including the lack of access to equalities data by the SQA. The SQA does not routinely collect equality data, which meant it could not truly examine the 2020 approach to identify the possible impact on protected characteristics.

This has ultimately led to a situation where some of the impacts of the moderation model were not fully predicted or mitigated. More focus was seemingly placed on debating whether the ACM advantaged or disadvantaged cohorts from schools in socially and economically disadvantaged areas.

Communications

Although there is a specific section looking at the issues over the SQA’s communication and transparency, the concerns around this are a common theme running throughout the review.

While the review identifies that the SQA invested considerable resources in communicating key messages, there were many aspects in which this communication was lacking.

Specific concerns include:

• SQA communication throughout the process not being clear or comprehensive;
• Important information being distributed on Friday evenings, after schools had closed;
• these then often generated high numbers of parent queries over the weekend; and
• young people described SQA communications as ambiguous, unclear and inconsistent.

This encapsulates an issue that the NASUWT has consistently raised: that the lack of collaboration and engagement from the SQA is problematic. The Union had hoped, given the controversies caused by the decisions made in 2020, that the SQA would recognise the need to become more inclusive of other stakeholders. Given the exclusion of the NASUWT from the SQA’s National Qualifications 2021 Group, early signs suggest that the SQA and the Scottish Government are continuing with their selective approach to engagement from stakeholders.

The review categorically states that no single organisation could possibly have developed the best response to the challenge of awarding qualifications in such an unprecedented situation. To deliver this, participative planning and decision-making, drawing on collective expertise, were necessary.

Impact on young people

The review notes the importance of understanding the impact of the cancellation of exams on young people.

Following the announcement of the cancellation of examinations, young people reported feeling anxious and upset. There was confusion over how the grades would be awarded, which was then further fuelled by the decision to cancel all coursework. Some students whose schools had already submitted coursework to the SQA for marking had no access to it for evidence.

Young people described inconsistent approaches to applying the ACM at school level. There are variations noted between subjects, with traditional academic subjects being more rigorous in their estimates than arts-based subjects.

Some students were told their estimated grades or given an indication of a grade band, while others were told this was not permissible.

Looking at equity, many students felt that extenuating circumstances were not taken into account during estimation. Examples include extended periods of illness around the time of the prelims, or young carers not having generated as much evidence for estimated grades subsequently being disadvantaged.

Lack of access to technology for some students also created barriers to producing evidence that was comparable to their more fortunate peers.

As well as young people themselves, parents reported the negative effects of the situation on the wellbeing on their children, especially widespread anxiety. For young people with Additional Support Needs (ASN), these pressures have been felt even more significantly.

There is a general feeling that the SQA did not consider the impact the ACM would have on young people’s future direction, including university offers and college places. Some students, who received results lower than expected, changed their university courses in line with this. For those students who went through clearing, following poorer than expected results, they were unable to revert back to their original course choice following the reversion to teacher estimates. This has altered their study and career path.

The review captures the approach that young people would like to see for future exam arrangements. Most notably are a direct appeals process available to individuals in 2021 and a move towards continuous assessment rather than the pressurised two-term examinations.

Impact on teachers and lecturers

The impact on teachers and lecturers is referenced throughout the review. The evidence gathered as part of the review process clearly points to teachers having been subjected to high levels of stress and anxiety, as well as feeling undermined and denigrated.

A significant part of this stems from the lack of clarity by the SQA over the balance between estimation and moderation. Had the message been clearer about the reasons for the use of national moderation and the intention behind it, teachers may not have felt that their judgement and estimates were not trusted.

The SQA has stated that their decision not to share detailed information about the ACM (in particular, the need for moderation and the need for an integral PCR) was to avoid undue stress for teachers, as well as students and parents/carers. This was clearly a misjudgement. In fact, it had the opposite effect.

Furthermore, teachers were left confused and uncertain over the guidance on estimates and local moderation. The review notes that a more comprehensive set of guidance around this would have been helpful and potentially removed some of the complexity from the system which led to teachers feeling under excessive pressure and suffering from stress.

Ultimately, some of the decisions made by the SQA have led to an erosion of trust and confidence in the organisation amongst teachers.

Final summary and recommendations

Final summary

The review is very clear that while it makes critical observations, it is not the panel’s intention to apportion blame. The review is an opportunity to offer constructive criticism to underpin future responses to the challenges caused by the ongoing Covid-19 pandemic.

Notwithstanding this, the review does identify a number of decisions that could have been taken differently and had they been, may have led to different outcomes and prevented the negative situation that ensued. These decisions relate to:

• better partnership working in the early stages to develop the ACM. The review notes such working was on offer (this includes from the NASUWT), but it was not taken up by the SQA;
• greater transparency around the moderation system and its implications, which again was consistently called for by stakeholders;
• a clear explanation of the PCR as an integral part of the awarding process, rather than the bolt-on appeals process it is normally used as;
• a greater level of integration of EIAs in the development of awards system, which was ignored despite repeated requests from the NASUWT;
• greater levels of co-operation between agencies, including between the SQA and the Scottish Government, for many reasons, including the analysis of data and national moderation; and
• more formal and organised engagement with young people, to inform the development of systems.

The review also offers observations about the future of qualifications, beyond the remit of this initial report. There is recognition of widespread support across all the stakeholder groups who engaged in the review process for a fundamental rethink of the long-term approach to awarding qualifications. In particular, there is an appetite amongst stakeholders, including young people and their parents, to reduce emphasis on terminal examinations as the basis for qualifications, with a shift towards continuous assessment.

Recommendations

1. Suspension of the National 5 examinations diet in 2021, with qualifications awarded on the basis of centre estimation based upon validated assessments.

2. The development of a nationally recognised, fully transparent and proportionate system for moderation of centre-based assessment.

3. The development of more extensive approaches to collaborative decision-making and co-construction by professional stakeholders of assessment practices related to National Qualifications.

4. A commitment to embedding equalities in all aspects of the development of qualifications systems.

5. The development of more systematic processes for working with and engaging young people, as stakeholders and rights holders in education.

6. The development of a clear communications strategy, co-constructed with stakeholders, to ensure that the extraordinary arrangements for 2021 are as fully as possible understood by all parties.

7. A review of qualification appeals systems, including consideration of the rights and roles of young people, in the context of the incorporation of the United Nations Convention on the Rights of the Child (UNCRC) into Scottish law.

8. The commissioning of independent research into the development and application of the 2020 ACM, involving full access to anonymised attainment data and the statistical algorithms used to moderate grades.

9. The development by the SQA and partners of digital materials and systems for producing, assessing and moderating assessment evidence, to ensure that operational processes for gathering candidate evidence for appeals is less reliant on paper-based systems.

These recommendations have all, with the exception of one, been accepted by the Scottish Government. The response is published on the Government website.

The anomaly is recommendation 8, which the Scottish Government notes, but has responded that it will be considered by the team that assesses a range of potential projects for inclusion in our research in education strategy.

Following the publication of the review, on 7 October, the Deputy First Minister announced that National 5 examinations would be cancelled for 2021 and replaced by a system where grades are awarded based on coursework and teacher judgement.

The full details of this are yet to be finalised and announced. The NASUWT notes there are still important issues to be addressed, such as those concerning teacher workload.

Next steps

The NASUWT will continue to monitor further developments related to the awarding of qualifications in Scotland, with a focus in the first instance on the 2020/21 cohort. The Union will pay specific attention to whether the recommendations accepted by the Scottish Government are being progressed and adopted.

The NASUWT will provide further updates to members as appropriate.

The Union will also continue to press the SQA to involve the NASUWT in its planning and preparation for awarding qualifications in 2020/21 and future cohorts.