Introduction
The Programme for International Student Assessment (PISA)
What are the benefits of PISA?
What are the concerns about PISA?
Trends in Mathematics and Science Study (TIMSS) and Programme in International Reading Literacy Study (PIRLS)
NASUWT’s position on PISA TIMSS and PIRLS
Further Information

Introduction

PISA, TIMSS and PIRLS are three international assessments of student performance. We provide is a brief explanation of each study and consider the benefits and limitations of the studies. We also set out NASUWT’s position on these studies. Use the links in the Further Information section to look at survey reports or find out more about the studies.

The Programme for International Student Assessment (PISA)

PISA is a standardised assessment of 15 years olds administered by the Organisation for Economic Co-operation and Development (OECD). It ran for the first time in 2000 and is run every three years.

The number of countries/jurisdictions participating in PISA has grown from 43 in 2000 to 81 in 2022.

PISA assesses students’ performance in reading literacy, maths and science. Each round of PISA focuses on one of the three areas as the major domain. The OECD says that PISA tests students’ ability to apply what they have learned to everyday situations.

Jurisdictions participating in PISA must adhere to strict criteria to ensure that the survey is representative of the student population for that jurisdiction.

Students selected to participate in PISA take computer based tests and complete a questionnaire about themselves, including their attitudes, dispositions and beliefs, their homes and their school and learning experiences.

Headteachers from the sample schools also complete a questionnaire which asks about school management and organisation and the school learning environment.

Jurisdictions can also opt to administer further questionnaires. These include a teacher questionnaire, a parents’ questionnaire and further questionnaires for students.

Jurisdiction test scores are standardised on a single scale with a mean of 500 and standard deviation of 100. Student performance is also reported through the use of proficiency scores on a seven-point scale ranging from 1b to 6.

The OECD considers level 2 to be the baseline of proficiency at which students demonstrate the knowledge and skills needed for adult life while a score at level 4 or above indicates strong performance. This information is used to indicate the spread of student performance within the jurisdiction.

The OECD publishes a number of reports following each PISA survey. These look at student performance across participating jurisdictions as well as factors such as the relationship between student performance and social background and education resources, policies and practice. Jurisdictions also produce ‘country’ reports and separate reports are published for England, Wales, Scotland and Northern Ireland.

What are the benefits of PISA?

PISA provides a wealth of data and this enables broad comparisons to be made between education systems. The fact that PISA results are standardised means that it is also possible to consider a system’s performance over time.

Data relating to student background enables comparisons between the performance of different groups of students, e.g. by socioeconomic background, by gender and by immigrant background. It is also possible to examine the relationship between education policies and student performance.

This means that PISA can be useful for when thinking about policy priorities or when considering what steps could be taken to address policy challenges.

PISA results may be useful for identifying common issues across jurisdictions. For instance, in all jurisdictions, students for disadvantaged backgrounds perform less well than their peers although some systems appear to have lower levels of inequity than others. Also, results from PISA 2022 find that while student performance across all education systems was adversely affected by the COVID-19 pandemic, some jurisdictions appear to be more ‘resilient’ with smaller declines in performance.

Therefore, PISA results can provide the basis for looking at what systems are doing in order to consider strategies that could be more effective.

What are the concerns about PISA?

While PISA data can be used to identify associations between factors, it does not provide causal explanations. Further research would be needed in order to make judgements about cause and effect.

A major criticism of PISA is that it is overtly political in nature and the reporting of results in the form of league tables encourages this practice. In particular, when PISA reports are published, politicians and the media often focus on a jurisdiction’s position in the league tables, using this to either claim that a high ranking is due to the government’s education policies or to condemn the education system and argue for reform. But as has already been stated, PISA does not explain cause and effect. Also, a higher position in the performance table does not necessarily equate to an improvement in performance; performance could have stagnated or even declined as a jurisdiction’s position is relative to the performance of other jurisdictions. Further, a jurisdiction’s score may be subject to sampling error so comparisons of performance need to focus on statistical significance rather than actual scores. However, a key question to ask is is PISA testing what is important?

While the OECD has taken some steps to address the concerns about performance tables, for example through reporting on issues such as equality and equity and students wellbeing, the main focus of PISA remains on the comparison of students’ performance in the reading literacy, maths and science tests.

Critics express concern that PISA focuses on reading literacy, maths and science and argue that insufficient attention is paid to other aspects of the school curriculum. It is worth noting that the OECD does seek to assess students’ performance in broader competencies although these are often available as optional additional assessments. For instance, PISA 2022 included an additional optional assessment covering creative thinking and global competency was an additional option in PISA 2018. Financial literacy is also an optional assessment in each round of PISA, however these surveys are optional. In the UK, only Scotland participated in the survey on global competency and to date, none of the UK administrations have participated in the financial education option.

The OECD claims that PISA tests the knowledge and skills that are essential for full participation in society. However, critics argue that a test lasting two hours is unlikely to assess these things. They also argue that PISA tests are culturally biased as ‘real life’ is different in different countries. Further, concerns are raised that PISA is likely to lead to the standardisation of education internationally.

More generally, some argue that students’ familiarity with the types of questions asked in PISA will affect the test results. Students who are used to answering PISA-style questions and who regularly sit tests are likely to perform better than students who are not familiar with such questions or test environments.

Trends in Mathematics and Science Study (TIMSS) and Programme in International Reading Literacy Study (PIRLS)

TIMSS assesses the performance of students aged 9-10 years and 13-14 years in maths and science. PIRLS assesses the reading literacy of students aged 9-10 years. Both assessments are managed by the International Association for the Evaluation of Educational Achievement (IEA). TIMSS was first conducted in 1995 and PIRLS was first held in 2001.

TIMSS is run every 4 years and PIRLS every 5 years. Sixty-four countries and eight benchmarking systems participated in TIMSS 2019, while fifty-seven countries and eight benchmarking systems participated in PIRLS 2021. In the UK, only England and Northern Ireland participate in TIMSS (Northern Ireland only participates in the survey for 9-10 year olds) and PIRLS.

TIMSS and PIRLS results are scaled using 500 as the mean average for countries participating in the original survey. Scores are also anchored using benchmarks to denote different levels of performance (advanced, high level, intermediate and low). This allows for an examination of the distribution of students performing at different levels in each jurisdiction as well as an examination of jurisdiction performance over time.

Both surveys issue contextual questionnaires to students, headteachers, class teachers and parents. These seek information about factors such as student background, school resources and student attitudes and behaviours.

TIMSS and PIRLS differ from PISA in that the surveys are drawn from the curricula of participating countries. The report for each survey focuses on country performance but also addresses areas such as home-socioeconomic status, school resources and trends.

Many of the benefits and concerns raised about PISA also apply to TIMSS and PIRLS.

NASUWT’s position on PISA TIMSS and PIRLS

NASUWT’s view is that PISA, TIMSS and PIRLS provide a wealth of evidence about students, schools and schooling internationally, but the results need to be treated with care.

We condemn the misappropriation of results in support of ideologically driven education policy reform.

International assessment data can provide pointers to things that warrant further examination.

Context is critical which means that there is a need to look beyond the data for explanations.

PISA, TIMSS and PIRLS are most useful as tools for policy reflection and policy learning. Crucially there is a need to recognise their limitations and consider the extent to which they are actually assessing what is important.

Further information