The Programme for International Student Assessment (PISA) is, as its name suggests, an international survey that assesses students’ performance. The Survey, which is run in three-year cycles, assesses the performance of 15 year olds in reading literacy, maths and science. It also covers issues such as students’ backgrounds and characteristics and indicators about schools.
PISA is administered by the Organisation for Economic Co-operation and Development (OECD). It ran for the first time in 2000. The number of countries/economies participating in PISA has grown since then and more than 70 countries/economies now participate in the survey.
What PISA Assesses
The OECD states that PISA:
- aims to evaluate education systems by testing the knowledge and skills of 15 year olds; and
- focuses on the knowledge and skills that are needed for full participation in society.
The primary purpose of the Survey is to determine the extent to which students can apply their knowledge and skills in reading, maths and science to the challenges that they will encounter in adult life.
How PISA Assessments Work
PISA is a sample survey. A minimum of 150 schools in each country/economy participate in the study. Generally, between 4,500 and 10,000 students in each country/economy will take the test.
Students from England, Wales, Scotland and Northern Ireland are included in the UK sample. The proportion of students included from Wales, Scotland and Northern Ireland is higher than that for England. This enables individual reports to be produced for each.
Whilst PISA assesses performance in reading literacy, maths and science, only one of the domains is examined in detail in each cycle. For example, in 2006 the major domain was science, in 2009 it was reading literacy and in 2012, maths.
Students are required to sit a series of tests and complete a questionnaire. The questionnaire asks for information about the student and the school, including, for example, behaviour and discipline at the school. Parents are also asked to complete a questionnaire which covers areas related to socioeconomic attitudes to education and learning, and education resources at home.
The headteacher of each school participating in the survey completes a questionnaire about the school, including information about teaching staff and school climate. PISA does not survey teachers. However, a teacher questionnaire will be included in the survey from 2015.
How the Results of PISA Surveys are Reported
PISA results are reported in several different ways. The most widely publicised and controversial is the publication of results in performance tables. Other PISA reports look at the relationship between students’ performance and social background, students’ attitudes and approaches to learning, and the relationship between students’ performance and education resources, policies and practice.
Country/economy test scores are standardised on a single scale with a mean of 500 and standard deviation of 100. Standardisation takes place when a subject is the major domain. The results for reading literacy were standardised in 2000, maths in 2003 and science in 2006.
Student performance is also reported through the use of proficiency scores on a seven-point scale ranging from 1b to 6. The OECD considers level 2 to be the baseline of proficiency at which students demonstrate the knowledge and skills needed for adult life. A score at level 4 or above is said to indicate strong performance. This information is used to indicate the spread of student performance within a country/economy. Combined with data about students’ background, this provides information about the relationship between performance and socioeconomic status.
The Main Concerns About PISA
There are many criticisms of PISA. These include criticism about the educational and cultural limitations of the survey, issues about the survey’s methodological limitations, questions about what PISA actually measures and concerns about political influence and interference.
Critics argue that PISA only focuses on reading literacy, maths and science and pays little attention to other aspects of the school curriculum. They also argue that the skills that are being assessed may not be commensurate with the educational goals and ethos of some countries. Further, they argue that the need for questions to be relevant across cultures and education contexts limits the questions that can be included and risks creating a ‘one size fits all’ assessment that is not a good match for any single system.
Whilst the OECD claims that PISA tests the knowledge and skills that are essential for full participation in society, critics argue that a test lasting just two hours and completed using a pencil and paper is unlikely to assess these things. They also argue that ‘real life’ will be different in different countries and that this cultural bias will disadvantage students from some countries. Some suggest that PISA will lead to a greater standardisation of education internationally. A particular concern is that there will be greater standardisation with those countries that have the greatest influence over the OECD and its work priorities (e.g. the USA, the UK and Australia).
Some argue that the format of PISA tests and students’ familiarity with the types of question will affect the test results. Students who are used to answering PISA-style questions and who regularly sit tests are likely to perform better than students who are not familiar with such questions or test environments.
A major criticism of PISA is that it is overtly political in nature and the reporting of students’ results in the form of league tables encourages this practice, distorting education policies and priorities. For example, some governments seek to introduce policies that will improve performance in PISA (effectively teaching to the PISA test). Governments also use performance in PISA to justify the need for policy change and then ‘cherry pick’ policies from ‘high.performing’ systems that reflect their ideological position.
The Use of International Assessments
The NASUWT believes that evidence about students’ performance in international assessments, including evidence from PISA, can be very useful. For example, it enables policy makers and others to draw comparisons between education systems. It also enables them to examine elements of different education systems and consider the reasons for common problems, as well as why some policies are effective. The evidence also provides opportunities for policy makers to reflect on the implications of these analyses for education (and social and economic) policy and practice.
The key point is that evidence from international assessments such as PISA is useful as a tool for examining education policy and practice.
However, this is very different from using the evidence to say that an education system is better or worse than other systems. It is not appropriate to use international evidence in this way as the following example that compares the results of PISA with those of another international assessment, Trends in Mathematics and Science Study (TIMSS), illustrates:
TIMSS assesses the performance of students aged 9-10 years and 13-14 years in maths and science and uses questions that are based on national curricula. England participates in both PISA and TIMSS, as does New Zealand. In PISA 2009, New Zealand was one of the highest performing countries and performed significantly better than England in reading literacy, maths and science. However, in TIMSS 2007, New Zealand’s results were below the country mean in maths and just above the country mean in science and England’s results were significantly better than New Zealand’s in both maths and science. In other words, deciding to use PISA rather than TIMSS assessments to make a judgement about the quality of the education system could lead to very different conclusions about the system.
Education systems are about much more than what international assessments test, so it is not appropriate for governments to adopt strategies to simply improve performance in international assessments. The focus of policy reforms should be on agreed education priorities and on ensuring that policy is consistent with and contributes to the aims, purposes and values that underpin the education system. Policy should be based on a wide and rich range of evidence, including evidence from teachers and school leaders about effective practice and the issues that impact on learning and teaching. Of course, this may improve performance in international assessments such as PISA, and international assessments may be useful as one of the indicators that a policy is successful.
Welsh Government Guidance – A Guide to Using PISA as a Learning Context
The Welsh Government has produced a document for schools, A guide to using PISA as a learning context. The document provides guidance to teachers and school leaders on using PISA and PISA-style questions as activities to help support and improve learning and teaching. The document is intended to ‘help demonstrate how PISA, and PISA-style questions can support pedagogy and help improve learning and teaching’. It includes questions that are drawn from PISA materials published on the OECD website.
As well as the guide, the Welsh Government has established workshops and a development programme to ‘to ensure that there is an understanding in the educational community in Wales as to how PISA assessments work in terms of contexts, demand and structure and how they can be used to support improved learning and teaching’. The guidance also says that ‘Using PISA and PISA-style materials as part of everyday learning and teaching will better prepare learners for these type assessments and support a more expansive pedagogy for teachers, which develop deeper levels of questioning, problem solving and the use of literacy and numeracy in cross-curricular contexts.’
Concerns About the Welsh Government’s Guidance
The Welsh Government’s preoccupation with PISA-style questions focuses attention on a narrow range of knowledge and skills, rather than on the full range of knowledge and skills that pupils need to develop. The approach is a classic example of the tail wagging the dog. It is very clear that the main motivation is to improve Wales’ performance in PISA tests. The guidance, workshops and related development programme have been designed to help teachers teach to the PISA test.
The NASUWT believes that the guidance about using PISA-style questions to improve teaching and learning is likely to have the opposite effect. The guidance encourages schools to adopt an ‘initiatives’ approach rather than to think about how they should design a curriculum that covers the range of knowledge and skills that pupils need and how these things should best be assessed. Failing to look at the issue holistically or as part of a planned approach is likely to have significant implications for teachers’ workload.
The NASUWT has other concerns. For example, it is not clear that the skills identified in the guidance are the most important skills that pupils should develop. Questions must be raised as to whether PISA questions actually assess the skills that the OECD claims they assess and, equally, there is nothing to indicate that the approach to assessment that PISA-style questions encourage, is even appropriate.
Teachers and school leaders should not use A guide to using PISA as a learning context to identify PISA-style questions to include in their lessons. Instead they should ensure that the school curriculum addresses the range of knowledge and skills that pupils need to be successful, confident and effective members of society in the 21st century.
Teachers and school leaders should look for sources of support locally and nationally to design a curriculum that meets the needs of the pupils in the school. They should also identify support about the design of assessments that assess different forms of knowledge and skills.
Closely linked to these points, schools should ensure that all teachers have both an entitlement and access to high-quality continuing professional development (CPD). CPD should be interpreted broadly to include time for teachers to reflect critically on their practice, opportunities to work collaboratively, opportunities for teachers to obtain support, as well as opportunities to undertake external training and development.
Schools should use research and other evidence about best practice and the latest thinking to inform their plans and decisions about the curriculum and assessment.
The NASUWT acknowledges that the Welsh Government guidance documents, issued in light of the School Teacher Appraisal (Wales) Regulations 2011, make reference to a process of reflective practice to inform performance management. The NASUWT has warned that such a process has the potential to impact adversely on working time and working hours but recognises that an entitlement to CPD based on the broad interpretation described above could be a way forward.
Challenging the Inappropriate Use of PISA and Other International Assessments
The Welsh Government is not alone in focusing on the results of international assessments and seeking to use the results of assessments to justify policy reforms. Other governments are using PISA and other international assessments to justify policy change and selectively to promote a particular education ideology, cherry picking policies consistent with that ideology.
The NASUWT is challenging the inappropriate use of international evidence in a variety of forums and publicly.
Part of the NASUWT’s international work involves raising awareness of the problems caused by the inappropriate use of international assessments. This includes commissioning research to highlight the problems that occur.
The NASUWT is also working to improve the quality of international assessments. This includes work with the OECD on PISA. One result of this work is that PISA 2015 will include a questionnaire for teachers.