When UN member states adopted 17 Sustainable Development Goals (SDGs) for 2030, there was little discussion about how to measure progress toward these goals. As we enter the Decade of Action, deciding on a measurement strategy for all SDGs and their targets has become a pressing issue.
International Large-Scale Assessments (ILSAs) in Education such as Programme for International Student Assessment (PISA) or Trends in International Mathematics and Science Study (TIMSS) were considered by many the most suitable source of information to measure and monitor progress towards several of these goals. ILSAs offer systematic information from more than 100 education systems, provide high coverage of the indicators included in the SDGs framework and have unrivalled data quality assurance mechanisms.
The scale of ILSAs is a strength and weakness. As national participation in ILSAs has grown (from just 12 countries in 1964, to more than 100 in 2018), it has become more difficult to produce information that is strictly comparable across countries. For example, constructs like socioeconomic status are traditionally estimated using items such as parental education, parental occupational status or household items. It is not clear, however, to what extent an undergraduate degree, being a shop owner, or having a TV mean the same in the UK than in Chile or Nigeria. Thus, we do not know whether the resulting ‘socioeconomic status’ marker can be validly compared across countries. This is of fundamental importance for researchers and policymakers in ensuring the fairness of comparison across countries.
Two strands of research
Research led by Dr Sandoval-Hernandez at the University of Bath has focused on improving the use of ILSAs to measure and monitor SDGs by conducting research into two strands. A technical strand focusing on the cross-cultural comparability of ILSA data; and an operational strand focusing on the use of ILSAs to measure and monitor the achievement of SDG Target 4.7 (which relates to the contribution of education to sustainability, citizenship and social justice).
Proposing an initial measurement strategy
Dr Sandoval-Hernandez’s work on the technical strand started with an invitation by the Director of the Global Education Monitoring Report (GEMR), to write a Background Paper for the GEMR 2019. The GEMR is an editorially independent, authoritative and evidence-based report published by UNESCO. Its purpose is to monitor the progress made towards education targets set out in the Sustainable Development Goals framework. Its influence is far reaching, from regional, national and international policy-makers in education and finance, to teachers, NGO’s, aid organisations and more.
The main objective of the report was to evaluate the extent to which ILSAs can be used to monitor specific aspects of the Sustainable Development Goals, in particular, SDG 4.7. The report concluded that there is enough evidence to consider a good fit between the concepts evaluated in one of the ILSAs (International Civic and Citizenship Education Study) and those included in SDG 4.7, and proposed an initial measurement strategy. A key limitation of this strategy was, however, the lack of evidence to evaluate the cross-cultural comparability of the data used to operationalise these concepts.
Building on these results, in 2019 and 2020 respectively, Dr Sandoval-Hernandez published a paper and a book chapter (in Spanish and English) that outlined the development of a systematic procedure to evaluate measurement invariance (also known as cross-cultural comparability) with ILSA data. This, along with his previous publications led to Dr Sandoval-Hernandez receiving an invitation to be part of the Technical Advisory Groups of two regional ILSAs: the UNESCO Regional Comparative and Explanatory Study (ERCE), and the UNICEF’s Southeast Asia Primary Learning Metrics (SEA-PLM).
Influencing key Steering Committees
Through his advisory role, Dr Sandoval-Hernandez influenced the Steering Committees of these two assessments so that they could adopt the concepts and procedures he had developed. The method is now being implemented and reported by both groups, improving validity of data provided by these ILSAs. Data is now received in a testable manner so that policy-makers and researchers can be confident in the fairness and comparison across countries. Dr Sandoval-Hernandez’s research work on this matter is highly cited in a Technical Report presented to the UNESCO Regional Comparative and Explanatory Study.
Additional reports as part of the operational research
The operational strand, focused on the use of ILSAs to measure and monitor the achievement of SDG Target 4.7. The interest generated by the GEMR Background paper and the advisory role of Dr Sandoval-Hernandez for ERCE abd SEA-PLM, led to the UNESCO Institute for Statistics commissioning him for two reports.
The first report commissioned by UNESCO developed a proposal for an improved Measurement Strategy for Thematic Indicators under Target 4.7 using ILSA data. This measurement strategy was taken to the 6th meeting of the Global Alliance to Monitor Learning in 2019. The attendees to this meeting, which included high profile educational policymakers from countries representing all regions, and international organisations such as the European Commission, the World Bank, UNICEF, OECD and the Global Partnership for Education voted Dr Sandoval-Hernandez’s proposal as the best option to measure targets 4.7.4 and 4.7.5.
The support of the international community led to the second report, that implemented this strategy to produce a set of scores that refer to knowledge about global citizenship and sustainability and attitudes/behaviours towards gender equality, multiculturality, human rights, freedom of expression, social justice, etc. and is available through UNESCO’s official data repository.
Dr Sandoval-Hernandez's measurement and monitoring strategy is now being used by all the UN member states.
