Key messages from a roundtable discussion in London, 17 November 2014This summary was submitted by Pauline Rose, Ed Barnett, Rob Whitby, Steve Packer and Susan Nicolai.
A roundtable discussion co-hosted by DFID, University of Cambridge, ODI and UKFIET* brought together technical experts to identify recommendations for indicators to track learning and equity post-2015. The discussions were positioned in the context of the Technical Advisory Group (TAG) to the Education For All Steering Committee (see consultation on education indicators) and the recently-released UN Data Revolution Report. The roundtable included over 40 participants from UK universities, think tanks, European donor agencies and NGOs. This note summarises the key messages from the day but does not attempt to fully represent the diversity of opinions shared.
Common themes throughout the discussions included:
- indicators are important to hold policymakers nationally and globally to account for agreed targets.
- recognition that the selection of indicators is both a technical and political process, but that ultimately indicators need to be technically-robust to be useful.
- the need to make sure concepts in the proposed targets are clearly defined for them to be measurable.
- the need to take a short-/medium-/and longer-term perspective to what can be measured.
- the importance of cooperation between agencies involved in data collection from an early stage.
- the need for education experts to engage with the broader data revolution process: to share, inform and learn from this process which is likely to be central to post-2015 monitoring.
Measuring learning and skills post-2015
Presentations reflected on the measurement of learning, focusing on Muscat target 2 and OWG target 4.1. A proposal for a common learning metric was presented and discussed. Lessons from measuring learning in the UK context were also discussed. The TAG document proposes that the aim should be for technically robust and globally comparable indicators whose measures are (1) valid and reliable across all countries and (2) assessed by a similar question or item. However, participants noted that, in reality, the measure of success should be “fitness for purpose” and not “perfection”.
Key messages on learning
- Measuring progress towards a global learning target can and should be based primarily on national valid and reliable assessment data.
- The primary purpose of such national assessments should be to inform policy, planning and practice within countries. Such assessments are also not a substitute for formative classroom assessments, which are vital for improving learning.
- Work on the common metric, drawing where possible on existing assessments, should be prioritised and efforts made to publicise and get feedback on the proposal. The common metric proposal should be made available online alongside the existing TAG consultation document.
- Learning from the UK: high-stakes learning assessments can lead to grade inflation, teaching to the test and a loss in credibility of the assessment as an objective measure of learning. There are benefits in having a combination of such assessments along side international ones (such as PISA) to allow triangulation, and so overcome manipulation that can happen with either type of assessment on their own.
- Equity must be a central concern in all efforts to track learning outcomes. This requires consideration of the age or grade of assessment, given that learning inequalities start at an early age. In the UK, for example, inequalities are so entrenched by secondary schooling that policies at this stage have had limited impact.
- To overcome inequality, priority should be given to indicators that are comparable across countries, in particular basic reading in the language of instruction and numeracy skills. These are important both in their own right, and as a means to learning other skills.
- Learning should be tracked globally at an age by which children would be expected to have spent some time in school (for example equivalent to grade 3), and by the time they should have completed primary school.
Key messages on skills
- Beyond literacy and numeracy, one possible area to focus on for cross-country comparability more immediately could be ICT skills. It is unlikely to be possible to measure “non-cognitive” skills in an internationally comparative manner in the short term, but certain measures of non-cognitive skills could be measured in specific national contexts.
- The OECD PIAAC and World Bank STEP routes are a good starting point for measuring skills comparatively, and should be adapted to be shorter, cheaper, and more suitable for developing country contexts.
- The focus of measuring skills should be on adults of all ages, not just the young.
- Another approach could be to collect data on policy areas, such as whether a functional national qualification system exists. The advantage of this is that governments are taking skills seriously if they have such a system. The disadvantage is that the European qualification system has been criticised, owing to covering a too-narrow approach to skills.
Measuring equity post-2015
The session started by highlighting the importance of putting equity and leaving no one behind at the heart of post-2015 goals – given this has been one of the greatest failures of the MDG process. A ‘stepping stones’ approach was proposed, which entails setting interim targets to make sure no one is left behind. A focus was then given to tracking the progress of people with disabilities who are often invisible in data, and thus overlooked by policymakers. While UNICEF and the Washington Group initiatives offer an immediate means to improve data collection on access to education for people with disabilities, approaches to measuring their learning might need to be longer-term.
- Tracking progress to achieve equity should be integral to all targets, regardless of whether equity is also identified as a separate target.
- To ensure no one is left behind, it is important to track progress for disadvantaged groups in absolute terms, as well as to track the gap between them and advantaged groups.
- A ‘stepping-stones approach’ should be prioritised, including setting of interim targets.
- To accompany a stepping-stones approach, ‘quick-wins’ for policymakers should be identified so that progress can be established early-on, but with a longer-term planning horizon to make sure progress does not stall once low-hanging fruit have been captured.
- There is an urgent need to develop a globally-comparable measure of socio-economic status, given that income is not easily captured in all contexts. This is not only a concern for measuring progress towards education, so should be undertaken with those developing indicators for other targets.
- Household surveys are essential for tracking progress on inequality across time, so it is vital to improve their coverage and make sure they are undertaken at regular intervals. Surveys have most traction if they are designed to inform policy within countries, but with potential for greater use for comparability across countries.
- Existing household surveys track gender, rural/urban and, in some cases, income/wealth. It is important to recognise, however, that household surveys currently miss out important population groups (e.g. children with disabilities in institutions, nomadic population groups, those living in informal settlements in urban slums). Solutions are needed to collect data on these groups.
- Ideally, data from both administrative and survey data should be linked. In doing so, there is a need to be able to explain differences across different data collection instruments (whether due to sampling; questions etc.). The absence of recent census data is a major problem for some countries.
- There is a need to strengthen the inclusion of relevant education questions in wider (cross-country) household surveys, such as DHS and MICS.