Credibility, Relevance, and Policy Impact in the Evaluation of Adult Basic Skills Programs - The Case of the New Opportunities Initiative in Portugal

Oct 15, 2026

Adult basic education (ABE) policies target adults’ literacy, numeracy, and information and communications technology skills to improve qualifications and often economic outcomes such as employment and earnings. Large-scale skill gains have been treated as a “wicked” policy problem, with many evaluations showing weak or no impact. This article analyzes two NOI evaluations (2009 and 2012) in Portugal and explains why results diverged sharply. Both evaluations showed strengths but also serious methodological and/or theoretical weaknesses, suggesting a broader pattern of evaluation errors that can make ABE policy failure more likely.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages16

Reading Time40 min

AccessFree to download

Abstract

Adult basic education policy goals

The “wicked” policy problem and evaluation challenges

The New Opportunities Initiative (NOI) and its two evaluations

Methodological and theoretical weaknesses

Recommendations using conflicting NOI evidence

ADULT LITERACY EDUCATION Spring 2020 Research Article Credibility, Relevance, and Policy Impact in the Evaluation of Adult Basic Skills Programs: The Case of the New Opportunities Initiative in Portugal J. D. Carpentieri, University College London, Institute of Education David Mallows, University College London, Institute of Education José Pedro Amorim, University of Porto, Faculty of Psychology and Education Sciences, Centre for Research and Intervention in Education, and Paulo Freire Institute of Portugal Abstract Adult basic education (ABE) policies aim to help adults improve their literacy, numeracy and information and communications technology skills, as well as their qualifications, often in pursuit of economic gains such as better employment and earnings. The large-scale improvement of skills and qualifications has been referred to as a wicked policy problem, suggesting that it is extremely difficult and perhaps even impossible to achieve success in this policy domain. Evaluations have highlighted these challenges, with many programs showing little or no impact. Between 2006 and 2012, the Portuguese government ran a large-scale adult education program, the New Opportunities Initiative (NOI), which focused on the recognition and validation of adults’ existing skills and the development of literacy and numeracy. The NOI was evaluated twice, in 2009 and in 2012. These two evaluations produced very different findings and outcomes: the first evaluation found the NOI to be a success, and led to continued investment, but the second evaluation reached more negative conclusions and was used as a rationale for de-funding the program. In this article we analyze these two sets of evaluations, investigating the reasons for their starkly different conclusions. We find that, while both evaluations had strengths, they also suffered from serious methodological and/or theoretical weaknesses. These weaknesses are part of abroader pattern of evaluation errors that characterize the field of ABE more generally and which make it more likely that ABE policies will continue to fail. Using the conflicting NOI evaluations as case studies, we offer potential solutions to ABE’s evaluation problem, emphasizing the need to collect long-term longitudinal evidence on the causal mechanisms through which policy goals may be achieved. Correspondence: J. D. Carpentieri, [j.carpentieri@ucl.ac.uk](j.carpentieri@ucl.ac.uk); David Mallows, 6 [d.mallows@ucl.ac.uk](d.mallows@ucl.ac.uk); José Pedro Amorim, [jpamorim@fpce.up.pt](jpamorim@fpce.up.pt); Author Note: The contribution of Dr Amorim to the development of the paper was supported by the Portuguese Foundation for Science and Technology (FCT) and by the European Social Fund – Human Capital Operational Programme (POCH) from Portugal 2020 Programme –, in the framework of the contract established under the transitional rule of Decree Law 57/2016, amended by Law 57/2017; and by the Portuguese Government, through the FCT, under the strategic funding awarded to CIIE – Centre for Research and Intervention in Education [grant no. UID/CED/00167/2013; UID/CED/00167/2019]. There are no relevant stipulations to this funding. In modern economies, qualifications and skills are increasingly important. Studies such as the Programme for the International Assessment of Adult Competencies’ (PIAAC) Survey of Adult Skills (OECD, 2013) highlight strong correlations between low qualification levels, low levels of literacy and numeracy, and negative outcomes such as low wages, unemployment, poor health, and reduced social and political engagement. Comparisons of British cohorts born in 1958 and 1970 indicate that the negative impacts of poor basic skills and low qualifications grow over time as economies evolve (Bynner, 2002), and have lifelong impacts (Parsons & Bynner, 2007). Such evidence has had an impact on policy, moving skills and qualifications from the margins to the mainstream of policy (Hamilton & Hilli

FAQs

What is the main focus of this research article?

The article examines credibility, relevance, and policy impact in the evaluation of adult basic skills programs by analyzing two evaluations of Portugal’s New Opportunities Initiative (NOI).

Why did the NOI evaluations (2009 and 2012) produce very different conclusions?

Although both evaluations had strengths, they also had serious methodological and/or theoretical weaknesses, which contributed to starkly different findings and subsequent policy outcomes.

What problem does the article suggest affects adult basic education (ABE) evaluations more broadly?

The article identifies a broader pattern of evaluation errors in ABE, increasing the likelihood that ABE policies will continue to fail.

Oct 15, 2026

Uploaded by Chumphorn