“I think that the problem is that nobody comes here, nobody knows how we live or what we go through and that’s why everyone thinks we’re evil.” – Francesca
Francesca looked at the recorder on the table, as if she had offended someone, straightened her back and stared at me, waiting for another question. I was interviewing her as part of an impact evaluation of an urban upgrading program in Naples, Italy.
The program’s ultimate purpose was to support out-of-school children in finding vocational traineeships, particularly in the peripheral areas of Naples where the challenges of street violence and police brutality are an everyday reality for young people. In response to these daunting circumstances, the theory of change for the program was elegantly simple: the nonprofit organisation would offer morning lessons in literacy and numeracy to the kids. Then, in the afternoon sessions, they would be paired with local business owners to learn a trade, fostering a direct link between education and practical skill development. This holistic approach was anticipated to yield a spectrum of positive outcomes, ranging from increased rates of school re-enrolment and completion, to the cultivation of part-time employment opportunities and, perhaps most importantly, to a potential reduction in crime rates within these marginalised communities.
However, the issue I faced when trying to assess this intervention was that I lacked a full understanding of the reality the children lived in. While school completion rates showed improvement, the underlying issue of low academic attainment persisted. Furthermore, the anticipated participation of children in vocational training failed to materialise as expected. Towards the end of the month dedicated to observing and documenting the program, I realised that what I had been missing was an understanding of children as active agents and not just as a treatment group. This realisation quickly became a watershed moment for my own practice and conduct when driving impact evaluations both in my native Italy and for international development programs globally.
II. The Challenge of Understanding Participant Experiences
An image taken by Matias during his time on the project in Naples.
Francesca’s perspective highlighted the disconnect between outsiders’ perceptions and the reality of participants in social programs. When I came into the school, I knew these children had ‘failed’ other interventions. Nothing seemed to work and they were mostly framed through data points. They were aged 10-14, they had dropped out of school after completing primary, they struggled to read a full paragraph and do basic maths. As an outsider evaluating the impact of an intervention, it is easy to fall into the trap of simplification. We all enjoy modelling impact to simplify reality. Through an experimental lens, I could frame the program as a treatment that, when given to a particular group, would have caused an immediate positive reaction. This kind of attitude is what made me completely miss the impact the program was actually having, i.e., providing children with more agency over their futures and offering alternatives to the life-paths that were set for them. Previous evaluation methodologies, based on quantitative surveys, had helped nonprofits and funders in this and other similar settings identify that their programs weren’t working. What they failed to do, however, was to provide any useful information as to why they weren’t, and generate helpful insights to move forward. In a situation where a narrow perspective would have led donors and nonprofits to move away from this particular group of beneficiaries given that nothing seemed to work, the program staff and funders decided to keep iterating their intervention, learning, and developing more in-depth evaluation methodologies.
Given the complex reality on the ground, I decided to integrate a qualitative approach to the impact assessment, one that would avoid reducing program participants to mere objects of evaluation. In collaboration with the nonprofit’s staff, I chose to prioritise beneficiaries’ agency and stories, how they saw themselves represented in the media and social sector, and what decisions they made in response to various ‘treatments’ they took part in.
When I learnt about the different ways children in the program were making decisions about their schooling, work and life chances, I understood that the program was doing much more than addressing learning gaps or providing vocational training, it was offering a diverse set of life and career options to children that may not otherwise have had them. Keeping beneficiaries’ agency at the forefront of my evaluation helped me recognize participants as active agents with unique circumstances. It also helped the nonprofit’s staff gain a more nuanced understanding of the program’s mechanisms, its successes and failures. Most importantly it positioned program participants as vital partners in impact evaluation, generating a deeper understanding of context, program design and delivery that was used to improve the program’s impact.
This approach to impact evaluation is one that I now work to integrate whenever I look at a particular program. It is also at the core of IDInsight’s Dignity Initiative and is part of The Life You Can Save’s evaluation framework.
III. Practical Steps for a Agency-Focused Approach
Working as part of The Life You Can Save’s research and evaluation team, I face the challenges of impact evaluation day in and day out. We often have limited information about a nonprofit’s work, the context they work in, and the program partners they serve. Spending weeks on qualitative impact assessment is anything but practical, which is why placing our trust in those working on the ground is key. These developments are mirrored in the recent trends within the sector, which increasingly emphasize trust-based philanthropy and initiatives rooted in local communities. These shifts hold the promise of bridging the gap between funders and the intricate realities of impact work at the grassroots level. I strongly advocate for donors and grant-makers to complement rigorous impact assessment with more empathetic, participant-centered methodologies, as these approaches are better suited to accurately capturing the nuances of intervention impact.
In addition to following these positive trends, here are a few steps anyone working or interested in the sector can take:
Source: Unsplash
Learning: An invaluable resource I recommend to anyone engaged in international development is “Voices of the Poor”. This comprehensive research, curated by the World Bank, draws upon interviews and testimonials from over 60,000 individuals living in poverty. It provides a nuanced understanding of their lives and contexts, challenging prevailing stereotypes by highlighting their agency, dignity, resourcefulness and aspirations for change. Moreover, it underscores the highly contextual nature of poverty, shaped by diverse factors such as geography, culture, politics and social norms.
Immersion: While dedicating extended periods to observing specific programs may be impractical, site visits, such as the Australian International Development Network’s “Insight Tours”, offer invaluable learning opportunities. They allow for firsthand experience of on-the-ground realities and facilitate direct engagement in conversations with participants.
Dialogue with nonprofit staff: Engaging in conversations with staff involved in nonprofit organisations provides insights into how they view their programs and prioritise the agency of beneficiaries. In my own practice, I make a strong effort to gather knowledge from those working directly in the field. This includes understanding their perspectives on program effectiveness, identifying areas where it may lack depth, and acknowledging the challenges they encounter in their day-to-day operations, all while exploring how they learn about and actively support the aspirations of beneficiaries.
—————————–
Matías Nestore works as a Research and Evaluation Associate at The Life You Can Save, leveraging his background in Education and International Development. The Life You Can Save makes “smart giving simple” by recommending charities that save lives and improve well-being where each dollar goes the furthest. Prior to his current role, he co-led Shaping Horizons, a social enterprise incubator born out of the University of Cambridge.