**10 min read**
Evaluation criteria used in development contexts do not only show what evaluation commissioners, evaluators and other stakeholders value. They also provide an indication of the issues that could – and likely should – be a focus in the design and implementation of development interventions (i.e., projects, programmes, policies, events, relationships such as networks and coalitions, etc., institutions, systems, etc.).
In other words, they point to design principles intervention planners and implementers should keep in mind.
Evaluative practices – monitoring, reflection, assessment, review, evaluation, etc. – are of course intended to inform strategies, plans and implementation efforts. This is very well highlighted by the well-known Developmental Evaluation, the very useful yet still undervalued book Managing for Sustainable Development Impact, and many others. So it makes sense that our evaluation criteria should be reflected in intervention design principles, and vice versa.
This means tightening the connection between an intervention design, its implementation, its monitoring and evaluation – including when we think of the evaluation criteria we might want to use. But there is a caveat: that we anchor our focus on using what we know in order to optimise the chance of development success. This implies that we need to engage very explicitly and thoughtfully with what we mean by ‘development’ – and the values that underpin our understanding of the concept.
So – how is development reflected in our ‘theories of change’, or in our thinking about ‘large systems change’? Do we buy into that which the 2030 Agenda and its SDGs project? But then what about its notions of unlimited growth, captured among others in the ‘no-one left behind’ mantra? Do we agree with the neoliberal philosophy of development? Do we promote a focus on economic development through conventional capitalism? Or human security? Or rights-based human development? How important is transformative development? Do we believe in the urgency of the Anthropocene and the need for holistic perspectives on development? Do we have a micro or macro perspective – thinking about communities or countries or the planet as a whole? How important is the recognition of culture and cultural differences? Do we search for non-conventional approaches that explain the extremely impressive success of outlier countries? Or do we bask in the comfort of familiar ideas and models? Do we connect development directly with ‘resilience’? Do we think of development as a project or a programme, or as a trajectory? Do we recognise the crucial connections with global systems, and the complex adaptive systems behaviour of development interventions? Or do we believe in reductionism, working in fragmented bits that are easy to deal with – to the detriment of a holism? And the role of humanitarian assistance: how does that connect to development and how do our interventions and evaluations reflect such connections?
This is then the critical point when selecting design principles as well as evaluation criteria. Our conceptualisation of ‘development’ has to be reflected in the ‘valuing scheme’ we use for the design, implementation and the evaluation of a development intervention. We need to draw on what is known, yet without getting trapped in conventional thinking and dominant models – at best often not ‘the best’, and at worst damaging to societies and the ecosystems of which we are part.
This is the type of discourse we should engage with as global evaluation community. ‘Development’ now matters a great deal to all of us. But how many of us can speak with authority about our understanding and beliefs about development – including its connections with humanitarian assistance?
Our valuing scheme when we evaluate ‘development’
In the July 2018 volume of the journal ‘Evaluation’, Tom Schwandt raises the issue of the valuing schemes that evaluators apply in their work. He notes that a valuing scheme serves as the framing, the arrangement for putting a particular idea of valuing into effect. It includes assumptions and principles about social programming, policy and decision-making, and knowledge use. A valuing scheme is powerful. It shapes what is being evaluated and, more importantly, the values that are used in the valuing that we do when we make or facilitate judgments.
The DAC criteria present one such widely used valuing scheme, specifically tailored for development interventions. When we use these five criteria, we imply that we believe that they provide a sound basis for judgment of contributions to development – and hence that a synthesis of our findings will tell us enough to be able to judge whether a good contribution to development has been made.
This is of course not necessarily so – especially not in the era in which we now live, as I have pointed out in many of the preceding posts in this series.
The key problems with the current set of DAC criteria
As pointed out repeatedly in my series of posts about the DAC criteria:
- they fail to accommodate important norms very relevant for development – norms that sometimes but not always vary by context.
- the do not push us to recognise and consider the importance of culture, cultural differences, and the co-evolution of culture and contexts.
- they do not push us to consider – in the face of the urgent need for transformative change – how best to make use of the complex adaptive systems (CAS) concepts that can help us to design, implement and evaluate for better, accelerated, more sustainable, and/or more transformative development impact.
My effort to meet these challenges
In my posts 3-6 in this series I argued for having a long list of possible evaluation criteria that encourages us to think about why we select those we are using, rather than mechanically applying them. I proposed that we classify this list in at least three categories.
First, what stakeholders might want to know – flexible, depending on stakeholder wishes.
Second, the norms of the day, both universal and context-responsive – only to some extent flexible, as the norms of the day demand action by those with relevant mandates.
Third, the imperative of viewing development from a complex adaptive systems perspective – in my view, imperative if we want to optimise the chance of development success based on good design, implementation and evaluation.
The current DAC criteria revision process
In line with the notion among many that we should keep the DAC criteria KISS, I foresee that the current revision process will result in some enhanced definitions, several additions under each criterion, and perhaps one or two additions (likely ‘Coherence’ and ‘Resilience’?).
As I have noted in an earlier post, I do not really support stacking more under each criterion. There is the benefit that it will appear to be more manageable, especially where capacities are lacking. But it is also misleading. It will then in any case significantly expand the work. Importantly, it will not encourage people to think about why they use some and not other criteria. And many will continue to ignore those that are more difficult to implement – which are often the most important (for example, tracing negative consequences or impacts).
But it is possible to live with so few. The existing five can be adjusted to reflect some or all of the concerns Caroline Heider, some of the posts in this blog series, and others have raised. Additions can be made under an existing criterion, for example ‘Relevance’ can be formulated include relevance to culture and evolving contexts as interventions proceed; and include adaptive management and evolving norms. (Policy) Coherence and Resilience will likely be added. Environmental concerns can be reflected in several criteria. And as ALNAP and their very useful Guide has shown, lots more can be done to refine them.
My priority Design Principles / Evaluation Criteria
I would encourage those engaged in development interventions and in evaluation to consider using the following as priority Design Principles when conceptualising development strategies, theories of change, implementation approaches and the range of evaluative practices. They all relate to ‘interventions’ in the broadest sense – projects, programmes, policy implementation, events, relationships (networks, partnerships, etc.), institutions, systems, etc. They also follow the logic I have set out in my series on the DAC criteria.
I do not attempt to formulate each one here appropriately; to do this, we need to take to heart Michael Quinn Patton’s excellent GUIDE approach in Principles-focused Evaluation (P-FE) – which in my view should become a widely used approach in our practice.
Clear definitions, vignettes and rubrics illuminating the principle or criterion – and the underlying values attached to it – will be essential.
Many of the evaluation criteria will require qualitative rather than quantitative data collection and analysis. The value of the information will far exceed the time spent on understanding the issues through carefully formulated open questions and systematic analysis. I believe that comments that more than 5 or 6 criteria will be ‘too many’ are exaggerated; I routinely work with many more. But it will demand that qualitative analysis and synthesis are done with the necessary intelligence and rigour.
Please click the table image to view the larger version.
It is the design of interventions, and the way they are implemented, that need to be updated to reflect a better understanding of, and dealing with development – not only our DAC criteria. The fragmentation, the reductionist thinking, the soft power agendas, the belief in outdated models and ideologies at the cost of learning from real success stories, the lack of engagement with concepts and the obsession with ‘methods’ and ‘tools’, the lack of innovation in capacity building that spans the interests and capacities of the Global North and Global South – all of this needs our concerted attention. We need to revise not only the DAC criteria, but also apply the wisdom we have gleaned from our engagement with development in order to improve the principles on which intervention designs are based.
I trust that not (only) the OECD DAC, ECG and UNEG, but in particular the evaluation communities in the Global South, and also in the Global North – perhaps through EvalPartners and IOCE – will apply our minds to how we can improve and rally around a new Global Evaluation Agenda that grips the imagination of others far beyond our own rather small (in global terms) community.
I love our profession and practice. It has done a lot of good, and can do so much more. I look forward to collaborating and aligning with everyone who shares the vision of a better world, supported by evaluation as profession and practice that contributes as well as it possibly can to the type of development that our world now so urgently needs.