The highly influential DAC evaluation criteria need to be revised. Some say reformed. What will modernised evaluation criteria for development look like? On what basis would we establish such criteria? How will we ensure that they suit different contexts, yet compel evaluation commissioners and evaluators to focus on what matters for sustainable and sustained development? Do we need a new set that again dominates the design of evaluations of ‘development’?
This series of posts aims to address some of these important questions.
The DAC criteria for development evaluation
Already in 1991 the OECD Development Assistance Committee defined evaluation in the development aid context as “to determine the relevance and fulfilment of objectives, developmental efficiency, effectiveness, impact and sustainability”. What became widely known as the ‘DAC criteria’ were first systematically defined in 2002. The very large number of evaluations across the world shaped by this set of five criteria are testimony to their credibility and utility, as well as to the power of the highly influential networks of evaluation commissioners who promoted their use (UNEG, the OECD DAC Network on Development Evaluation, and ECG).
Where these agencies go, our governments and many other agencies tend to follow. So our discussions about evaluation criteria, and about the DAC criteria in particular, are not only about aid (ODA) evaluation; it is also about the evaluation of development more broadly.
Let us not forget that Agenda 2030 now professes that all countries are ‘developing’; the era when one set of countries was seen as developed and setting examples for others to follow and strive towards has passed. Evaluation criteria for development effectiveness suddenly become relevant for evaluation efforts across the world.
Criteria for development evaluation, and the DAC criteria in particular, have over the years become a vital element shaping our evaluation questions (rather than the other way around) - and hence our findings. After all, evaluation criteria provide the basis for our evaluative judgments. Thomas Schwandt reminds us that they are those aspects, qualities or dimensions that enable us to determine the extent to which something is good, successful, effective, useful, and so on. They reflect the values held by evaluators and other stakeholders about how things should be; their preferences for particular outcomes; or those principles that individuals or groups hold to be desirable, good or worthy.
The pervasive use of the DAC criteria suggests a development evaluation ‘theory of action’ that in its most simple form can be formulated as follows:
If we use these five dimensions to design our evaluations, and conduct them well, we will know enough to make sound synthesis sound synthesis judgements about the value and merit of, and contributions of the interventions to development.
This is not quite so, as we will discuss in subsequent posts. But we have to start our narrative by highlighting and applauding the value and contributions of the DAC criteria.
The many positive aspects of the DAC criteria for development evaluation
The following seven points summarise the most important benefits that the DAC criteria have brought to our practice:
- Evaluation quality. At the time they offered a very significant improvement in the quality of evaluations, moving forward from output assessment and implementation fidelity.
- Evaluating for development. They reflect very important dimensions that are crucial to consider when judging ‘development’.
- Wide application. They are applicable across the full range of interventions, from single projects to portfolios of programmes within or across sectors, including at country, regional or global levels.
Development as CAS. They acknowledge, albeit only to some extent, the complex adaptive systems nature of development - for example, by highlighting the importance of unintended impacts, and of sustainability.
Innovation. They helped to propel innovation in evaluation designs and methodologies.
Focus. They were never positioned as the only useful or appropriate criteria. Their popularity and influence evolved with circumstances. They succeeded in focusing attention on key aspects that mattered for development.
Yet some evolution. Some commissioning agencies have made an effort to bring greater clarity, useful adjustments or much-needed expansions (see for example UNDP, IFAD, and ALNAP as humanitarian network). These efforts are scattered and done without any coordination or systematic sharing of experiences. They indicate that a set of five criteria is not enough, and that adjustments and flexibility are needed.
But what is even more prominent in this new ‘SDG era’ is that we acknowledge - at last! - both the importance of, and the complex adaptive systems (CAS) nature of ‘sustainable development’, and of development trajectories that need to sustain.
This growing realisation in the global evaluation community compels us to move beyond tinkering to a fundamental rethink of how we conceptualise and use our evaluation criteria for development. This is the topic of the next two posts.
Zenda Ofir is an independent South African evaluator at present based near Geneva. She works primarily in Africa and Asia, and advises organisations around the world. She is a former AfrEA President, IOCE and IDEAS Vice-President, AEA Board member, Honorary Professor at Stellenbosch University, Richard von Weizsäcker Fellow, and at present Interim Council Chair of the new International Evaluation Academy.
Thanks, dear Zenda, for taking this discussion further. As you say, the DAC criteria have been useful but after decades they need to be updated or rethought. One of the practical consequences that I’ve seen of the use of these criteria has been their mechanistic application. I’ve seen too many evaluation reports written by “purists” that use these criteria as the analytical framework and, consequently, as the headings of the report, thereby often missing the big picture (not to mention leading to repetitive and uninspiring reports).
Juha
This is indeed a key problem. They are a good guide – even though evaluation questions are supposed to lead the criteria rather than the other way around. But they have also had the unintended consequence of making us lazy or ignorant – not thinking sufficiently why we consider them as the key guide to our evaluation foci. Any new set will have the same problem. We need to think carefully how to avoid the same situation in future.
Dear Zenda,
Thank you for this paper. I really appreciate the benefits of DAC criteria you have presented in this paper.
As global benefit of DAC criteria, i think you could add the improvement of the design of projects, programmes and interventions.
In fact, at the design phase of development project, programme or intervention, more importance is given to the evaluability of the project, programmes or interventions. Based on that, a particular importance is given to the quality of the project, programme or intervention. The project, programme are designed with strong result framework. The Theory of Change is also considered.
Some specific benefits can also be identify for each DAC criteria.
It is some comments of this section.
Oumar
You are right that there are additional benefits from the use of the DAC criteria. Thank you for pointing this out. If for any reason we would stop using a “standard” set, we need to consider what benefits we would lose. And we should carefully consider what the ubiquitous use of a set like the DAC criteria means for evaluation practice, since this is not supposed to be convention.
This is good for development sectors because we have to must know about what actually will be evaluate and what we can able to see through evaluation.