The highly influential DAC evaluation criteria need to be revised. Some say reformed. What will modernised evaluation criteria for development look like? On what basis would we establish such criteria? How will we ensure that they suit different contexts, yet compel evaluation commissioners and evaluators to focus on what matters for sustainable and sustained development? Do we need a new set that again dominates the design of evaluations of ‘development’?
This series of posts aims to address some of these important questions.
The DAC criteria for development evaluation
Already in 1991 the OECD Development Assistance Committee defined evaluation in the development aid context as “to determine the relevance and fulfilment of objectives, developmental efficiency, effectiveness, impact and sustainability”. What became widely known as the ‘DAC criteria’ were first systematically defined in 2002. The very large number of evaluations across the world shaped by this set of five criteria are testimony to their credibility and utility, as well as to the power of the highly influential networks of evaluation commissioners who promoted their use (UNEG, the OECD DAC Network on Development Evaluation, and ECG).
Where these agencies go, our governments and many other agencies tend to follow. So our discussions about evaluation criteria, and about the DAC criteria in particular, are not only about aid (ODA) evaluation; it is also about the evaluation of development more broadly.
Let us not forget that Agenda 2030 now professes that all countries are ‘developing’; the era when one set of countries was seen as developed and setting examples for others to follow and strive towards has passed. Evaluation criteria for development effectiveness suddenly become relevant for evaluation efforts across the world.
Criteria for development evaluation, and the DAC criteria in particular, have over the years become a vital element shaping our evaluation questions (rather than the other way around) – and hence our findings. After all, evaluation criteria provide the basis for our evaluative judgments. Thomas Schwandt reminds us that they are those aspects, qualities or dimensions that enable us to determine the extent to which something is good, successful, effective, useful, and so on. They reflect the values held by evaluators and other stakeholders about how things should be; their preferences for particular outcomes; or those principles that individuals or groups hold to be desirable, good or worthy.
The pervasive use of the DAC criteria suggests a development evaluation ‘theory of action’ that in its most simple form can be formulated as follows:
If we use these five dimensions to design our evaluations, and conduct them well, we will know enough to make sound synthesis sound synthesis judgements about the value and merit of, and contributions of the interventions to development.
This is not quite so, as we will discuss in subsequent posts. But we have to start our narrative by highlighting and applauding the value and contributions of the DAC criteria.
The many positive aspects of the DAC criteria for development evaluation
The following seven points summarise the most important benefits that the DAC criteria have brought to our practice:
- Evaluation quality. At the time they offered a very significant improvement in the quality of evaluations, moving forward from output assessment and implementation fidelity.
- Evaluating for development. They reflect very important dimensions that are crucial to consider when judging ‘development’.
- Wide application. They are applicable across the full range of interventions, from single projects to portfolios of programmes within or across sectors, including at country, regional or global levels.
- Development as CAS. They acknowledge, albeit only to some extent, the complex adaptive systems nature of development – for example, by highlighting the importance of unintended impacts, and of sustainability.
- Innovation. They helped to propel innovation in evaluation designs and methodologies.
- Focus. They were never positioned as the only useful or appropriate criteria. Their popularity and influence evolved with circumstances. They succeeded in focusing attention on key aspects that mattered for development.
- Yet some evolution. Some commissioning agencies have made an effort to bring greater clarity, useful adjustments or much-needed expansions (see for example UNDP, IFAD, and ALNAP as humanitarian network). These efforts are scattered and done without any coordination or systematic sharing of experiences. They indicate that a set of five criteria is not enough, and that adjustments and flexibility are needed.
But what is even more prominent in this new ‘SDG era’ is that we acknowledge – at last! – both the importance of, and the complex adaptive systems (CAS) nature of ‘sustainable development’, and of development trajectories that need to sustain.
This growing realisation in the global evaluation community compels us to move beyond tinkering to a fundamental rethink of how we conceptualise and use our evaluation criteria for development. This is the topic of the next two posts.