**6 min read**
Our evaluation criteria are fundamental to the credibility and utility of our evaluations. Yet we have allowed one set of criteria – the DAC criteria – to dominate the focus of our evaluations. What they become, or whether we allow this dominance to continue, really matters now that they are under revision. We have an obligation to consider this with the attention it deserves, and to respond explicitly through our discussions and our practice.
Note: I discuss criteria from a development rather than humanitarian perspective. The latter has done significant work in this space.
Critical issues we have to consider
It is not about your project. It is about my country.
Issue 1. DAC criteria – and evaluation criteria in general – matter a great deal (more) in the Global South.
The DAC criteria continue to dominate our work in the Global South. Yet our needs and contexts matter more than we allow. Here, where it is very clear that our responsibility for doing good that sustains is both much greater and much more difficult to exercise (see my posts 8). Here, where everything we do as professionals really matters because we may support destruction or help destroy hope. Where everything – the good and the bad, and their impact – is more intensive and magnified. Where we cannot afford to waste time and resources, or step on people’s dreams, or destroy increasingly fragile ecosystems. Where our evaluation criteria must help us do the best and most useful assessment we can.
Here we are not serving our people and our planet if we do not attend sufficiently to matters such as synergistic effects, catalytic action or leverage points, negative consequences, choices and trade-offs, the sustainability of positive changes, power asymmetries, responsiveness to contexts and values and culture, intervention and development trajectories, and transformative development. So, we need to spend time considering how to balance these important issues within an environment of limited resources, lack of political interest and a rather slow-evolving evaluation commissioning system. What role should a set of evaluation criteria play in supporting us when we attend to this?
Issue 2. We do not need a dominant set of criteria. But we need a framing or categorisation that alerts us to which criteria are ‘essential’, ‘important’ and ‘nice’ to have if we wish to serve development well.
Each time we do an evaluation we can determine anew what is important, and which criteria to use; many espouse this as the best practice, especially if it makes us think carefully about which criteria we use. BUT, this will still cause us to neglect important matters that have to be addressed – both when we conceptualise, plan and implement initiatives (projects, programmes, events, institutions, systems, etc.) and subject them to some form of evaluative practice.
Many will say that ‘stakeholders’ need to determine evaluation questions, or agree to the use of specific criteria. I, too, believe in the crucial importance of authentic voices at all levels and from all groupings throughout our evaluation processes. I abhor approaches that treat some stakeholder groups as marginal or worse, as laboratory animals.
But ‘stakeholders’ are not all-knowing gods who alone should direct what we do; let us stop treating them as such. There are many times when everyone will benefit from clearer direction about what matters when dealing with development, especially from a ‘big picture’, strategic or long-term perspective. There are other important things that should determine what we evaluate, especially if we want development to be fast, effective and/or transformative. Evaluation criteria (and evaluation questions) have to draw attention to these.
This is the reason for the arguments around the fact that we need frameworks and classifications for our evaluation criteria, for example as proposed in my previous post 3 up to post 5 in this series.
Issue 3. We can cluster more and more ‘stuff’ under the current set of five DAC criteria to make it appear simple to implement, but then accountability should follow.
Some influential voices believe the best solution is to cluster a bunch of new issues under the existing or slightly expanded set of DAC criteria (usually including Coherence to make it six). This can work if we want the appearance of simplicity. I agree that nothing is quite as off-putting as a ream of questions and especially criteria that seem impossible to deal with in any one evaluation. Clustering can work and even incorporate most of my suggestions.
But are we fooling ourselves? In such clustering, much that is new will not get sufficient attention because they will be hidden under a few ‘umbrella’ criteria. It will be easier for us to ignore or water down aspects that are difficult or inconvenient to assess. And these are bound to be some of the most important issues for success.
The famous Impact criterion is a fine example. How often do we neglect the systematic search for negative consequences or impacts in our obsession with proving positive impacts!
So how will we hold ourselves accountable for attending to all crucial matters defined by any expanded criterion? And what does it mean if we do not care about such accountability?
Issue 4. We need framing of our criteria in ways that reflect freedoms and responsibilities, and balance tensions between available resources, political will and interests, and what needs to be done.
I remain amazed at how readily we have allowed – at least in certain parts of the world – the very dominant narratives about “freedom, democracy and human rights” to weaken equally important narratives about our responsibilities and obligations towards one another, towards the society in which we live, and towards our planet.
It is similar in the evaluation sphere. Our valued freedoms – to choose evaluands, boundaries, approaches, methods, questions and criteria – have to be balanced by the fact that there are responsibilities and obligations too when dealing with what to evaluate, why, how and when.
Our freedoms around our evaluation criteria relate to that which stakeholders want to know, and what we think might be important to assess given a specific context.
And our responsibilities? In the current era with its very specific framing of “development”, one of our most important responsibilities has to be to focus all our systematic evaluative actions wherever possible on
• how to best serve sustainable development and
• ensure the best chance of positive transformational change within, and across our societies and ecosystems.
Evaluation criteria (and questions) need to reflect these obligations. And they have to be identified though one or more frameworks that remind us of the reasoning for the selection of specific criteria for specific purposes.