**10 min read**
Evaluation criteria used in development contexts do not only show what evaluation commissioners, evaluators and other stakeholders value. They also provide an indication of the issues that could - and likely should - be a focus in the design and implementation of development interventions (i.e., projects, programmes, policies, events, relationships such as networks and coalitions, etc., institutions, systems, etc.).
In other words, they point to design principles intervention planners and implementers should keep in mind.
Evaluative practices – monitoring, reflection, assessment, review, evaluation, etc. – are of course intended to inform strategies, plans and implementation efforts. This is very well highlighted by the well-known Developmental Evaluation, the very useful yet still undervalued book Managing for Sustainable Development Impact, and many others. So it makes sense that our evaluation criteria should be reflected in intervention design principles, and vice versa.
This means tightening the connection between an intervention design, its implementation, its monitoring and evaluation - including when we think of the evaluation criteria we might want to use. But there is a caveat: that we anchor our focus on using what we know in order to optimise the chance of development success. This implies that we need to engage very explicitly and thoughtfully with what we mean by 'development' - and the values that underpin our understanding of the concept.
So - how is development reflected in our 'theories of change', or in our thinking about 'large systems change'? Do we buy into that which the 2030 Agenda and its SDGs project? But then what about its notions of unlimited growth, captured among others in the 'no-one left behind' mantra? Do we agree with the neoliberal philosophy of development? Do we promote a focus on economic development through conventional capitalism? Or human security? Or rights-based human development? How important is transformative development? Do we believe in the urgency of the Anthropocene and the need for holistic perspectives on development? Do we have a micro or macro perspective - thinking about communities or countries or the planet as a whole? How important is the recognition of culture and cultural differences? Do we search for non-conventional approaches that explain the extremely impressive success of outlier countries? Or do we bask in the comfort of familiar ideas and models? Do we connect development directly with 'resilience'? Do we think of development as a project or a programme, or as a trajectory? Do we recognise the crucial connections with global systems, and the complex adaptive systems behaviour of development interventions? Or do we believe in reductionism, working in fragmented bits that are easy to deal with - to the detriment of a holism? And the role of humanitarian assistance: how does that connect to development and how do our interventions and evaluations reflect such connections?
This is then the critical point when selecting design principles as well as evaluation criteria. Our conceptualisation of 'development' has to be reflected in the 'valuing scheme' we use for the design, implementation and the evaluation of a development intervention. We need to draw on what is known, yet without getting trapped in conventional thinking and dominant models - at best often not ‘the best’, and at worst damaging to societies and the ecosystems of which we are part.
This is the type of discourse we should engage with as global evaluation community. ‘Development’ now matters a great deal to all of us. But how many of us can speak with authority about our understanding and beliefs about development - including its connections with humanitarian assistance?
Our valuing scheme when we evaluate ‘development’
In the July 2018 volume of the journal ‘Evaluation’, Tom Schwandt raises the issue of the valuing schemes that evaluators apply in their work. He notes that a valuing scheme serves as the framing, the arrangement for putting a particular idea of valuing into effect. It includes assumptions and principles about social programming, policy and decision-making, and knowledge use. A valuing scheme is powerful. It shapes what is being evaluated and, more importantly, the values that are used in the valuing that we do when we make or facilitate judgments.
The DAC criteria present one such widely used valuing scheme, specifically tailored for development interventions. When we use these five criteria, we imply that we believe that they provide a sound basis for judgment of contributions to development - and hence that a synthesis of our findings will tell us enough to be able to judge whether a good contribution to development has been made.
This is of course not necessarily so – especially not in the era in which we now live, as I have pointed out in many of the preceding posts in this series.
The key problems with the current set of DAC criteria
As pointed out repeatedly in my series of posts about the DAC criteria:
- they fail to accommodate important norms very relevant for development – norms that sometimes but not always vary by context.
- the do not push us to recognise and consider the importance of culture, cultural differences, and the co-evolution of culture and contexts.
- they do not push us to consider – in the face of the urgent need for transformative change – how best to make use of the complex adaptive systems (CAS) concepts that can help us to design, implement and evaluate for better, accelerated, more sustainable, and/or more transformative development impact.
My effort to meet these challenges
In my posts 3-6 in this series I argued for having a long list of possible evaluation criteria that encourages us to think about why we select those we are using, rather than mechanically applying them. I proposed that we classify this list in at least three categories.
First, what stakeholders might want to know – flexible, depending on stakeholder wishes.
Second, the norms of the day, both universal and context-responsive – only to some extent flexible, as the norms of the day demand action by those with relevant mandates.
Third, the imperative of viewing development from a complex adaptive systems perspective – in my view, imperative if we want to optimise the chance of development success based on good design, implementation and evaluation.
The list of examples of relevant criteria can be found in my earlier post 3 and post 6.
The current DAC criteria revision process
In line with the notion among many that we should keep the DAC criteria KISS, I foresee that the current revision process will result in some enhanced definitions, several additions under each criterion, and perhaps one or two additions (likely ‘Coherence’ and 'Resilience'?).
As I have noted in an earlier post, I do not really support stacking more under each criterion. There is the benefit that it will appear to be more manageable, especially where capacities are lacking. But it is also misleading. It will then in any case significantly expand the work. Importantly, it will not encourage people to think about why they use some and not other criteria. And many will continue to ignore those that are more difficult to implement - which are often the most important (for example, tracing negative consequences or impacts).
But it is possible to live with so few. The existing five can be adjusted to reflect some or all of the concerns Caroline Heider, some of the posts in this blog series, and others have raised. Additions can be made under an existing criterion, for example ‘Relevance’ can be formulated include relevance to culture and evolving contexts as interventions proceed; and include adaptive management and evolving norms. (Policy) Coherence and Resilience will likely be added. Environmental concerns can be reflected in several criteria. And as ALNAP and their very useful Guide has shown, lots more can be done to refine them.
My priority Design Principles / Evaluation Criteria
I would encourage those engaged in development interventions and in evaluation to consider using the following as priority Design Principles when conceptualising development strategies, theories of change, implementation approaches and the range of evaluative practices. They all relate to ‘interventions’ in the broadest sense – projects, programmes, policy implementation, events, relationships (networks, partnerships, etc.), institutions, systems, etc. They also follow the logic I have set out in my series on the DAC criteria.
I do not attempt to formulate each one here appropriately; to do this, we need to take to heart Michael Quinn Patton’s excellent GUIDE approach in Principles-focused Evaluation (P-FE) - which in my view should become a widely used approach in our practice.
Clear definitions, vignettes and rubrics illuminating the principle or criterion - and the underlying values attached to it - will be essential.
Many of the evaluation criteria will require qualitative rather than quantitative data collection and analysis. The value of the information will far exceed the time spent on understanding the issues through carefully formulated open questions and systematic analysis. I believe that comments that more than 5 or 6 criteria will be ‘too many’ are exaggerated; I routinely work with many more. But it will demand that qualitative analysis and synthesis are done with the necessary intelligence and rigour.
Please click the table image to view the larger version.
In conclusion
It is the design of interventions, and the way they are implemented, that need to be updated to reflect a better understanding of, and dealing with development - not only our DAC criteria. The fragmentation, the reductionist thinking, the soft power agendas, the belief in outdated models and ideologies at the cost of learning from real success stories, the lack of engagement with concepts and the obsession with 'methods' and 'tools', the lack of innovation in capacity building that spans the interests and capacities of the Global North and Global South - all of this needs our concerted attention. We need to revise not only the DAC criteria, but also apply the wisdom we have gleaned from our engagement with development in order to improve the principles on which intervention designs are based.
I trust that not (only) the OECD DAC, ECG and UNEG, but in particular the evaluation communities in the Global South, and also in the Global North - perhaps through EvalPartners and IOCE - will apply our minds to how we can improve and rally around a new Global Evaluation Agenda that grips the imagination of others far beyond our own rather small (in global terms) community.
I love our profession and practice. It has done a lot of good, and can do so much more. I look forward to collaborating and aligning with everyone who shares the vision of a better world, supported by evaluation as profession and practice that contributes as well as it possibly can to the type of development that our world now so urgently needs.
Zenda Ofir is an independent South African evaluator at present based near Geneva. She works primarily in Africa and Asia, and advises organisations around the world. She is a former AfrEA President, IOCE and IDEAS Vice-President, AEA Board member, Honorary Professor at Stellenbosch University, Richard von Weizsäcker Fellow, and at present Interim Council Chair of the new International Evaluation Academy.
Dear Zenda
Thank you for this great contribution on design principles in support of development and also on evaluation criteria. I really appreciate them. Congratulations!
But i have some comments.
First on design principles, you don’t mention the principle of accountability. I think it is very important in the context of results-based management. It can help too much in the management of the project and help inform stakeholders on the implementation of the project. It is a tool for good governement.
You didn’t mention also the “make do” principle. It seems important to strengthen the decentralization process in Africa. Central Government can’t implement all the interventions alone and also need local governements, civil society and development agencies. It is important in the context of SDGs where it is promoted “no-one left behind”
Secondly in evaluation criteria, i suggest that ecological sustainability be a part of intersecting impact and sustainability. I think it is also important to save efficiency criteria because this criteria is useful to appreciate the achievement of development result with ressources used for that.
You didn’t mention also the criteria on culture and cultural differences. It can be important to consider them in some specific contexts. In Africa context, there is Agenda 2063. The objective 16 of the Agenda 2063 is focus on culture and all the african countries are working to align theirs development plans on the 20 objectives of Agenda 2063. So have a specific evaluation criteria on culture can be useful.
Best
Dear Zenda thank you for very thought provoking series. I would like to comment on some aspects from this overall.
Question: Are evaluators sleepwalking to irrelevance? Should we not focus more on design principles rather than obsess about evaluation criteria?
Observation: One of the key aspects for me is to know who the powers are and what they expect – or want to “really achieve” – the closer you get to the truth the better you will be able to make sense from a CAS perspective. Unfortunately herein lies a problem. In a lot of cases the intervention is just a mere vehicle to get “other” information or some kind of leverage or foot in the door or just seeing the change. Am I being cynical about whether people are doing development for the good of development? Not always, according to my view. Experiences gained in conducting evaluations in conflict countries (in semi states of war) really showed the value of keeping an open mind and being much more aware of agendas around you and then finding the “truth” somewhere in between. Question is, perhaps, do other organisations have other multiple agendas and if so, the best would be to have both a micro and macro picture and ability to deal with “changes” at very short notice. There will always be a divide between the one with funds and the one receiving it – but as you say if we have and keep an open mind and be true to agreed principles, values and even beliefs – up front – then we might have better shot at being “true” in the evaluation… and perhaps even be in a position to get a glimpse of a global view. One of the most difficult aspects to achieve is continuous human behaviour change and if a number or most developmental programmes or Aid funding has its focus on that aspect then the complexities are even more ad infinitum. Just as I believe there cannot be continuous positive growth ALL the time, so I also think that sustainability needs to be viewed in this regard – to give an example to eradicate poverty: if poverty is a state of mind then the complexities are immense, even if that is not your view then it still remains clear as mud because it is so interrelated with a myriad other factors. So link to criteria… make criteria fluid and state what you are using upfront in the case of the South – ensure it is orientated to the good of the South – or use the concept of “do no harm” in the broadest sense of the word.
Question: …So it makes sense that our evaluation criteria should be reflected in intervention design principles, and vice versa…that we anchor our focus on using what we know in order to optimise the chance of development success. This implies that we need to engage very explicitly and thoughtfully with what we mean by ‘development’ – and the values that underpin our understanding of the concept.
Observation: I am in favour of a valuing scheme, but again not going the way of cookie cutter…. and yes, it will be very useful to look at your suggestion in terms of the core criteria and then some variations in flexibility of criteria. I put in a question – have the DAC criteria really changed the lives of the ones being evaluated – AT SCALE? in Africa..will it in future ? Sustainability over 5, 10, 20 years…
I agree that we need to have clearly stated principles at design state and then take process to fruition. Looking at your list I would have as key – ensuring the “entire” community is reflected rather than stipulate various groups since new groups will be formed no doubt as days pass. The rest of the list is great.
Your list re: CAS – If the principle of holism is adhered to then one hopes not to have a need to be that specific about the interactions between things. When it comes to significance it will be complex since the role of power, funder and stakeholder will be key. Under significance the discourse around change is key – the required change needs to be put upfront, and who wants it and why, and to what good. If we want the farmers to change their farming methods and use more fertiliser to grow better crops, and a fertiliser company funds the “programme”, then as an evaluator I will probably not evaluate such a programme. Others will since they have other “principles” or values and that is also a choice. Where does it leave the bigger picture – do no harm, to whom…? for the next 4 areas I have the same question: the required change is what and who wants it and why, to what good? If this is clearly spelled out at the start one can use the criteria and at the end reflect on what you achieved and whether there was a significant impact or change or not. The key however for me will be, did the participants / stakeholders – whoever they are – really find anything positive or new, or learn new skills from being part of this endeavour?
I think a lot can be done by all evaluators as you state, but then a question I was asked … do I spend the limited funds on the intervention or on the evaluation. If it is legislated and as evaluator we have to follow the “rule” then we might lose a lot of the good. (Move towards centralisation & amp; authorisation). Perhaps here I am interested in the funding models and issues around that, especially regarding evaluation
Dear Zenda
The results of the OECD Consultation are now available and they reflect some of the concerns you have raised.
See:
https://ieg.worldbankgroup.org/sites/default/files/Data/DAC-Criteria/ConsultationReport_EvaluationCriteria.pdf
Bob
Dear Bob, Indeed, and no surprise at the results – maintain more or less the status quo. But in the end my major concern – that the combination of criteria does not help us to address sufficiently ‘development’ of the types required globally – may not be addressed. It is interesting how the (carefully selected?) comments quoted on the slides almost all reflect the notion that the set should not be adapted dramatically. I do not necessarily believe that the majority of people are right; rather that they appreciate comfort and dislike change. These responses reflect this notion. I would have liked to see more thoughtful engagement with what is essential in today’s world if evaluation for local to global development is to be truly effective. And of course, we will see how well ‘complex systems’ as concept is addressed in the final proposed set and their descriptions. This will be crucial.
Hi Zenda,
I’m late chiming in but interesting article and timely for me. I was just at a meeting where they were asking about responsiveness and wondering where that might come in for measurement as obviously a great criterion.
Do you think within the DAC headings (aside from being boring), people could explore more (and deeper) if they try? I think there is lots of room for thoughtfulness as long as people ask some of the questions you pose above.
I also think different methods could be used to get answers (borrowing from complexity) but that its tough for the donors to go into unknown territory.
Thanks for the blog series, I’ve been following and learning along the way!
Hi Michelle, Nice to hear my blog has been useful. There is definitely room within the criteria for their more thoughtful application. I suspect the DAC criteria revision will suggest some of that. But we also need to get new ones more ingrained in our practice. The key point is to get away from the recipe they have become for many. Our biggest problem is people – commissioners and evaluators – who don’t want to change. If we are evaluation specialists we should be able to engage with complex systems insights. It is a matter of will rather than capability.