Updating the DAC Criteria, Part 10. Sleepwalking towards irrelevance

**8 min read**

We are sleepwalking towards irrelevance. Not because we do not do some worthwhile things. We do. But because those with power – and evaluation professionals in general – appear to be unaware or unmoved by much of what is important in the time in which we now live. We fail to evolve what we do fast enough in ways that can make a real difference to the world we are supposed to serve.

This is displayed in how we think about our evaluation criteria, including – and especially – the DAC criteria.

I left the EES Conference in Thessaloniki with a large dose of concern. In my experience the EES is one of the most stimulating VOPE conferences in the world. A lot of smart people address important issues. It was the same this time. But what concerns me is what was not present in narratives at the conference:

A clear linking of the growing understanding of the interconnectedness of things – their complex adaptive systems (CAS) behaviour – with the ‘big picture’ of the world in which we now live.

A clear understanding that we should not be unduly influenced by dominant narratives and approaches to ‘development’ or humanitarian work.

A clear engagement with the urgency with which transformative change has to take place in order to ensure healthy ecosystems and prospering societies.

Our presentations and writings should start with this era, with reference to relevant achievements, needs and challenges. Then we can work ourselves down to the granularity of ‘methods’ and ‘tools’ and ‘building capacity’. Instead, we start the other way around. We seldom get to the ‘big picture’ that needs to frame and direct the issues that matter. Yes, we can have ‘top-down’, ‘bottom-up’ or ‘up-and-down’ interventions and strategies. But we need to be much more aware of the realities within which we should aim to make a difference.

We are working ourselves into a technocratic, simplistic notion of development, humanitarian work and evaluation that makes us increasingly irrelevant for that which matters now. Yes, it is in part the result of the political economy in which we work. But we are not that powerless to change key aspects of our work. It is a matter of will and conviction.

What is the ‘big picture’?

Obviously the ‘big picture’ is rather large, and open to interpretation. But for some context, I quote an extract from one of my presentations at EES:

Every well-informed person today knows the world we live in today is in crisis – a crisis of unsustainability. To a great extent the status quo has been forged by unfettered capitalism. Consumerism is rampant around the world, driven by a set of values that has been deliberately instilled in all of us. Shareholder profit needs to be maximised and in doing so, it is necessary to exploit those without power, yet who have much to offer.

A circular economy seems far from reality. We have been upsetting every imaginable balance in the biosphere and in society. Our population growth is much too high, and the devastating effects of our actions are being felt – from the disasters stemming from climate change to the tremendous inequalities that are now affecting political spheres around the world, most visibly displayed in the dissatisfaction with political systems in the US and in Europe.

Yet our development strategies somehow still assume that unlimited growth is possible – as displayed in the mantra ‘no-one left behind’. Certainly, our politicians know this is not possible, hence the fierce competition for global power and resources, including through conflicts such as those in Syria, Libya, Yemen and the Sahel. Someone has to lose. Understandably, few will give up power over resources and prosperity without a fight.

The recent ‘World in 2050’ report compiled by three major networks, with as authors 150 leading policymakers, analysts, and modelling teams from 60 organisations, points out six global transformations necessary to succeed in achieving the SDGs. They note that it is not enough to meet aspirational social goals; transition into a new logic of world development is needed for a healthy planet, focusing on 2050 and beyond.

Yesterday’s Special IPCC report on global warming of 1.5 degrees Celsius is a milestone in a series of wake-up calls of increasing intensity.

This is the world in which we now have to conduct our work. We need to help accelerate progress in important parts of the world with a sense of urgency, yet ensure that we live within planetary boundaries. We have to see ourselves as just one species among many – but one on a path of destruction that will affect everything. It is very clear that in this era everyone needs to work with a set of values and norms that go far beyond the very common foci of ‘equity’ and ‘no-one left behind’. Or our civilisation as we know it will collapse.

Evaluation as ‘boundary practice’

We are the perfect ‘boundary practice’, just like boundary organisations. We have to bridge key divides and work on key intersections – between diverse stakeholder groupings, between the powerful and the less powerful, and between evidence generation and its application in policy and practice for better societies and individual wellbeing. This gives us a unique advantage. Let us use it well.

So how should we shift our thinking?

Without aiming to be comprehensive, the following are some of the key shifts we have to make as international evaluation community:

One, we live in the era of the Anthropocene. We need to recognise that accelerated or transformative positive change is urgently needed. We need to reflect this in our work.

Geological Time Spiral

Click on the image to view the larger version.

The SDGs provide a lens through which to view sustainable development with greater purpose and clarity. The 2030 Agenda also helps us to understand the crucial importance of transformation in this era. Yet there was almost no reference to these critical issues at the EES conference. Little mention of the fact that we are but one part of complex ecosystems – or to be more precise and practical, of social-ecological systems, or that we need to do everything in our power to help enable transformative (positive) change.

How is this gap in our foci possible?

Everything we do today should be framed by extreme and urgent concerns around the fact that our ‘interventions’ and their evaluation will not matter if humankind does not succeed in being less ignorant, greedy and uncaring about “the other”.

Most importantly, as I have repeatedly emphasised in this series, this inevitably means engaging with development and evaluation from a complex adaptive systems (CAS) perspective. I don’t believe this is a choice we have based on any particular set of values; instead, this is essential because of how our world works, and how we can make it work better for what we need to achieve.

Two, we live in an era of dominant and often misleading (meta)-narratives, driven by extreme competition for power, both geopolitical and otherwise. We have to ensure that we also consider other narratives that can inform or direct our work.

Trump-mania, Brexit depression, ‘fake news’, ‘terrorism’ and Kim Kardashian’s latest outings are only some of the many examples of dominant narratives that I believe are often fed to media to distract us from what really matters.

We are living in a time of great significance. We are witnessing the destruction of our ecosystems at a tremendous pace; growing efforts to revive neoliberal strategies in key parts of the world while downplaying the achievements of nations whose governance systems are different from convention; propaganda and manipulation of public opinion with a ferocity I have not seen in 30 years of tracking disinformation in societies around the world; economic and related forms of warfare and conflict aimed at bringing down nations; complete erosion of personal and societal privacy in ‘democracies’; and global value chains controlled by monopolies that could end up determining how we should live.

Evaluation professionals are required to have open minds; see situations and systems from different perspectives while recognising different worldviews and models; be explicit about the values and principles we consider – and why – when we make assessments; and able to integrate and synthesise in order to make or facilitate informed judgments. We need to show that we understand at least to some extent that what happens at global, international and/or societal level influences what we experience at local level.

I believe that this is what will protect our profession from being taken over by artificial intelligence (AI) in the near future. Yet our narratives are seldom different from those found in dominant parts of the world; the values, principles and mental models on which we base our evaluations, often the same despite major differences between different societies. In many respects we need to decolonise our minds. In the meantime, let us at least interrogate what this situation might mean for the quality and relevance of our work. Are we helping to cultivate a new logic of world development? Should we do so as specialists in evaluation – a boundary practice?

Three, development and humanitarian work is complex. Let us stop pretending it is not.

We need to stop watering down the concepts of development and humanitarian intervention by thinking about it in terms of a few single ‘interventions’ – an action, a project, a programme or, if lucky, a policy or two. Once we acknowledge the interconnections between things, as the SDGs clearly ask for and demonstrate, we have to evaluate accordingly.

Co-evolution matters. Historical contexts and evolutions matter. Societal cultures and dispositions matter. Impact and development trajectories matter. Synergies matter. Power asymmetries in systems matter.

So why still the emphasis on designing and evaluating single interventions in isolation of the systems in which they are nested, dealing with logframes or linear theories of change, asking ‘what’ instead of also emphasising ‘why’, ‘how’, ‘for whom’, ‘under what conditions, ‘with whose values’, etc.? And why consider ‘impact’ without considering ‘sustainability’ at the same time? Why blindly accepting that unrealistic objectives have to be met when conditions change? Why are we still stuck with the notion of a ‘hierarchy of evidence’ cooked up by people with a narrow focus on statistical rigour that suits a too-reductionist view of the world we live in today?

Is this mantra of KISS – keep it simple, stupid – not too often making it too simple and too stupid?

Four, education and other forms of ‘capacity strengthening’ do not take place in a vacuum; they should reflect that which is important for this era.

Given all the above: why do our postgraduate curricula, our research on evaluation, our short courses and flagship efforts such as IPDET pay so little attention to bringing our profession and practice into this era?

And where is the Global South voice in all of this? Why are we so ineffective in shifting the evaluation rhetoric to better serve our continents?

The most fundamental issue confronting our thinking about evaluation criteria

We need to consider what changes in our global evaluation system will be most powerful in enabling those shifts that will make our work more meaningful.

We need to determine not only how and what we should evaluate, but how we can ensure that we support the design and implementation of development and humanitarian work itself to be more effective not just in small operational ways (this is where most evaluation recommendations appear to focus), but in more impactful conceptual ways.

Among others this means we should today be able to use our evaluations to enable and support a very dynamic engagement with designing, implementing, monitoring and evaluating for sustainable development and, where appropriate, for transformative change.

Our evaluation criteria (or evaluation questions) should inspire us to do this. They do not at present. This is an important problem in our framing of the valuing scheme represented by the DAC criteria.

In my last post on this topic – yes, really! – I will translate this important issue into a final explanation of what I expect from our evaluation criteria in practice.

Share this article

Zenda Ofir

Zenda Ofir is an independent South African evaluator. Based near Geneva, she works across Africa and Asia. A former AfrEA President, IOCE Vice-President and AEA Board member, she is at present IDEAS Vice-President, Lead Steward of the SDG Transformations Forum A&E Working Group and Honorary Professor at Stellenbosch University.

7 Comments

  1. Hi Zenda

    I have a concern that is at a tangent to yours, if not perhaps slightly in (polite) opposition. That is the ever continuing inflation of expectations that seems to be present in the language/rhetoric used to describe the designs of development aid programmes, their operations and in evaluations of these. My current bugbear, present in a very big programme
    I am currently associated with, is “transformational change”. This is difficult enough to define let alone achieve. Most of my experience over the last few decades suggests that if most development agencies could simply achieve some form of incremental change then that in itself would be pretty impressive. Perhaps we need to be more modest in our claims and pretentions?

    So if the DAC criteria are to be substantially revised, is this because we are totally on top of these kinds of achievements and we now need something more substantial to get our teeth into? Or have all our efforts in past decades been totally misdirected, and we now need to radically change course? Or will this be yet another exercise in inflationary rhetoric that creates an even bigger gap between reality and capacity?

    All that said, I am still in favour of keeping an eye on the big picture, Especially climate change, which keeps me awake at night..

  2. Hi Rick
    As usual, I appreciate your astute observations. I totally agree that suggestions to move on to more more “complex” things when many in a field has not managed basics warrant some debate. I just don’t see an alternative if we believe we have to support “development” (defined in the broadest sense) effectively – and even more so “sustainable development”. So I hope at least a good number of financiers, commissioners and evaluators will move ahead with advancing the field in tune with the demands of the time. That said, our real problem is that we do not (yet) attract the quality of people (in large enough numbers) that need to enter the field to do justice to the profession/practice of evaluation. It is much more difficult in my view to be a good evaluator in line with the demands of this era than a medical doctor (I can quite easily justify this statement). Yet we are probably still a long way away from the day when a little child will say “Mommy when I grow up I want to be an evaluator ….”

  3. Thank you for this Zenda. The EES conference was also a wake up call for me, not always a pleasant one, and for many of the same reasons you have highlighted here. My own sense is that evaluation got away with sidestepping a lot of the issues around examining the ‘bigger picture’ for many years, perhaps partly due to rapid ‘advances’ the profession made in terms of methods, tools and analysis approaches over that time. For many, this created the comforting illusion of a profession that was keeping pace with its complex subject. I myself started out my career in evaluation with a broad, complex-systems, socio-ecological perspective, but my enthusiasm soon got exhausted by the demands this approach places on the evaluator to be responsive, adaptive and innovative when considering the ‘real’ complexities of humanitarian and development work. There was also, frankly, little demand for this type of work – most donors and development practitioners seemed quite content with an approach that emphasizes methods and tools over awareness of complexity – and indeed often actively rejected an alternate view. The latest EES conference reminded me that I need to revise my thinking on these complex approaches. Working increasingly as we do on the SDGs, and opening our eyes to the challenges we will face when we consider the truly terrifying anticipated impact of climate change (as referenced in the latest IPCC report you linked to), how can we think otherwise?

    But here is the rub – as a full-time academic and university educator who teaches evaluation at a postgraduate level, we are struggling in a very real way to teach the next generation of evaluators to ‘do’ evaluation in this way. But we are trying in earnest to make some headway on this. This very week on our MPhil in Programme Evaluation at the University of Cape Town, my colleagues are commencing for the first time with a new, taught module entitled ‘Alternate Approaches for Complex Evaluations’ . We hope many of our students will respond positively to the new course – but we also need to learn ourselves how to teach this new content. Of course, there are some excellent resources, papers and books out there – but many students understandably want to shut their eyes to these frankly more determining perspectives, and just focus on mastering and delivering on the same old stale evaluation ‘criteria’ and ‘reductionist methodologies’ that as we know, still litter the RFPs.

    So, thank you for generating debate in this area. We desperately need a new body of evaluators who make headway in these areas, so that a shared understanding can begin to gain momentum, and trickle into our teaching and practice.

    • Sarah, great that you are doing this at UCT. I agree that it can be disheartening, and tough to try to implement against convention. It is essential to advance practice from this perspective, but we need to be pragmatic and show worth/success/improvements so that influential people can be convinced over time that it is worthwhile to go down this path. I agree very much with your last observation. There is an increasing number of evaluators who “get” it, but most who are truly influential remain trapped in the political economy of evaluation. We need to advance and share our experience, collaborating in this space. Very important for the Global South to sink their teeth into this aspect of evaluation – it is us for whom it matters most.

  4. Thought-provoking article as always Mme Zenda. Firstly, I totally concur with your observation of the minimality of the Global South during the EES conference. Hopefully, at the next conference, our participation will be improved. Please allow me to attempt to contribute to your insightful article using this structure; on the politics of development, and proposal for evaluators to apply unoppressive research approaches.

    1. The politics of evaluation
    In recent times, an interview by Francis Fukuyama https://www.newstatesman.com/culture/observations/2018/10/francis-fukuyama-interview-socialism-ought-come-back attempts to grapple with some of the political economy issues that you are partly raising. This article left me with more questions than answers. I guess for me the new era demands that we explore what I refer as “transformative development”. This unpolished concept is still drumming and lingering in my head. Therefore, trying to define it might be premature. However, personally, the question remains, for who are we evaluating and why? Interestingly, poverty and inequality are continuously spiraling. What does that say about the role of evaluation which purports to improve decision making? how is the nascent but growing field contributing towards the “my undefined” transformative development I ask? For me, “transformative development” is political and the evaluation praxis is perceived as a mechanism to be applied in the political environment geared to improve society. Thus, as evaluators, we should resist the temptation to isolate ourselves from the politics of transformative development.

    2. Unoppressive research approaches

    Social change and justice are two key components that should preoccupy researchers/evaluators. My sense is that these two components will empower evaluators to grapple with the questions of for whom and why raised earlier. As an African young evaluator, I propose that we embrace the following quote by Potts and Brown borrowed from a book called from a book titled “Research as Resistance: Critical, Indigenous and Anti-oppressive Approaches edited by Leslie Allison Brown, Susan Strega”

    “we need you to see yourself as potentially both oppressor and oppressed. We ask that you believe in your capacity for “agency”—that is, your capacity to act and alter the relations of oppression in your own world. Most of us can recognize oppression when it occurs or when we are being oppressed ourselves, but can we also recognize the complicity that each of us has in creating and sustaining oppression over others? This is even harder, especially for all of us who are well-meaning people. For White, middle-class, able-bodied, heterosexual people, this is our most important work in anti-oppressive practice—recognizing our own privilege and working to dismantle the unjust systems that keep us in that privileged space. The key in recognizing oppression is seeing the oppression that occurs through the various activities, social relations, and social practices we
    engage in with others. One such activity is the research process, even when as an evaluator you feel like the least powerful person in the world”.

    Mme Zenda, keep on shining the light and raising our eyebrows in some cases. We appreciate the fact that you are reawakening our conscience and allowing to pause and reflect on our practices.

    All of the best

  5. Thought provoking Zenda. I heartily agree that evaluators have put themselves in a small box when we focus on methodology in the absence of broader picture which poses evaluative questions like: What we are solving for? Who is at the table developing solutions-and how are the voices of the most affected incorporated? What are we learning along the way? Who and what are the ecosystem players? What contributions are all players making toward addressing the issues?
    There is a host of evaluative questions that are lost in the pursuit of developing narrow strategies that leave out context, interconnectedness, feedback loops, leverage points, etc.

    I fear that the growth of market solutions such as impact investing is falling into the same trap. It is important to disrupt this type of thinking and evaluators are positioned to be the boundary spanners and bridgers that can catalyze and connect.

  6. Thank you Zenda for this thoughtful debate. I recently attended a session at the 2018 American Evaluation Association Annual Conference which focused on the principles. Relating to this debate I concluded that evaluation criteria is more of “hardware” looking at the process of evaluation while guidelines are more of “software” looking at the person of the evaluator and how he/she conducts himself/herself to make the evaluation credible and useful.

    The world today is not short of evaluation methodologies in fact we can even borrow from applied research. But the desire to have good evaluators who are ethical and responsive is growing by day. So our intention should guide us.

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.