**8 min read**
We are sleepwalking towards irrelevance. Not because we do not do some worthwhile things. We do. But because those with power – and evaluation professionals in general – appear to be unaware or unmoved by much of what is important in the time in which we now live. We fail to evolve what we do fast enough in ways that can make a real difference to the world we are supposed to serve.
This is displayed in how we think about our evaluation criteria, including – and especially – the DAC criteria.
I left the EES Conference in Thessaloniki with a large dose of concern. In my experience the EES is one of the most stimulating VOPE conferences in the world. A lot of smart people address important issues. It was the same this time. But what concerns me is what was not present in narratives at the conference:
A clear linking of the growing understanding of the interconnectedness of things – their complex adaptive systems (CAS) behaviour – with the ‘big picture’ of the world in which we now live.
A clear understanding that we should not be unduly influenced by dominant narratives and approaches to ‘development’ or humanitarian work.
A clear engagement with the urgency with which transformative change has to take place in order to ensure healthy ecosystems and prospering societies.
Our presentations and writings should start with this era, with reference to relevant achievements, needs and challenges. Then we can work ourselves down to the granularity of ‘methods’ and ‘tools’ and ‘building capacity’. Instead, we start the other way around. We seldom get to the ‘big picture’ that needs to frame and direct the issues that matter. Yes, we can have ‘top-down’, ‘bottom-up’ or ‘up-and-down’ interventions and strategies. But we need to be much more aware of the realities within which we should aim to make a difference.
We are working ourselves into a technocratic, simplistic notion of development, humanitarian work and evaluation that makes us increasingly irrelevant for that which matters now. Yes, it is in part the result of the political economy in which we work. But we are not that powerless to change key aspects of our work. It is a matter of will and conviction.
What is the ‘big picture’?
Obviously the ‘big picture’ is rather large, and open to interpretation. But for some context, I quote an extract from one of my presentations at EES:
Every well-informed person today knows the world we live in today is in crisis – a crisis of unsustainability. To a great extent the status quo has been forged by unfettered capitalism. Consumerism is rampant around the world, driven by a set of values that has been deliberately instilled in all of us. Shareholder profit needs to be maximised and in doing so, it is necessary to exploit those without power, yet who have much to offer.
A circular economy seems far from reality. We have been upsetting every imaginable balance in the biosphere and in society. Our population growth is much too high, and the devastating effects of our actions are being felt – from the disasters stemming from climate change to the tremendous inequalities that are now affecting political spheres around the world, most visibly displayed in the dissatisfaction with political systems in the US and in Europe.
Yet our development strategies somehow still assume that unlimited growth is possible – as displayed in the mantra ‘no-one left behind’. Certainly, our politicians know this is not possible, hence the fierce competition for global power and resources, including through conflicts such as those in Syria, Libya, Yemen and the Sahel. Someone has to lose. Understandably, few will give up power over resources and prosperity without a fight.
The recent ‘World in 2050’ report compiled by three major networks, with as authors 150 leading policymakers, analysts, and modelling teams from 60 organisations, points out six global transformations necessary to succeed in achieving the SDGs. They note that it is not enough to meet aspirational social goals; transition into a new logic of world development is needed for a healthy planet, focusing on 2050 and beyond.
Yesterday’s Special IPCC report on global warming of 1.5 degrees Celsius is a milestone in a series of wake-up calls of increasing intensity.
This is the world in which we now have to conduct our work. We need to help accelerate progress in important parts of the world with a sense of urgency, yet ensure that we live within planetary boundaries. We have to see ourselves as just one species among many – but one on a path of destruction that will affect everything. It is very clear that in this era everyone needs to work with a set of values and norms that go far beyond the very common foci of ‘equity’ and ‘no-one left behind’. Or our civilisation as we know it will collapse.
Evaluation as ‘boundary practice’
We are the perfect ‘boundary practice’, just like boundary organisations. We have to bridge key divides and work on key intersections – between diverse stakeholder groupings, between the powerful and the less powerful, and between evidence generation and its application in policy and practice for better societies and individual wellbeing. This gives us a unique advantage. Let us use it well.
So how should we shift our thinking?
Without aiming to be comprehensive, the following are some of the key shifts we have to make as international evaluation community:
One, we live in the era of the Anthropocene. We need to recognise that accelerated or transformative positive change is urgently needed. We need to reflect this in our work.
The SDGs provide a lens through which to view sustainable development with greater purpose and clarity. The 2030 Agenda also helps us to understand the crucial importance of transformation in this era. Yet there was almost no reference to these critical issues at the EES conference. Little mention of the fact that we are but one part of complex ecosystems – or to be more precise and practical, of social-ecological systems, or that we need to do everything in our power to help enable transformative (positive) change.
How is this gap in our foci possible?
Everything we do today should be framed by extreme and urgent concerns around the fact that our ‘interventions’ and their evaluation will not matter if humankind does not succeed in being less ignorant, greedy and uncaring about “the other”.
Most importantly, as I have repeatedly emphasised in this series, this inevitably means engaging with development and evaluation from a complex adaptive systems (CAS) perspective. I don’t believe this is a choice we have based on any particular set of values; instead, this is essential because of how our world works, and how we can make it work better for what we need to achieve.
Two, we live in an era of dominant and often misleading (meta)-narratives, driven by extreme competition for power, both geopolitical and otherwise. We have to ensure that we also consider other narratives that can inform or direct our work.
Trump-mania, Brexit depression, ‘fake news’, ‘terrorism’ and Kim Kardashian’s latest outings are only some of the many examples of dominant narratives that I believe are often fed to media to distract us from what really matters.
We are living in a time of great significance. We are witnessing the destruction of our ecosystems at a tremendous pace; growing efforts to revive neoliberal strategies in key parts of the world while downplaying the achievements of nations whose governance systems are different from convention; propaganda and manipulation of public opinion with a ferocity I have not seen in 30 years of tracking disinformation in societies around the world; economic and related forms of warfare and conflict aimed at bringing down nations; complete erosion of personal and societal privacy in ‘democracies’; and global value chains controlled by monopolies that could end up determining how we should live.
Evaluation professionals are required to have open minds; see situations and systems from different perspectives while recognising different worldviews and models; be explicit about the values and principles we consider – and why – when we make assessments; and able to integrate and synthesise in order to make or facilitate informed judgments. We need to show that we understand at least to some extent that what happens at global, international and/or societal level influences what we experience at local level.
I believe that this is what will protect our profession from being taken over by artificial intelligence (AI) in the near future. Yet our narratives are seldom different from those found in dominant parts of the world; the values, principles and mental models on which we base our evaluations, often the same despite major differences between different societies. In many respects we need to decolonise our minds. In the meantime, let us at least interrogate what this situation might mean for the quality and relevance of our work. Are we helping to cultivate a new logic of world development? Should we do so as specialists in evaluation – a boundary practice?
Three, development and humanitarian work is complex. Let us stop pretending it is not.
We need to stop watering down the concepts of development and humanitarian intervention by thinking about it in terms of a few single ‘interventions’ – an action, a project, a programme or, if lucky, a policy or two. Once we acknowledge the interconnections between things, as the SDGs clearly ask for and demonstrate, we have to evaluate accordingly.
Co-evolution matters. Historical contexts and evolutions matter. Societal cultures and dispositions matter. Impact and development trajectories matter. Synergies matter. Power asymmetries in systems matter.
So why still the emphasis on designing and evaluating single interventions in isolation of the systems in which they are nested, dealing with logframes or linear theories of change, asking ‘what’ instead of also emphasising ‘why’, ‘how’, ‘for whom’, ‘under what conditions, ‘with whose values’, etc.? And why consider ‘impact’ without considering ‘sustainability’ at the same time? Why blindly accepting that unrealistic objectives have to be met when conditions change? Why are we still stuck with the notion of a ‘hierarchy of evidence’ cooked up by people with a narrow focus on statistical rigour that suits a too-reductionist view of the world we live in today?
Is this mantra of KISS – keep it simple, stupid – not too often making it too simple and too stupid?
Four, education and other forms of ‘capacity strengthening’ do not take place in a vacuum; they should reflect that which is important for this era.
Given all the above: why do our postgraduate curricula, our research on evaluation, our short courses and flagship efforts such as IPDET pay so little attention to bringing our profession and practice into this era?
And where is the Global South voice in all of this? Why are we so ineffective in shifting the evaluation rhetoric to better serve our continents?
The most fundamental issue confronting our thinking about evaluation criteria
We need to consider what changes in our global evaluation system will be most powerful in enabling those shifts that will make our work more meaningful.
We need to determine not only how and what we should evaluate, but how we can ensure that we support the design and implementation of development and humanitarian work itself to be more effective not just in small operational ways (this is where most evaluation recommendations appear to focus), but in more impactful conceptual ways.
Among others this means we should today be able to use our evaluations to enable and support a very dynamic engagement with designing, implementing, monitoring and evaluating for sustainable development and, where appropriate, for transformative change.
Our evaluation criteria (or evaluation questions) should inspire us to do this. They do not at present. This is an important problem in our framing of the valuing scheme represented by the DAC criteria.
In my last post on this topic – yes, really! – I will translate this important issue into a final explanation of what I expect from our evaluation criteria in practice.