**8 min read**
We are sleepwalking towards irrelevance. Not because we do not do some worthwhile things. We do. But because those with power – and evaluation professionals in general - appear to be unaware or unmoved by much of what is important in the time in which we now live. We fail to evolve what we do fast enough in ways that can make a real difference to the world we are supposed to serve.
This is displayed in how we think about our evaluation criteria, including – and especially - the DAC criteria.
I left the EES Conference in Thessaloniki with a large dose of concern. In my experience the EES is one of the most stimulating VOPE conferences in the world. A lot of smart people address important issues. It was the same this time. But what concerns me is what was not present in narratives at the conference:
A clear linking of the growing understanding of the interconnectedness of things – their complex adaptive systems (CAS) behaviour - with the ‘big picture’ of the world in which we now live.
A clear understanding that we should not be unduly influenced by dominant narratives and approaches to ‘development’ or humanitarian work.
A clear engagement with the urgency with which transformative change has to take place in order to ensure healthy ecosystems and prospering societies.
Our presentations and writings should start with this era, with reference to relevant achievements, needs and challenges. Then we can work ourselves down to the granularity of ‘methods’ and ‘tools’ and ‘building capacity’. Instead, we start the other way around. We seldom get to the ‘big picture’ that needs to frame and direct the issues that matter. Yes, we can have ‘top-down’, ‘bottom-up’ or ‘up-and-down’ interventions and strategies. But we need to be much more aware of the realities within which we should aim to make a difference.
We are working ourselves into a technocratic, simplistic notion of development, humanitarian work and evaluation that makes us increasingly irrelevant for that which matters now. Yes, it is in part the result of the political economy in which we work. But we are not that powerless to change key aspects of our work. It is a matter of will and conviction.
What is the ‘big picture’?
Obviously the ‘big picture’ is rather large, and open to interpretation. But for some context, I quote an extract from one of my presentations at EES:
Every well-informed person today knows the world we live in today is in crisis – a crisis of unsustainability. To a great extent the status quo has been forged by unfettered capitalism. Consumerism is rampant around the world, driven by a set of values that has been deliberately instilled in all of us. Shareholder profit needs to be maximised and in doing so, it is necessary to exploit those without power, yet who have much to offer.
A circular economy seems far from reality. We have been upsetting every imaginable balance in the biosphere and in society. Our population growth is much too high, and the devastating effects of our actions are being felt - from the disasters stemming from climate change to the tremendous inequalities that are now affecting political spheres around the world, most visibly displayed in the dissatisfaction with political systems in the US and in Europe.
Yet our development strategies somehow still assume that unlimited growth is possible – as displayed in the mantra ‘no-one left behind’. Certainly, our politicians know this is not possible, hence the fierce competition for global power and resources, including through conflicts such as those in Syria, Libya, Yemen and the Sahel. Someone has to lose. Understandably, few will give up power over resources and prosperity without a fight.
The recent ‘World in 2050’ report compiled by three major networks, with as authors 150 leading policymakers, analysts, and modelling teams from 60 organisations, points out six global transformations necessary to succeed in achieving the SDGs. They note that it is not enough to meet aspirational social goals; transition into a new logic of world development is needed for a healthy planet, focusing on 2050 and beyond.
Yesterday’s Special IPCC report on global warming of 1.5 degrees Celsius is a milestone in a series of wake-up calls of increasing intensity.
This is the world in which we now have to conduct our work. We need to help accelerate progress in important parts of the world with a sense of urgency, yet ensure that we live within planetary boundaries. We have to see ourselves as just one species among many - but one on a path of destruction that will affect everything. It is very clear that in this era everyone needs to work with a set of values and norms that go far beyond the very common foci of ‘equity’ and ‘no-one left behind’. Or our civilisation as we know it will collapse.
Evaluation as ‘boundary practice’
We are the perfect ‘boundary practice', just like boundary organisations. We have to bridge key divides and work on key intersections - between diverse stakeholder groupings, between the powerful and the less powerful, and between evidence generation and its application in policy and practice for better societies and individual wellbeing. This gives us a unique advantage. Let us use it well.
So how should we shift our thinking?
Without aiming to be comprehensive, the following are some of the key shifts we have to make as international evaluation community:
One, we live in the era of the Anthropocene. We need to recognise that accelerated or transformative positive change is urgently needed. We need to reflect this in our work.
The SDGs provide a lens through which to view sustainable development with greater purpose and clarity. The 2030 Agenda also helps us to understand the crucial importance of transformation in this era. Yet there was almost no reference to these critical issues at the EES conference. Little mention of the fact that we are but one part of complex ecosystems - or to be more precise and practical, of social-ecological systems, or that we need to do everything in our power to help enable transformative (positive) change.
Click on the image to view the larger version.
How is this gap in our foci possible?
Everything we do today should be framed by extreme and urgent concerns around the fact that our ‘interventions’ and their evaluation will not matter if humankind does not succeed in being less ignorant, greedy and uncaring about "the other".
Most importantly, as I have repeatedly emphasised in this series, this inevitably means engaging with development and evaluation from a complex adaptive systems (CAS) perspective. I don't believe this is a choice we have based on any particular set of values; instead, this is essential because of how our world works, and how we can make it work better for what we need to achieve.
Two, we live in an era of dominant and often misleading (meta)-narratives, driven by extreme competition for power, both geopolitical and otherwise. We have to ensure that we also consider other narratives that can inform or direct our work.
Trump-mania, Brexit depression, ‘fake news’, ‘terrorism’ and Kim Kardashian’s latest outings are only some of the many examples of dominant narratives that I believe are often fed to media to distract us from what really matters.
We are living in a time of great significance. We are witnessing the destruction of our ecosystems at a tremendous pace; growing efforts to revive neoliberal strategies in key parts of the world while downplaying the achievements of nations whose governance systems are different from convention; propaganda and manipulation of public opinion with a ferocity I have not seen in 30 years of tracking disinformation in societies around the world; economic and related forms of warfare and conflict aimed at bringing down nations; complete erosion of personal and societal privacy in 'democracies'; and global value chains controlled by monopolies that could end up determining how we should live.
Evaluation professionals are required to have open minds; see situations and systems from different perspectives while recognising different worldviews and models; be explicit about the values and principles we consider - and why - when we make assessments; and able to integrate and synthesise in order to make or facilitate informed judgments. We need to show that we understand at least to some extent that what happens at global, international and/or societal level influences what we experience at local level.
I believe that this is what will protect our profession from being taken over by artificial intelligence (AI) in the near future. Yet our narratives are seldom different from those found in dominant parts of the world; the values, principles and mental models on which we base our evaluations, often the same despite major differences between different societies. In many respects we need to decolonise our minds. In the meantime, let us at least interrogate what this situation might mean for the quality and relevance of our work. Are we helping to cultivate a new logic of world development? Should we do so as specialists in evaluation - a boundary practice?
Three, development and humanitarian work is complex. Let us stop pretending it is not.
We need to stop watering down the concepts of development and humanitarian intervention by thinking about it in terms of a few single ‘interventions’ – an action, a project, a programme or, if lucky, a policy or two. Once we acknowledge the interconnections between things, as the SDGs clearly ask for and demonstrate, we have to evaluate accordingly.
Co-evolution matters. Historical contexts and evolutions matter. Societal cultures and dispositions matter. Impact and development trajectories matter. Synergies matter. Power asymmetries in systems matter.
So why still the emphasis on designing and evaluating single interventions in isolation of the systems in which they are nested, dealing with logframes or linear theories of change, asking ‘what’ instead of also emphasising ‘why’, ‘how’, ‘for whom’, ‘under what conditions, ‘with whose values’, etc.? And why consider ‘impact’ without considering ‘sustainability’ at the same time? Why blindly accepting that unrealistic objectives have to be met when conditions change? Why are we still stuck with the notion of a ‘hierarchy of evidence’ cooked up by people with a narrow focus on statistical rigour that suits a too-reductionist view of the world we live in today?
Is this mantra of KISS – keep it simple, stupid – not too often making it too simple and too stupid?
Four, education and other forms of ‘capacity strengthening’ do not take place in a vacuum; they should reflect that which is important for this era.
Given all the above: why do our postgraduate curricula, our research on evaluation, our short courses and flagship efforts such as IPDET pay so little attention to bringing our profession and practice into this era?
And where is the Global South voice in all of this? Why are we so ineffective in shifting the evaluation rhetoric to better serve our continents?
The most fundamental issue confronting our thinking about evaluation criteria
We need to consider what changes in our global evaluation system will be most powerful in enabling those shifts that will make our work more meaningful.
We need to determine not only how and what we should evaluate, but how we can ensure that we support the design and implementation of development and humanitarian work itself to be more effective not just in small operational ways (this is where most evaluation recommendations appear to focus), but in more impactful conceptual ways.
Among others this means we should today be able to use our evaluations to enable and support a very dynamic engagement with designing, implementing, monitoring and evaluating for sustainable development and, where appropriate, for transformative change.
Our evaluation criteria (or evaluation questions) should inspire us to do this. They do not at present. This is an important problem in our framing of the valuing scheme represented by the DAC criteria.
In my last post on this topic – yes, really! – I will translate this important issue into a final explanation of what I expect from our evaluation criteria in practice.
Zenda Ofir is an independent South African evaluator at present based near Geneva. She works primarily in Africa and Asia, and advises organisations around the world. She is a former AfrEA President, IOCE and IDEAS Vice-President, AEA Board member, Honorary Professor at Stellenbosch University, Richard von Weizsäcker Fellow, and at present Interim Council Chair of the new International Evaluation Academy.
Hi Zenda
I have a concern that is at a tangent to yours, if not perhaps slightly in (polite) opposition. That is the ever continuing inflation of expectations that seems to be present in the language/rhetoric used to describe the designs of development aid programmes, their operations and in evaluations of these. My current bugbear, present in a very big programme
I am currently associated with, is “transformational change”. This is difficult enough to define let alone achieve. Most of my experience over the last few decades suggests that if most development agencies could simply achieve some form of incremental change then that in itself would be pretty impressive. Perhaps we need to be more modest in our claims and pretentions?
So if the DAC criteria are to be substantially revised, is this because we are totally on top of these kinds of achievements and we now need something more substantial to get our teeth into? Or have all our efforts in past decades been totally misdirected, and we now need to radically change course? Or will this be yet another exercise in inflationary rhetoric that creates an even bigger gap between reality and capacity?
All that said, I am still in favour of keeping an eye on the big picture, Especially climate change, which keeps me awake at night..
Hi Rick
As usual, I appreciate your astute observations. I totally agree that suggestions to move on to more more “complex” things when many in a field has not managed basics warrant some debate. I just don’t see an alternative if we believe we have to support “development” (defined in the broadest sense) effectively – and even more so “sustainable development”. So I hope at least a good number of financiers, commissioners and evaluators will move ahead with advancing the field in tune with the demands of the time. That said, our real problem is that we do not (yet) attract the quality of people (in large enough numbers) that need to enter the field to do justice to the profession/practice of evaluation. It is much more difficult in my view to be a good evaluator in line with the demands of this era than a medical doctor (I can quite easily justify this statement). Yet we are probably still a long way away from the day when a little child will say “Mommy when I grow up I want to be an evaluator ….”
Thank you for this Zenda. The EES conference was also a wake up call for me, not always a pleasant one, and for many of the same reasons you have highlighted here. My own sense is that evaluation got away with sidestepping a lot of the issues around examining the ‘bigger picture’ for many years, perhaps partly due to rapid ‘advances’ the profession made in terms of methods, tools and analysis approaches over that time. For many, this created the comforting illusion of a profession that was keeping pace with its complex subject. I myself started out my career in evaluation with a broad, complex-systems, socio-ecological perspective, but my enthusiasm soon got exhausted by the demands this approach places on the evaluator to be responsive, adaptive and innovative when considering the ‘real’ complexities of humanitarian and development work. There was also, frankly, little demand for this type of work – most donors and development practitioners seemed quite content with an approach that emphasizes methods and tools over awareness of complexity – and indeed often actively rejected an alternate view. The latest EES conference reminded me that I need to revise my thinking on these complex approaches. Working increasingly as we do on the SDGs, and opening our eyes to the challenges we will face when we consider the truly terrifying anticipated impact of climate change (as referenced in the latest IPCC report you linked to), how can we think otherwise?
But here is the rub – as a full-time academic and university educator who teaches evaluation at a postgraduate level, we are struggling in a very real way to teach the next generation of evaluators to ‘do’ evaluation in this way. But we are trying in earnest to make some headway on this. This very week on our MPhil in Programme Evaluation at the University of Cape Town, my colleagues are commencing for the first time with a new, taught module entitled ‘Alternate Approaches for Complex Evaluations’ . We hope many of our students will respond positively to the new course – but we also need to learn ourselves how to teach this new content. Of course, there are some excellent resources, papers and books out there – but many students understandably want to shut their eyes to these frankly more determining perspectives, and just focus on mastering and delivering on the same old stale evaluation ‘criteria’ and ‘reductionist methodologies’ that as we know, still litter the RFPs.
So, thank you for generating debate in this area. We desperately need a new body of evaluators who make headway in these areas, so that a shared understanding can begin to gain momentum, and trickle into our teaching and practice.
Sarah, great that you are doing this at UCT. I agree that it can be disheartening, and tough to try to implement against convention. It is essential to advance practice from this perspective, but we need to be pragmatic and show worth/success/improvements so that influential people can be convinced over time that it is worthwhile to go down this path. I agree very much with your last observation. There is an increasing number of evaluators who “get” it, but most who are truly influential remain trapped in the political economy of evaluation. We need to advance and share our experience, collaborating in this space. Very important for the Global South to sink their teeth into this aspect of evaluation – it is us for whom it matters most.
Thought-provoking article as always Mme Zenda. Firstly, I totally concur with your observation of the minimality of the Global South during the EES conference. Hopefully, at the next conference, our participation will be improved. Please allow me to attempt to contribute to your insightful article using this structure; on the politics of development, and proposal for evaluators to apply unoppressive research approaches.
1. The politics of evaluation
In recent times, an interview by Francis Fukuyama https://www.newstatesman.com/culture/observations/2018/10/francis-fukuyama-interview-socialism-ought-come-back attempts to grapple with some of the political economy issues that you are partly raising. This article left me with more questions than answers. I guess for me the new era demands that we explore what I refer as “transformative development”. This unpolished concept is still drumming and lingering in my head. Therefore, trying to define it might be premature. However, personally, the question remains, for who are we evaluating and why? Interestingly, poverty and inequality are continuously spiraling. What does that say about the role of evaluation which purports to improve decision making? how is the nascent but growing field contributing towards the “my undefined” transformative development I ask? For me, “transformative development” is political and the evaluation praxis is perceived as a mechanism to be applied in the political environment geared to improve society. Thus, as evaluators, we should resist the temptation to isolate ourselves from the politics of transformative development.
2. Unoppressive research approaches
Social change and justice are two key components that should preoccupy researchers/evaluators. My sense is that these two components will empower evaluators to grapple with the questions of for whom and why raised earlier. As an African young evaluator, I propose that we embrace the following quote by Potts and Brown borrowed from a book called from a book titled “Research as Resistance: Critical, Indigenous and Anti-oppressive Approaches edited by Leslie Allison Brown, Susan Strega”
“we need you to see yourself as potentially both oppressor and oppressed. We ask that you believe in your capacity for “agency”—that is, your capacity to act and alter the relations of oppression in your own world. Most of us can recognize oppression when it occurs or when we are being oppressed ourselves, but can we also recognize the complicity that each of us has in creating and sustaining oppression over others? This is even harder, especially for all of us who are well-meaning people. For White, middle-class, able-bodied, heterosexual people, this is our most important work in anti-oppressive practice—recognizing our own privilege and working to dismantle the unjust systems that keep us in that privileged space. The key in recognizing oppression is seeing the oppression that occurs through the various activities, social relations, and social practices we
engage in with others. One such activity is the research process, even when as an evaluator you feel like the least powerful person in the world”.
Mme Zenda, keep on shining the light and raising our eyebrows in some cases. We appreciate the fact that you are reawakening our conscience and allowing to pause and reflect on our practices.
All of the best
Thought provoking Zenda. I heartily agree that evaluators have put themselves in a small box when we focus on methodology in the absence of broader picture which poses evaluative questions like: What we are solving for? Who is at the table developing solutions-and how are the voices of the most affected incorporated? What are we learning along the way? Who and what are the ecosystem players? What contributions are all players making toward addressing the issues?
There is a host of evaluative questions that are lost in the pursuit of developing narrow strategies that leave out context, interconnectedness, feedback loops, leverage points, etc.
I fear that the growth of market solutions such as impact investing is falling into the same trap. It is important to disrupt this type of thinking and evaluators are positioned to be the boundary spanners and bridgers that can catalyze and connect.
Thank you Zenda for this thoughtful debate. I recently attended a session at the 2018 American Evaluation Association Annual Conference which focused on the principles. Relating to this debate I concluded that evaluation criteria is more of “hardware” looking at the process of evaluation while guidelines are more of “software” looking at the person of the evaluator and how he/she conducts himself/herself to make the evaluation credible and useful.
The world today is not short of evaluation methodologies in fact we can even borrow from applied research. But the desire to have good evaluators who are ethical and responsive is growing by day. So our intention should guide us.
Zenda
You are onto something critically important: international development needs to reorient its policy priorities and development evaluation needs to facilitate the shift. Past development initiatives focused on the nation state. They were deliberately focused on inducing increased connectivity across borders. This strategy generated important benefits: it facilitated poverty reduction at the global level and it induced economic convergence between the global south and the global north. In parallel, social indicators improved at the global level… But the dark side of this market-oriented strategy soon came into view. My own work on policy coherence for development had presaged the need to evaluate all the transmission belts of globalization but my call to do so went largely unheeded. And as your blog eloquently stresses, evaluation has remained largely focused on individual interventions and country assistance programs while the development strategies in place left the least developed countries behind, led to surging inequality within countries, and generated massive environmental costs. In parallel, a new global oligarchy has emerged along with populist movements that now threaten democracy. As you correctly highlight, the planet currently faces existential risks as social inequities multiply, natural resources get depleted and climate change looms. The SDGs capture the immense challenges now facing humanity. What then are the implications for evaluation? Michael Q. Patton has once again shown us the way. His Blue Marble Evaluation initiative (assisted by Michael Scriven’s Fast Forward Fund) aims to treat the planet as the unit of account for development evaluation. The initiative promotes evaluation focused on “climate change, massive cross-border movement of displaced persons, virulent super-viruses and contagious diseases that threaten world health, dying oceans, global terrorism, global food insecurity, global economic interdependence, and multinational capitalism”, etc. Blue Marble Evaluation recognizes that “evaluators need special perspectives and competences to engage and evaluate these global change efforts, to monitor, improve, help develop, and ultimately judge the effectiveness, efficiency, relevance, and sustainability of these global change efforts”. It does not require any significant change in the DAC criteria. Rather it “means bringing the full arsenal of evaluation thinking, tools, methods, and processes to bear at a global level — and creating new approaches appropriate to the challenges of global systems evaluation”. This is what needs to be done as your powerful EES address confirms. .Fine tuning the DAC criteria will not solve the problems you have identified. The real challenge lies in generating a demand for Blue Marble evaluations. In turn, this means building alliances with civil society groups intent on reforming the global development enterprise,
Bob
Dear Bob, I am very pleased that you support the perspectives in this post. Like Michael Patton, you are a visionary in (development) evaluation who has always been ahead of the times. You have raised issues of policy coherence and the need for a global focus long before anyone else. In fact, I have often quoted you about these matters over the past decade, and your thinking remains as fresh and important today as 10 or 20 years ago. The critical question now is how to shift the dynamics and especially the political economy embedded in the evaluation system – or at least in the most visible parts of it – to address these issues of such crucial importance to the world. We need both the technical and political power for this purpose. It has to be a major, concerted effort by a collective of powerful people, and we are still a long way away from that. In the meantime, I remain very grateful for your many contributions to development evaluation over the decades.
Dear Zenda,
For family reasons, I couldn’t be at the EES conference, but before I got there I had reservations about what was ‘on offer’. In the keynote line-up, I could see the tried and true mostly old guard of evaluation (I respect them all and their work). I haven’t managed to make it to an EES conference, so I can’t compare it to others, but I was disappointed by this line up. I wanted to see the evaluation community being provoked, emotionally engaged, enraged – by other perspectives than our own, by young voices, by indigenous voices, by counter narratives – all of these have a major stake in the world of the future. But they weren’t all that visible on the programme.
It is my view that if, as you say, ‘Everything we do today should be framed by extreme and urgent concerns around the fact that our ‘interventions’ and their evaluation will not matter if humankind does not succeed in being less ignorant, greedy and uncaring about “the other”‘, then evaluation needs to also become more caring, more relational, more interested in counter narratives and practices, more concerned with the interconnectedness of things, etc etc.
As I said at a recent conference, to be transformational, we have to transform ourselves; we have to do our own decolonisation and reconciliation with our past dominant narratives – as these are often unhelpful as we hurtle towards the global crises you describe. And for those of us from white, privileged, colonising cultures, we have a LOT of work to do. Colonisation is not over, and it is not consensual – and we perpetuate it if we don’t urgently address how, and in what ways our practice is complicit in its maintenance.
And it’s not just development that complex, there is wickedness everywhere – and as we face the real work that is needed for change that is complex, not static, not even dynamic, we will realise that there are a whole set of new competencies and behaviours required if we are to be effective at supporting people to think and reason evaluatively about what we value, and what is valuable.
We are way finding for a totally different world. If we are make it to that place we don’t yet know about without destroying everything and everyone, we need to embrace creativity, love, trust, care, kindness, respect, honesty, generosity, spirituality, humour – these aspects of our humanity are what connects us to the earth, to the oceans and water, to the skies and heavens, to each other. We must embrace these things at every level of the system. They are the fuel of a new paradigm in my view.