**Five minute read.**
“Aid projects might yield satisfying micro-results, but they generally do little to change the systems that produce the problems in the first place.” On 16 July 2018 fifteen prominent economists, including three Nobel winners, wrote in the Guardian that – my summary – the aid system is broken because it focuses mostly on short-term, time-bounded micro-level interventions with easily observed and measurable results, does not address what they call “root” causes (from a complex systems perspective they are “deep” causes) and depends too much on the flawed notion that randomised control trials (RCTs) provide truly useful solutions for development.
They argue for much more holistic, systems-oriented approaches, coordinated policy responses and major cross-boundary efforts that will make a real difference to nations in the global south – tackling destructive trade and tax practices, ending excessive agricultural subsidies, regulating commodity derivatives markets, ensuring major coordinated responses to climate change, and so on.
The authors point to the limitations of projects with a singular variable focus such as water purification tablets, handing out bonuses to teachers and microcredit services. They argue for collective action by countries across the global south and north, and for development interventions appropriate to the challenges articulated in the 2030 Agenda with its Sustainable Development Goals.
In essence, they argue for a complex systems approach to aid and thus by implication, to the evaluation of aid initiatives.
Revisiting the past
All of this is very inspiring, but none of it is new. Hundreds if not thousands of articles, books and presentations have provided critiques of aid in exactly these terms; just analyse for example Dead Aid by Dambisa Moyo. Over several decades many voices have called for more integrated, holistic approaches in the aid system, although increasing emphasis on integrated plans in the 1980s and 1990s showed the challenges inherent in such work. The micro-macro paradox has been a long-standing point of discussion and debate. Some of us have worked to lessen the impact of the tsunami created by sellers of RCTs as the solution to all development problems: In 2007 the European Evaluation Society (EES) issued a statement on methodological diversity in impact evaluation. Also in 2007, the African Evaluation Association (AfrEA) held a special day-long session that was initiated because of the belief that RCTs would not serve African society well in the long run; the supposedly powerful (then) Network of Networks on Impact Evaluation (NONIE) a collective effort by over 200 international organisations and associations (including IOCE, ECG, UNEG and OECD DAC), produced an early guidance on impact evaluation. It was relatively open-minded about the plurality of designs available for impact evaluation, even though it did not go far enough in including non-experimental designs – something later addressed i.a. in an excellent working paper by Elliot Stern and his team for DFID, and by the InterAction series on Impact Evaluation.
But as well-known evaluator Bob Williams reminded us at the time, the RCT wave was bound to be more about politics than about technical matters, as our NONIE experience had also proven. We predicted at the time that it would run its course in about 10-15 years, and this appears to be more or less accurate – encouraged in part by the renewed emphasis on interconnections and integrated planning espoused in the 2030 Agenda with its SDGs.
What is new in the Guardian article is the collective voice of a good number of prominent economists calling quite forcefully for these types of changes in a very public space – and at a time when the chance of success is better. Perhaps the power behind these economists’ voices will be enough to cause a shift in the rather lethargic aid system – from still too linear thinking and overly rigid applications of results-based management towards greater risk-taking within complex adaptive systems approaches.
The power of economics and economists
For years I have been following developments in the discipline of (development) economics. I do this due to the outsized effect it tends to have on aid and government policies in the global south – primarily macroeconomists but increasingly also through microeconomists, most notably those driving and selling RCTs for development. Across the world many persons in powerful business as well as government positions such as planning and finance have been educated as economists in prestigious universities in North America and Europe (and in recent years advertisements for evaluation positions in the aid system have also tended to refer to economics expertise). This has helped to sustain dominant development narratives and models in spite of sometimes very dark shadows cast over their purpose and effectiveness.
The recent tremors – if not earthquake – across the economics discipline should therefore be of great interest to planners and evaluators working in development. It has led to some introspection and sometimes vociferous debates among leaders in the economics profession. The fundamentals of the discipline are being harshly questioned and defended (see also here). In the meantime, behavioural economists have been speaking out. Since around 2014 students have been campaigning to “democratise economics”, urging greater plurality in their curricula. The astounding development of China with its own context-tailored economic model has been upending a raft of conventional economic narratives and models. And prompted by the dire state of the world at the same time as that of the field, economists such as Kate Raworth of Doughnut Economics fame have called for a radical overhaul and reframing of economics based on a much more holistic (eco)systems approach.
The point is that there is a growing chance of a shift in this powerful profession – away from the overly simplistic, reductionist and/or linear reasoning and assumptions that have directed much of its work in the past, including in development in the global south. The Guardian article highlights such shifts in thinking already taking place. But we also know that this remains an issue largely of politics among the powerful (neoliberal economics versus the rest); time will tell who wins.
Will this help to bring about transformative change in the aid and aid evaluation systems?
I often give credit to the aid system in spite of being very well aware of its flaws and wrongdoings (and I have also been a vocal critic of the latter). It has done much that is good in the realm of development evaluation. I know many people at all levels of the aid and development evaluation systems who are very smart and very committed to doing what is best for the people they have to serve in the global south. In particular I dislike the mud constantly being thrown at the UN system. I have worked across many UN agencies for years while upholding my values and ethics, and appreciate the difficulties inherent in working in such a complex intergovernmental and politically charged institution.
Yet I am also very concerned about the slow pace of change, and the lack of more dynamic efforts and risk-taking experimentation to move evaluation in directions that will serve development better than the status quo is likely to allow; this is imperative in the current context where the SDGs meet the Fourth Industrial Revolution. It will be important for evaluators as well as evaluation commissioners and funders to engage with the work of new efforts such as the SDG Transformations Forum with its Transforming Evaluation Working Group, the Blue Marble evaluators initiative of evaluation pioneer Michael Quinn Patton, and the emerging South to South Evaluation (S2SE) initiative. They are all aimed at transformative systems change, and driven by the urgency that we feel around the state of the planet and the nearly 3 billion people who still live on less than US$2.50 a day – nearly all in the global south.
And very interestingly, the results of the current study on the “reform” of the DAC evaluation criteria will be a very important measure of the extent to which the development evaluation stakeholders are willing to engage with the realities that our practice is facing.