Economists, the ‘broken’ aid system and turning the RCT tide

**Five minute read.**

“Aid projects might yield satisfying micro-results, but they generally do little to change the systems that produce the problems in the first place.” On 16 July 2018 fifteen prominent economists, including three Nobel winners, wrote in the Guardian that – my summary – the aid system is broken because it focuses mostly on short-term, time-bounded micro-level interventions with easily observed and measurable results, does not address what they call “root” causes (from a complex systems perspective they are “deep” causes) and depends too much on the flawed notion that randomised control trials (RCTs) provide truly useful solutions for development.

They argue for much more holistic, systems-oriented approaches, coordinated policy responses and major cross-boundary efforts that will make a real difference to nations in the global south – tackling destructive trade and tax practices, ending excessive agricultural subsidies, regulating commodity derivatives markets, ensuring major coordinated responses to climate change, and so on.

The authors point to the limitations of projects with a singular variable focus such as water purification tablets, handing out bonuses to teachers and microcredit services. They argue for collective action by countries across the global south and north, and for development interventions appropriate to the challenges articulated in the 2030 Agenda with its Sustainable Development Goals.

In essence, they argue for a complex systems approach to aid and thus by implication, to the evaluation of aid initiatives.

Revisiting the past

All of this is very inspiring, but none of it is new. Hundreds if not thousands of articles, books and presentations have provided critiques of aid in exactly these terms; just analyse for example Dead Aid by Dambisa Moyo. Over several decades many voices have called for more integrated, holistic approaches in the aid system, although increasing emphasis on integrated plans in the 1980s and 1990s showed the challenges inherent in such work. The micro-macro paradox has been a long-standing point of discussion and debate. Some of us have worked to lessen the impact of the tsunami created by sellers of RCTs as the solution to all development problems: In 2007 the European Evaluation Society (EES) issued a statement on methodological diversity in impact evaluation. Also in 2007, the African Evaluation Association (AfrEA) held a special day-long session that was initiated because of the belief that RCTs would not serve African society well in the long run; the supposedly powerful (then) Network of Networks on Impact Evaluation (NONIE) a collective effort by over 200 international organisations and associations (including IOCE, ECG, UNEG and OECD DAC), produced an early guidance on impact evaluation. It was relatively open-minded about the plurality of designs available for impact evaluation, even though it did not go far enough in including non-experimental designs – something later addressed i.a. in an excellent working paper by Elliot Stern and his team for DFID, and by the InterAction series on Impact Evaluation.

But as well-known evaluator Bob Williams reminded us at the time, the RCT wave was bound to be more about politics than about technical matters, as our NONIE experience had also proven. We predicted at the time that it would run its course in about 10-15 years, and this appears to be more or less accurate – encouraged in part by the renewed emphasis on interconnections and integrated planning espoused in the 2030 Agenda with its SDGs.

What is new in the Guardian article is the collective voice of a good number of prominent economists calling quite forcefully for these types of changes in a very public space – and at a time when the chance of success is better. Perhaps the power behind these  economists’ voices will be enough to cause a shift in the rather lethargic aid system – from still too linear thinking and overly rigid applications of results-based management towards greater risk-taking within complex adaptive systems approaches.

The power of economics and economists

For years I have been following developments in the discipline of (development) economics. I do this due to the outsized effect it tends to have on aid and government policies in the global south – primarily macroeconomists but increasingly also through microeconomists, most notably those driving and selling RCTs for development. Across the world many persons in powerful business as well as government positions such as planning and finance have been educated as economists in prestigious universities in North America and Europe (and in recent years advertisements for evaluation positions in the aid system have also tended to refer to economics expertise). This has helped to sustain dominant development narratives and models in spite of sometimes very dark shadows cast over their purpose and effectiveness.

The recent tremors – if not earthquake – across the economics discipline should therefore be of great interest to planners and evaluators working in development. It has led to some introspection and sometimes vociferous debates among leaders in the economics profession. The fundamentals of the discipline are being harshly questioned and defended (see also here). In the meantime, behavioural economists have been speaking out. Since around 2014 students have been campaigning to “democratise economics”, urging greater plurality in their curricula. The astounding development of China with its own context-tailored economic model has been upending a raft of conventional economic narratives and models. And prompted by the dire state of the world at the same time as that of the field, economists such as Kate Raworth of Doughnut Economics fame have called for a radical overhaul and reframing of economics based on a much more holistic (eco)systems approach.

The point is that there is a growing chance of a shift in this powerful profession – away from the overly simplistic, reductionist and/or linear reasoning and assumptions that have directed much of its work in the past, including in development in the global south. The Guardian article highlights such shifts in thinking already taking place. But we also know that this remains an issue largely of politics among the powerful (neoliberal economics versus the rest); time will tell who wins.

Will this help to bring about transformative change in the aid and aid evaluation systems?

I often give credit to the aid system in spite of being very well aware of its flaws and wrongdoings (and I have also been a vocal critic of the latter). It has done much that is good in the realm of development evaluation. I know many people at all levels of the aid and development evaluation systems who are very smart and very committed to doing what is best for the people they have to serve in the global south. In particular I dislike the mud constantly being thrown at the UN system. I have worked across many UN agencies for years while upholding my values and ethics, and appreciate the difficulties inherent in working in such a complex intergovernmental and politically charged institution.

Yet I am also very concerned about the slow pace of change, and the lack of more dynamic efforts and risk-taking experimentation to move evaluation in directions that will serve development better than the status quo is likely to allow; this is imperative in the current context where the SDGs meet the Fourth Industrial Revolution. It will be important for evaluators as well as evaluation commissioners and funders to engage with the work of new efforts such as the SDG Transformations Forum with its Transforming Evaluation Working Group, the Blue Marble evaluators initiative of evaluation pioneer Michael Quinn Patton, and the emerging South to South Evaluation (S2SE) initiative. They are all aimed at transformative systems change, and driven by the urgency that we feel around the state of the planet and the nearly 3 billion people who still live on less than US$2.50 a day – nearly all in the global south.

And very interestingly, the results of the current study on the “reform” of the DAC evaluation criteria will be a very important measure of the extent to which the development evaluation stakeholders are willing to engage with the realities that our practice is facing.

Share this article

Zenda Ofir

Zenda Ofir is an independent South African evaluator. Based near Geneva, she works across Africa and Asia. A former AfrEA President, IOCE Vice-President and AEA Board member, she is at present IDEAS Vice-President, Lead Steward of the SDG Transformations Forum A&E Working Group and Honorary Professor at Stellenbosch University.

5 Comments

  1. Zenda, Thank you for once again providing big picture perspective and integrating multiple pathways towards transformation. The work continues on many fronts even as the urgency deepens. MQP

  2. Thank you Michael. As you point out, a deep sense of urgency is essential. One aspect I have to highlight more robustly is that several well-known economists have in the past called for the type of approaches espoused in the article. The difference is that collective efforts by prominent economists from a range of backgrounds – such as this one – might have a better chance of making a difference. I do hope that it will help give impetus to new practices in the aid system. Doing Development Differently is already going on; Doing Evaluation Differently still needs to infuse the system.

  3. Hi Zenda,

    Thanks for your relevant and rich (the links are very useful) feedback on the Guardian article. As an aspiring, young evaluator, I’m getting increasingly intrigued by this ‘systems’ thinking you (and others) increasingly refer to. Would you have any entry-level reading recommendations on systems-thinking to get a good understanding of it and develop a sort of ‘mental framework’ for applying it in my work please? Many thanks in advance.

  4. Gregory, the easiest yet substantive relevant reads that I can recommend are the books by Danny Burns and Stuart Worsley, Navigating Complexity in International Development or Ben Ramalingham’s Aid on the Edge of Chaos. I recommend them as they are written in reader-friendly ways and immediately put you in the context of complex systems in development,with good explanations of basic concepts and why it is important. You can check basic materials such as blogs on the internet – see for example the posts in the series by Leyla Acaroglu on Medium https://medium.com/disruptive-design/tools-for-systems-thinkers-the-6-fundamental-concepts-of-systems-thinking-379cdac3dc6a. Then there are articles, podcasts, videoclips and books specifically in the evaluation realm – most prominently by Bob Williams who was the first to write extensively about it in evaluation. You can learn almost everything you need to know from his comments and articles and books – you will find them on the Internet, and they are mostly written in quite an accessible style. Many people like to refer to the Cynefin framework of Dave Snowden as basic insight. To deepen your evaluation-specific knowledge around systems there are articles, books, blogs etc. by people like Michael Bamberger, Jos Vaessen, Kim Forss, Jonny Morell, Ray Pawson and Patricia Rogers. Good groups doing interesting work are CDI / CECAN in the UK and CoPs such as ClimateEval and the Systems (I)TIGs of IDEAS and AEA that should have scores of materials available. My personal favourite book is The systems view of life by Capra and Luisi.
    I know this is a lot of information but hope you will be inspired to delve into some of them. I am sure other readers can also recommend specific articles they like as starting points.

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.