**10 min read**
Thanks to more than 60 evaluation reports over 13 years by the Special Inspector General for Afghanistan Reconstruction (SIGAR), we have a perfect example of how valuable good evaluation could have been to avoid the catastrophe that was Afghanistan over the last two decades. Instead it displays a key contradiction in evaluation as practice: its power to help accelerate change, and its powerlessness in the face of unsavoury political agendas, hubris, obfuscation, ignorance, corruption or haplessness – all of which played out in Afghanistan.
The July 2021 Q2 report with the great title, “Doing the wrong thing perfectly”, is illuminating; it dissects the (very flawed) M&E approaches of US agencies who managed Afghanistan’s reconstruction contracts. But of even greater interest is the truly devastating synthesis published on 16 August - What We need to Learn: Lessons from Twenty Years of Afghanistan Reconstruction.
Hopefully all of us - especially evaluation commissioners - will take note of what these findings mean for the field of evaluation.
The cost of an interventionist war
What could have been gained if SIGAR’s reports were taken seriously?
The costs of ignoring the evaluative findings for more than a decade are staggering. Brown University Costs of War project is widely seen as the most authoritative data source on the war in Afghanistan:
|US$2.2 trillion = US$300 million/day for 20 years|
|“Warfighting” costs by the US DOD||US$1.38 trillion|
|Interest on war financing debt||US$500 billion, and ongoing|
|Medical care for 20,000 wounded Americans||US$300 billion, and ongoing|
|Afghan military training (collapsed in 11 days)||US$85 billion|
|Reconstruction of Afghanistan||US$145 billion (much went back to US pockets)|
|Total cost in lives||241,000 (at least) – 80 times those of 9/11|
|Afghan civilians||More than 71,000|
|Afghan soldiers and police||More than 78,000|
|Opposition (mostly Taliban) fighters||More than 84,000|
|US service members||Around 2,400|
|Allied troops||Around 1,100|
|US contractors||Around 4,000|
|Aid workers||Around 440|
These numbers exclude deaths from disease; from loss of access to food, water and infrastructure; or from any other indirect consequences of the war. Nearly 7 million - around 20% of the population - were internally displaced or fled the country. The suffering and death in Afghanistan will certainly continue; the West tends to bring their economic tools to bear to punish those that oppose them: 80% of Afghanistan’s reserves as well as promised aid have already been “frozen” by the US, aid withdrawn by Europe and the IMF, and so on. Expect many more similar actions to come – including sanctions aimed at devastating the economy.
Some interesting titbits that reflect hubris and ignorance at best, and corruption at worst. During the Vietnam war, US lawmakers discussed the costs of the conflict 42 times. In the past 20 years, this was done only 5 times for the costs of the Afghanistan and Iraq wars. And the top tax rates the wealthiest Americans had to pay for their main wars: 92% for the Korean War and 77% for the Vietnam War. In contrast, Bush cut tax on the wealthiest by at least 8% at the outset of the Afghanistan and Iraq wars.
It is amazing that US citizens are not in their millions on the streets in outrage at it all.
There have been some gains, but not nearly to the extent generally perceived. Health care, maternal mortality and education have improved. Most impressive is the drop in infant mortality rate, cut by half. Life expectancy improved slightly more than that of rest of the world over the same time.
But “progress has been elusive and the prospects for sustaining this progress are dubious.” And …. “After 13 years of oversight, the cumulative list of systemic challenges SIGAR and other oversight bodies have identified is staggering”, says the SIGAR report, written before the Taliban took over. Afghanistan still ranks second to last in the world in the Women Peace and Security Index. Human rights improved somewhat from -2.51 to -2.31, yet the country slipped five places on the Human Development Index despite improvements. The poverty rate increased from 33.6% in 2007 to 54.5% in 2016.
Numbers must be interpreted with nuance, and the ‘back story’ well understood. Afghanistan was already on an upward development trajectory since the Russians left in 1989. Crucially, the positive trends disguise the main flaw in so many simplistic indicators - including 'GDP growth': a country dependent on aid will remain fragile, and much of the progress will reverse unless huge care is taken to do programmes that will increase the chance of positive change that will sustain when the financing stops.
Lessons from the SIGAR synthesis report
There are seven main takeaways from the August 2021 synthesis report - each with many lessons embedded. I interpret them here. Evaluators, and especially evaluation commissioners, can use them as food for thought to see whether their approaches and criteria would have focused on these.
First, strategy. Have a coherent strategy with enough flexibility to change tactics when necessary, but not direction - if the overall goal is sound. Mobilise appropriate organisations across sectors and other types of silos to support the strategy in an integrated way, recognising important connections between actions and their results. Make sure that what is done does not maintain or worsen vulnerabilities in the society and in the systems that affect their lives.
Second, timelines. Think and support development with a long-term vision. A devastating statement by SIGAR: “The U.S. reconstruction effort in Afghanistan could be described as 20 one-year reconstruction efforts, rather than one 20-year effort”. ‘Impact deadlines’ are unacceptable when accompanied by unrealistic expectations and too few resources. Don’t push own agendas and narratives onto other people. Look out for perverse incentives based on simplistic key performance indicators (KPIs), such as needing to spend money quickly to demonstrate that something is being done, or having to show progress in the face of short-term, unsustainable results.
Third, sustainability. Even more than development, reconstruction is supposed to provide a strong foundation for the development of the institutions and systems on which a society depends. This needs to be set in place through appropriate approaches, timelines and incentives. But much depends on how ‘success’ is defined and measured, on incentives structures, and on whether those responsible are prepared to grapple with solutions to systemic problems. Says SIGAR, “Demands to make fast progress incentivised U.S. officials to identify and implement short-term projects with little consideration for host government capacity and long-term sustainability. U.S. agencies were seldom judged by their projects’ continued utility, but by the number of projects completed and dollars spent”, says SIGAR.
This once again confirms my firm standpoint that no intervention in the lives of others or in nature should be done unless the sustainability of positive progress is carefully considered – and then also not through the simplistic ‘capacity building’ initiatives so rife in Western aid programmes. In my view, assessments based on this evaluation criterion – or on ‘adaptive sustainability’ espoused in Blue Marble Evaluation – have to be positive before almost any intervention to be assessed as successful.
Fourth, people. Getting enough resources and the right people in time in crucial jobs are clearly prerequisites for progress. This has been a miserable failure in Afghanistan, and shows the dismissive attitudes of the occupying forces towards Afghan society: “DOD police advisors watched American TV shows to learn about policing, civil affairs teams were mass-produced via PowerPoint presentations, and every agency experienced annual lobotomies as staff constantly rotated out, leaving successors to start from scratch and make similar mistakes all over again.” The DOD also did not work in synergy with USAID and others.
Fifth, connections. When one pivotal thing goes wrong, one that influences many others, don’t cast around for inappropriate short-term solutions or compromises. Look for negative consequences. Recognise inter-dependencies. Focus on finding smart rather than dumb tactics otherwise everything is badly affected.
Clearly, understanding complex adaptive systems concepts is not a fad or luxury under these circumstances; it is absolutely essential for success.
Sixth, context and culture. One should be outraged by the following statement in the SIGAR report - an extreme example of the damage done when culture and context are ignored, and when dominant narratives about societies and about change are miserably wrong: “The US government also clumsily forced Western technocratic models onto Afghan economic institutions; trained security forces in advanced weapon systems they could not understand, much less maintain; imposed formal rule of law on a country that addressed 80 to 90 percent of its disputes through informal means; and often struggled to understand or mitigate the cultural and social barriers to supporting women and girls. Without this background knowledge, U.S. officials often empowered powerbrokers who preyed on the population or diverted U.S. assistance away from its intended recipients to enrich and empower themselves and their allies. Lack of knowledge at the local level meant projects intended to mitigate conflict often exacerbated it, and even inadvertently funded insurgents.”
Seventh, monitoring and evaluation. “The absence of periodic reality checks created the risk of doing the wrong thing perfectly”, says SIGAR eloquently. The litany of problems with the M&E of reconstruction contracts in Afghanistan is familiar: US government agencies struggled to measure results and used “shaky data” to claim success, and M&E were neglected because of a focus on doing as much as possible as quickly as possible. As a result, programmes were poorly designed and could not be adapted as circumstances evolved. “The US government missed many opportunities to identify critical flaws in its interventions or to act on those that were identified. These shortcomings endangered the lives of US, Afghan, and coalition government personnel and civilians, and undermined progress toward strategic goals.”
Implications for what we do
More than anything, we need to be aware of subtle narratives that might not reflect reality - and that will influence our view of the world, and hence our practice. Narratives are continuously being created by the media and others to shape the mental models in our heads, controlling how we think and what we do. One that will soon emerge is that failures will be the fault of the Afghan nation and their government - not the litany of failures over two decades recorded by SIGAR. The media will continue to propagate micro narratives such as that 'poor countries' need 'white saviours' to be successful. Or that anything that does not reflect 'our values' or our notion of 'development' is inferior.
But for evaluation commissioners, and evaluation specialists in general, the experience in Afghanistan shows why we need to focus on using thoughtful evaluation criteria and questions, especially when dealing with international aid portfolios. Responses to some of my favourite evaluation questions can be very revealing, for example -
One: How has ‘success’ in this intervention (or set of interventions, i.e. portfolios of projects, programmes, strategies, etc.) been defined, by whom, when and how? How appropriate is this definition given the ultimate intent of the interventions?
Two: How well do the intervention approaches, strategies and tactics reflect a well-reasoned understanding of how (systems) change may happen in these contexts? On which and whose perspectives and narratives is this based?
Three: To what extent have essential preconditions for success been (put) in place before and during implementation? If not in place, what are the implications?
Four: Were appropriate and sufficient steps taken to ensure that the results of the interventions have a good chance to sustain, or contribute to a trajectory of positive change beyond the scope and duration of support? If so, how and if not, why not?
Five: To what extent were risks, trade-offs and (potentially) negative consequences of decisions and actions accounted for and mitigated? Where this was not sufficiently done, what are the implications?
Six: How well were appropriate connections made, and/or alignments created, to enhance the chance and scope of success?
Seven: What has been the effect of power asymmetries and other factors that might prevent desired shifts (in systems)? How well have these been accounted for?
And for the evaluation itself: Have sufficient steps been taken to ensure the quality, usefulness and use of the evaluation?
In subsequent posts I will take such evaluation questions, and more, and dissect their implications for intervention design as well as for evaluation practice, especially in the context of transformation.
Zenda Ofir is an independent South African evaluator at present based near Geneva. She works primarily in Africa and Asia, and advises organisations around the world. She is a former AfrEA President, IOCE and IDEAS Vice-President, AEA Board member, Honorary Professor at Stellenbosch University, Richard von Weizsäcker Fellow, and at present Interim Council Chair of the new International Evaluation Academy.