The Power(lessness) of Evaluation: The case of Afghanistan

Share this post

**10 min read**

Thanks to more than 60 evaluation reports over 13 years by the Special Inspector General for Afghanistan Reconstruction (SIGAR), we have a perfect example of how valuable good evaluation could have been to avoid the catastrophe that was Afghanistan over the last two decades. Instead it displays a key contradiction in evaluation as practice: its power to help accelerate change, and its powerlessness in the face of unsavoury political agendas, hubris, obfuscation, ignorance, corruption or haplessness – all of which played out in Afghanistan.

The July 2021 Q2 report with the great title, “Doing the wrong thing perfectly”, is illuminating; it dissects the (very flawed) M&E approaches of US agencies who managed Afghanistan’s reconstruction contracts. But of even greater interest is the truly devastating synthesis published on 16 August - What We need to Learn: Lessons from Twenty Years of Afghanistan Reconstruction.

Hopefully all of us - especially evaluation commissioners - will take note of what these findings mean for the field of evaluation.

The cost of an interventionist war

What could have been gained if SIGAR’s reports were taken seriously?

The costs of ignoring the evaluative findings for more than a decade are staggering. Brown University Costs of War project is widely seen as the most authoritative data source on the war in Afghanistan:

US$2.2 trillion = US$300 million/day for 20 years
“Warfighting” costs by the US DOD US$1.38 trillion
Interest on war financing debt US$500 billion, and ongoing
Medical care for 20,000 wounded Americans US$300 billion, and ongoing
Afghan military training (collapsed in 11 days) US$85 billion
Reconstruction of Afghanistan US$145 billion (much went back to US pockets)
Total cost in lives 241,000 (at least) – 80 times those of 9/11
Afghan civilians More than 71,000
Afghan soldiers and police More than 78,000
Opposition (mostly Taliban) fighters More than 84,000
US service members Around 2,400
Allied troops Around 1,100
US contractors Around 4,000
Aid workers Around 440
Journalists Around 70

These numbers exclude deaths from disease; from loss of access to food, water and infrastructure; or from any other indirect consequences of the war. Nearly 7 million - around 20% of the population - were internally displaced or fled the country. The suffering and death in Afghanistan will certainly continue; the West tends to bring their economic tools to bear to punish those that oppose them: 80% of Afghanistan’s reserves as well as promised aid have already been “frozen” by the US, aid withdrawn by Europe and the IMF, and so on. Expect many more similar actions to come – including sanctions aimed at devastating the economy.

Some interesting titbits that reflect hubris and ignorance at best, and corruption at worst. During the Vietnam war, US lawmakers discussed the costs of the conflict 42 times. In the past 20 years, this was done only 5 times for the costs of the Afghanistan and Iraq wars. And the top tax rates the wealthiest Americans had to pay for their main wars: 92% for the Korean War and 77% for the Vietnam War. In contrast, Bush cut tax on the wealthiest by at least 8% at the outset of the Afghanistan and Iraq wars.

It is amazing that US citizens are not in their millions on the streets in outrage at it all.

The gains?

There have been some gains, but not nearly to the extent generally perceived. Health care, maternal mortality and education have improved. Most impressive is the drop in infant mortality rate, cut by half. Life expectancy improved slightly more than that of rest of the world over the same time.

But “progress has been elusive and the prospects for sustaining this progress are dubious.” And …. “After 13 years of oversight, the cumulative list of systemic challenges SIGAR and other oversight bodies have identified is staggering”, says the SIGAR report, written before the Taliban took over. Afghanistan still ranks second to last in the world in the Women Peace and Security Index. Human rights improved somewhat from -2.51 to -2.31, yet the country slipped five places on the Human Development Index despite improvements. The poverty rate increased from 33.6% in 2007 to 54.5% in 2016.

Numbers must be interpreted with nuance, and the ‘back story’ well understood. Afghanistan was already on an upward development trajectory since the Russians left in 1989. Crucially, the positive trends disguise the main flaw in so many simplistic indicators - including 'GDP growth': a country dependent on aid will remain fragile, and much of the progress will reverse unless huge care is taken to do programmes that will increase the chance of positive change that will sustain when the financing stops.

Lessons from the SIGAR synthesis report

There are seven main takeaways from the August 2021 synthesis report - each with many lessons embedded. I interpret them here. Evaluators, and especially evaluation commissioners, can use them as food for thought to see whether their approaches and criteria would have focused on these.

First, strategy. Have a coherent strategy with enough flexibility to change tactics when necessary, but not direction - if the overall goal is sound. Mobilise appropriate organisations across sectors and other types of silos to support the strategy in an integrated way, recognising important connections between actions and their results. Make sure that what is done does not maintain or worsen vulnerabilities in the society and in the systems that affect their lives.

Second, timelines. Think and support development with a long-term vision. A devastating statement by SIGAR: “The U.S. reconstruction effort in Afghanistan could be described as 20 one-year reconstruction efforts, rather than one 20-year effort”. ‘Impact deadlines’ are unacceptable when accompanied by unrealistic expectations and too few resources. Don’t push own agendas and narratives onto other people. Look out for perverse incentives based on simplistic key performance indicators (KPIs), such as needing to spend money quickly to demonstrate that something is being done, or having to show progress in the face of short-term, unsustainable results.

Third, sustainability. Even more than development, reconstruction is supposed to provide a strong foundation for the development of the institutions and systems on which a society depends. This needs to be set in place through appropriate approaches, timelines and incentives. But much depends on how ‘success’ is defined and measured, on incentives structures, and on whether those responsible are prepared to grapple with solutions to systemic problems. Says SIGAR, “Demands to make fast progress incentivised U.S. officials to identify and implement short-term projects with little consideration for host government capacity and long-term sustainability. U.S. agencies were seldom judged by their projects’ continued utility, but by the number of projects completed and dollars spent”, says SIGAR.

This once again confirms my firm standpoint that no intervention in the lives of others or in nature should be done unless the sustainability of positive progress is carefully considered – and then also not through the simplistic ‘capacity building’ initiatives so rife in Western aid programmes. In my view, assessments based on this evaluation criterion – or on ‘adaptive sustainability’ espoused in Blue Marble Evaluation – have to be positive before almost any intervention to be assessed as successful.

Fourth, people. Getting enough resources and the right people in time in crucial jobs are clearly prerequisites for progress. This has been a miserable failure in Afghanistan, and shows the dismissive attitudes of the occupying forces towards Afghan society: “DOD police advisors watched American TV shows to learn about policing, civil affairs teams were mass-produced via PowerPoint presentations, and every agency experienced annual lobotomies as staff constantly rotated out, leaving successors to start from scratch and make similar mistakes all over again.” The DOD also did not work in synergy with USAID and others.

Fifth, connections. When one pivotal thing goes wrong, one that influences many others, don’t cast around for inappropriate short-term solutions or compromises. Look for negative consequences. Recognise inter-dependencies. Focus on finding smart rather than dumb tactics otherwise everything is badly affected.

Clearly, understanding complex adaptive systems concepts is not a fad or luxury under these circumstances; it is absolutely essential for success.

Sixth, context and culture. One should be outraged by the following statement in the SIGAR report - an extreme example of the damage done when culture and context are ignored, and when dominant narratives about societies and about change are miserably wrong: “The US government also clumsily forced Western technocratic models onto Afghan economic institutions; trained security forces in advanced weapon systems they could not understand, much less maintain; imposed formal rule of law on a country that addressed 80 to 90 percent of its disputes through informal means; and often struggled to understand or mitigate the cultural and social barriers to supporting women and girls. Without this background knowledge, U.S. officials often empowered powerbrokers who preyed on the population or diverted U.S. assistance away from its intended recipients to enrich and empower themselves and their allies. Lack of knowledge at the local level meant projects intended to mitigate conflict often exacerbated it, and even inadvertently funded insurgents.”

Seventh, monitoring and evaluation. “The absence of periodic reality checks created the risk of doing the wrong thing perfectly”, says SIGAR eloquently. The litany of problems with the M&E of reconstruction contracts in Afghanistan is familiar: US government agencies struggled to measure results and used “shaky data” to claim success, and M&E were neglected because of a focus on doing as much as possible as quickly as possible. As a result, programmes were poorly designed and could not be adapted as circumstances evolved. “The US government missed many opportunities to identify critical flaws in its interventions or to act on those that were identified. These shortcomings endangered the lives of US, Afghan, and coalition government personnel and civilians, and undermined progress toward strategic goals.”

Devastating.

Implications for what we do

More than anything, we need to be aware of subtle narratives that might not reflect reality - and that will influence our view of the world, and hence our practice. Narratives are continuously being created by the media and others to shape the mental models in our heads, controlling how we think and what we do. One that will soon emerge is that failures will be the fault of the Afghan nation and their government - not the litany of failures over two decades recorded by SIGAR. The media will continue to propagate micro narratives such as that 'poor countries' need 'white saviours' to be successful. Or that anything that does not reflect 'our values' or our notion of 'development' is inferior.

But for evaluation commissioners, and evaluation specialists in general, the experience in Afghanistan shows why we need to focus on using thoughtful evaluation criteria and questions, especially when dealing with international aid portfolios. Responses to some of my favourite evaluation questions can be very revealing, for example -

One: How has ‘success’ in this intervention (or set of interventions, i.e. portfolios of projects, programmes, strategies, etc.) been defined, by whom, when and how? How appropriate is this definition given the ultimate intent of the interventions?

Two: How well do the intervention approaches, strategies and tactics reflect a well-reasoned understanding of how (systems) change may happen in these contexts? On which and whose perspectives and narratives is this based?

Three: To what extent have essential preconditions for success been (put) in place before and during implementation? If not in place, what are the implications?

Four: Were appropriate and sufficient steps taken to ensure that the results of the interventions have a good chance to sustain, or contribute to a trajectory of positive change beyond the scope and duration of support? If so, how and if not, why not?

Five: To what extent were risks, trade-offs and (potentially) negative consequences of decisions and actions accounted for and mitigated? Where this was not sufficiently done, what are the implications?

Six: How well were appropriate connections made, and/or alignments created, to enhance the chance and scope of success?

Seven: What has been the effect of power asymmetries and other factors that might prevent desired shifts (in systems)? How well have these been accounted for?

And for the evaluation itself: Have sufficient steps been taken to ensure the quality, usefulness and use of the evaluation?

In subsequent posts I will take such evaluation questions, and more, and dissect their implications for intervention design as well as for evaluation practice, especially in the context of transformation.


Share this post

6 thoughts on “The Power(lessness) of Evaluation: The case of Afghanistan”

  1. isha wedasinghe miranda

    Dear Zenda,
    Good analysis .
    one of the key issue was , that non of the programs or that matter projects of infrastructure did not follow the basic criteria’s. issues were that down poring trillions of dollars to the country, went in to bad governance in all aspects, including the governments, private sectors as well as NGO’s. None were designed (interventions) to prevent conflict sensitivities, or prevent bad corruptions governance from all sectors and other issues was connectivity to afgan people’s needs.
    Very sadly most agencies evaluations reports so unprofessional, lack of focus and lack of competencies in resources.
    The behavior’s were unnoticed, needs were unnoticed, but programs, projects flooding without or lack of mechanism in Monitoring and evaluation. It is still happening with our evaluation communities.

  2. Zenda, Thank you for bringing these important evaluations to our collective attention.
    Ineffectiveness is compounded by deceit and secrecy as documented in The Afghanistan Papers.
    https://www.washingtonpost.com/graphics/2019/investigations/afghanistan-papers/afghanistan-war-confidential-documents/
    The war against truth is insidious ad longstanding. It’s not just a matter of speaking truth to power, for they known the truth, but hide it from the public. Exposing the truth is a prerequisite to speaking it.
    MQP

  3. Wonderful piece Zenda. Much to learn and take away from this devastating situation. Can I ask a very simplistic question though. Regardless of the evaluation questions, quality of evaluation, dollars that go into them, were the reports ever read by people in positions to do something with the learnings and recommendations?

  4. Tiroyaone Kebalepile

    A very informative and practical analysis…The issue of context and culture cannot be overemphasized. The need to engage the evaluand or the intended intervention beneficiary form the beginning of the intervention throughout to the end is critical. Some kind of ‘MADE IN AFGHAN..’ as in Made In Africa (MAE) would have yielded much better results. Ownership and empowerment would have achieved at least to a greater extent….The ‘Power’ would have rested with the Afghan people. This article is a must read for evaluators as it allows us to reflect in our own practice and ACT. This will allow us to inform and guide sustainable development which is actually socially situated.

Leave a Comment

Your email address will not be published. Required fields are marked *

Scroll to Top