Snippet: Judging ‘success’. The example of capacity development

How good are we at defining and judging ‘success’ – the essence of what evaluators do? Let us consider a pertinent example: Every year the aid industry spends a quarter of its funding, around US$15 billion, on ‘capacity development’ (CD). So-called ‘state-building’ is especially popular. Inevitably, evaluations in the Global South very frequently focus on this aspect.

A 2017 report by the Secure Livelihoods Research Consortium highlights a series of studies conducted over six years on diverse international efforts to build state capacity for service delivery in eight fragile, conflict-affected countries. Service delivery is a core function of a modern state, and the study was aimed at understanding whether the dominant models used for capacity development were fit for this important purpose.

The results make for uncomfortable reading. You can read the whole report (it is worthwhile to do so) or you can have quick read here. It finds that the same mistakes have persisted over decades, and that planners and implementers do not engage sufficiently with those limitations that cause the inertia in the system. They also overlook the implications of shortcomings in the dominant approaches to capacity development.

So do we when we evaluate.

Flaws in CD strategies, and the implications for evaluation

Their findings have the following implications for evaluators and evaluation commissioners, with reference to state capacity development, but the issues are applicable in other contexts too:

Evaluations need to focus on how change really happens, beyond training and ‘gaining knowledge’. “Too often, grand efforts to rebuild states and societies boil down to training or providing equipment and resources.” Programming tends to favour short-term training, and its tangible and quantifiable approaches. Yet getting more knowledge is only part of a much bigger picture. Many of the training initiatives have internal design weaknesses such as flawed learning methodologies, too short timeframes and insufficient localisation. The authorising environment in which new skills and knowledge have to be applied has to be taken into account. Intangibles such as values, vision, leadership, management style, and organisational culture are at least as important as ‘countable’ CD activities.

Evaluations need to consider the underlying power, politics and systems that prevent success.Power and politics are central to how services are delivered, but capacity development often concentrates on technical aspects only.” State capacity to delivery services is shaped by factors related to power and politics, such as elite behaviour or the invisible forces of informal institutions and social norms. The incentives and influence of local power brokers are ignored, or the asymmetries in relationships within the household, for example.

Evaluations need to focus beyond the obvious in understanding how new capacities are applied in practice, recognising existing limitations. In particular in the illustrative cases in the report, relational aspects of development, for example how citizens view, trust and interact with different parts of government across different locations are neglected, despite playing a strong role in people’s perception and experience of services. People’s decision-making incentives and the breadth of providers people rely on, often extend beyond the state. Under-investments are made at district or provincial level.

Evaluations need to think more holistically about the capacity of systems. “By focusing on individuals and organisations, capacity development is typically approached as a modular exercise, assuming micro-capacities naturally aggregate up to build better systems.” With a focus on individuals or organisations, the systems themselves – say a network of organisations with formal regulations and mechanisms, and their power relations, informal institutions, social interactions and other contextual influences – receive inadequate attention. Tweaking one part of a system is not necessarily enough. It is important to consider what else needs to be in place for the new capacities to make a real difference.

Limitations persist because the aid system itself is flawed. Staff are hired for their technical skills rather than contextual or conceptual knowledge. Short timeframes have to support long-term change processes. There is a focus on the tangible and quantifiable. Accountability for aid is to donor publics, not to beneficiaries. There is a push for results and a diminishing appetite for risk. Social change is projectised. “The aid industry turns fundamentally political processes of social and institutional change into projects – which are in turn cloaked in value-neutral technocracy.”

What does this mean for our understanding and assessment of ‘success’?

The problem is that I hardly ever see evaluations of capacity development with a robust focus on these issues. Yet many CD evaluations make quite positive judgments of progress and results without every studying the issues discussed in this report.

This is an example of a critical matter that deserves our attention: ‘Success’ is often defined in a way that is too limited, even just plain wrong. So our assessment of it is flawed.

Success is most often seen as achieving objectives or getting within a specified timeframe to some pre-conceived results (outcomes or impacts) without considering whether they have been appropriately conceptualized in the first place.

Or it is seen as any kind of positive impact – even if the ‘impact’ is only a very incremental, even minute change, or one that is easily reversed (cue: what if in month or year’s time there is no sign of the change, or of other positive changes flowing from the initial change?).

Or when an intervention satisfies some of the (DAC) criteria, without any summative assessment of contributions to development for example (where the understanding of “development” is clearly defined).

And as this report points out, without considering important aspects known to prevent success in a specific domain: misunderstanding how change really happens; forgetting about power asymmetries; or ignoring the larger systems within which new capacities need to be applied. People need to have the opportunity to exert their agency.

And capacity development is by no means the only domain with these problems.

In essence – many of us do not do enough systems thinking when we commission or do evaluations. And as the report points out, there may be reasons why people choose to ignore a better definition of ‘success’.

Then there is the fact that we do not have the incentives – or is it knowledge and methods? – to question the overarching models that direct ideas about why, how, when, for whom, and what capacities are supposed to be developed. For example, conventional models argue that strong institutions are an imperative for development. China’s development over three decades has just convincingly proven the opposite. Yet there is still a major focus on ‘institution building’ in the aid environment. But that is another story ….

Share this Article

Zenda Ofir

Zenda Ofir is an independent South African evaluator. Based near Geneva, she works across Africa and Asia. A former AfrEA President, IOCE Vice-President and AEA Board member, she is at present IDEAS Vice-President, Lead Steward of the SDG Transformations Forum A&E Working Group and Honorary Professor at Stellenbosch University.

7 Comments

  1. I was surprised that the study does not cite three 2013-2014 cases of capacity-building outcomes of the World Bank Institute, then the capacity building arm of the WB Group:

    Burundi: Develop leadership for developing capacity and delivering results toward public sector reform in Burundi

    Kenya, Tanzania and Uganda: Improve governance in pharmaceutical procurement and supply chain management in Kenya, Tanzania and Uganda

    Africa: Strengthen parliamentary oversight of national budgets in Africa

    These assessments were done using Outcome Harvesting, which defines “outcome” as a change in behaviour of a societal actor, which in these cases were virtually all state actors of those countries. Although the purpose was to pilot Outcome Harvesting for the WB, the achievements of the ten capacity-building projects with state actors represented by roughly 30 outcomes per project are not as grim as the Secure Livelihoods study found. They may be, of course, outliers but the full set of ten cases, including these three, are available at https://openknowledge.worldbank.org/handle/10986/20015.

    • Ricardo
      Your comment gives me the opportunity to highlight your big contribution in bringing us outcome harvesting, which I encourage everyone to consider in their practice. The cases to which you refer also have very rich material on the topic. On the other hand, you may still agree with my main contention in the post, which is not specifically about this particular report or even about CD. My problem is with how glib we are about “success” – especially in terms of what we believe to be successful contributions to development. In addition – even if there are many positive outcomes without neutralising negative ones, we fail to consider for example their significance, or whether together they can be assessed as a successful contribution to development. Even if it is difficult to assess or predict, I firmly believe we should engage with the idea of “significance” from a development perspective – at least in the Global South, where fiddling while Rome burns (i.e., planners and evaluators being too busy with micro interventions that are not really contributing to development) is not exactly a good strategy.

      • Zenda,

        Thank you for being so supportive of Outcome Harvesting.

        Yes, I certainly agree that there is a tendency for evaluators and commissioners who are dependent on bi- and multi-lateral funding to be glib and worse, often reductionist and arrogant about what constitutes “success”. Personally, my biggest concern is the enchantment with numbers — the bigger the better. Even in Outcome Harvesting, which is basically a qualitative evaluation methodology, I find that often people try and inappropriately squeeze out numbers — for example, the numbers of beneficiaries or people with whom an intervention works — instead of making sense of the quality (the nature) of the outcomes and how an intervention influenced them. Equally concerning, too often evaluators and commissioners give value to the number rather than the quality of positive and negative outcomes that have been harvested. The real added value of evaluating outcomes is that it permits you to see them in relation to each other, revealing processes and patterns of change.

        So yes again, quality has everything to do with the significance of the outcomes for development or social change — answering the question not just what was achieved and how, but so what does it mean?

        • Ricardo
          Your comment that working with outcomes permits one to “see them in relation to each other, revealing processes and patterns of change” is extremely important, yet often not considered. We have to encourage greater emphasis on this type of work. About numbers: We have a strange dichotomy. Everyone in communication knows the power of stories – apparently proven in psychology. The media/journalists are very good (and irritating) in this respect – highlighting one story to supposedly highlight a trend or common situation. In evaluation and related activities numbers have the most power, yet people also want stories to give the numbers life. The psychology AND politics of numbers vs stories warrant some more attention by evaluators.

  2. Two points on an interesting issue. First, the capacity and position of donors to work on enabling environment issues is challenging. For it is often in the enabling environment that actors are blocked from action. The laws and regulations that govern work in a country are not easily addressed through donor initiatives; indeed donor intervention in laws and regulations is often see as inappropriate. It requires working with and through local advocates, be they in government, CSOs or the private sector. And it is a slow, uncertain process that does not fit well with donor funding regimes and timing.
    Second, in working with theory of change – which is how many CB initiatives operate – we do not have a central registry a la Campbell Collaboration in which we can deposit a theory of change and related evaluations so that meta evaluation can more easily be done. We only know about Ricardo’s work because he mentions it here. How many organizations and evaluators are aware of it? The main way we find out what others have done is contacting them, a slow process that assumes we know who to contact. Web searchers bring some of the material up but it is not systematic in any way. A central registry for such studies would go a long way to helping our capacity to learn from what others have done. We still have to pause and do the learning of course.

    • Good points Fred. The issues highlighted in the report are of course also relevant to our own activities in our own countries, not just challenges for donors. And you are right that we will all benefit from well designed repositories. We need more initiatives like the Global Delivery Initiative http://globaldeliveryinitiative.org/ – as long as they are kept updated and useful.

      • GDI looks interesting but it is hard to tell who they are. When I clicked on the Learning link, I ended up at USAID’s Learning Lab. But, yes they idea of some sort of repository, and this is a useful exploration of the idea. They seem to be adding material fairly regularly if the blog is any indication. And it is great to see a focus on implementation. I was thinking of something organized around theories of change.

Leave a Reply

Your email address will not be published. Required fields are marked *