I do not do “development evaluation”. I do not practice the evaluation OF development. I evaluate FOR development.
There is a difference.
These days we are told that the need for the special category called “development evaluation” has receded because all countries are actually “developing” – among others because of growing inequalities within countries. This means that evaluators around the world have to grapple with similar challenges and methodologies. In essence, there is now only “evaluation”, whether working in the Global South or the Global North.
Perhaps. There is still room for debate on this issue, although I am pleased about the challenge to the conventional wisdom that some countries set the standard for being “developed” while others do their best to strive to become like them (especially in terms of “economic development”). This narrative has had to change.
But in my view, more than ever we now need to emphasise evaluation FOR development.
The evaluation OF development – i.e., “development evaluation”
“Development evaluation” has its own community of evaluators and professional association. It refers to any evaluation conducted in a development context in “developing” or “transitional economies (conventionally in the Global South). Its qualities are reflected in the definition and description of development evaluation in the introductory module of IPDET’s, which is probably the most credible source available for this purpose:
“Development evaluation is a systematic search for answers about development interventions and can be done at different times in the life cycle of an intervention. The aim is to determine the relevance, efficiency, effectiveness, impact, and sustainability of a program, policy, or project so as to incorporate lessons learned into the decision-making process. …
Development evaluation is expanding the scope of reviews, from project monitoring and evaluation of a specific program toward more complex ‘thematic’ evaluations (evaluations of a set of programs addressing a common theme), interventions within countries, and across countries. … In response to the need for relevant, rapid feedback at reasonable cost, development evaluation has evolved toward approaches that are fast, flexible, and participatory. … The goal of development evaluation has shifted from simply judging success or failure to one of understanding and continuous learning.”
It appears as if the qualities of development evaluation are not very different from evaluation in general, nor is “evaluating development” placing any special demands on evaluation practitioners.
Evaluation FOR development (E4D)
In my view, evaluating FOR development entails more than what is presented in the definition above. It requires more emphasis on the macro level (national, regional and global), as well as attention to specific aspects that follow from the need to deal explicitly with the notion of “development”.
In a book chapter published in 2013, AK Shiva Kumar and I argue that most low income countries in (yes, we called them “developing countries”) have important characteristics that need to influence how evaluation is conceived and done.
Around 95 percent of the people in the Global North have enough food and shelter, and a functioning education system. In the Global South only five percent of the population has enough food and shelter, and with three-quarters of the world’s population they have only 20 percent of world income.
Such differences mean that they have a less developed industrial base and a relatively low human development index (HDI). Their societies and institutional systems tend to be more vulnerable and hence more readily destabilised, and in spite of recent global power shifts, the least powerful remain the most vulnerable and exposed to exploitation by internal and external forces. Life is about survival, a daily struggle to cope. In many of the poorest countries they have a sense of victimisation and of dependence on uncontrollable forces. Citizens lack resources, opportunities, and exposure to new ways of thinking and doing. Their confidence and belief in their own have often (albeit not always) been eroded through foreign intervention and influence, as they have had to align with others’ notions of “success” and “development”. This easily leads to confusion and inter-generational tension about how that society’s ingrained values, norms and knowledge systems relate to the modern world. Responses differ from country to country.
This situation has significant implications for evaluators working in the Global South. I see our lack of attention to at least three issues as some of the main reasons why evaluations in the development arena are so often so ineffective.
Have I been doing my evaluations over the years with these issues in mind? Often not; sometimes yes. It requires grappling with answering hard questions, and with technical issues and methodologies that I have yet to find and master.
But I know that if we are serious about contributing effectively to “development” it is essential that we attend to these issues.
First, the E4D evaluator has to have some macro perspective on development. S/he will be challenged to work explicitly with the micro-macro disconnect, and with sensitivity to the broader frameworks and sets of values that determine the focus of development evaluations.
The E4D evaluator has to go beyond convention, which is normally to evaluate an intervention such as a policy, program or project against set goals and objectives (and plans and even desired outcomes or impacts), making judgments within these parameters. I believe we have to do more to link the evaluand and evaluation to the operating worldviews and models of development at macro level in order to be more thoughtful about the micro-macro disconnect that continues to plague our efforts.
In practice this means that we have to attempt judgments about an intervention’s “space” in development based on very challenging questions such as: “To what extent, and how, does this effort or intervention have the potential to contribute to development at national (or regional/global) level? How does it relate to current frameworks and values that guide development in this country (or region, or the world)?”
Countries have different framework and models determined by different sets of values and perspectives on development. In our 2013 chapter Shiv and I noted for example three streams of work that have shaped development dialogue and thinking in recent years: the human development approach, the human rights approach, and the human security approach.
The E4D evaluator will thus have to pay much more attention to the often neglected “higher levels” of the change logic (theory of change), linking the evaluand more explicitly and thoughtfully to national development models, and consider the implications for the desired positive development trajectory at that level.
Since such analysis demands internal circumspection about the values, models, theories of change and worldviews about development that influence the evaluator’s work, and against which judgments are made, it may take him/her into deeply political, ideological and value-laden territory. This should be handled with care – but not ignored.
Second, the E4D evaluator will explicitly engage with those critical challenges that are inherent in working with development in the Global South.
Shiv and I also argued that evaluators in the Global South working with development need to attend explicitly to the following (I will discuss the implications for evaluators in more detail in a next post on this blog). Other parts of the world may face the same challenges, but at the very least they are more intensely felt in the Global South.
First, we said that getting good and sustained results and impacts is more complicated and unpredictable in the Global South. Simplistic theories of change and superficial evaluations will conceal reasons for success or failure. We have to be sophisticated and nuanced in what we do.
Second, we noted that we are often insensitive to deep, sometimes hidden cultural differences between nations and regions. Evaluators should therefore understand and consider the role of culture in development contexts, especially where development strategies are not aligned with a society’s most deeply ingrained values, norms and ways of working (for example, when working with gender or age-based programmes).
Third, capacity and other constraints among individuals and institutional systems in the Global South are more serious and pervasive. When individuals and institutions have to meet notions of performance and worth that fall outside a society’s own culture, knowledge systems and ways of thinking and doing, this can have a marked effect on development success.
Fourth, development in the Global South goes seriously wrong when planners and evaluators fail to recognise and address privilege, power asymmetries and disempowerment, which are frequently much more pervasive and significant in scope than in the Global North. These issues also present notable transnational considerations, especially when they play out in global policy regimes and systems that affect trade and other important relationships between countries and regions.
Third, the E4D evaluator will be far more aware of the nuances and challenges in defining and judging “success”.
All of the above has implications for what we consider “success” (or as we tend to say, “development effectiveness”) when evaluating development interventions. We cannot claim that a development intervention has been successful in supporting development when its objectives have been achieved – yet we do so all the time. We can do so only if the theory that clarifies the contribution of those objectives to development at macro level is well understood and considered in the judgment, and the values or principles on which that is assessed are made explicit.
Moreover, and perhaps more importantly, we cannot claim development success unless
- the positive outcomes and impacts of the intervention are sustained or transformed into other emergent positive impacts;
- we are reasonably sure that positive outcomes are not being neutralised by unintended negative consequences or impacts;
- we consider the integrated nature of development – in other words, how this intervention relates to other efforts towards development, whether at policy, strategy or implementation level.
All of the above has implications for how we deal with the OECD DAC and other evaluation criteria, and set up our evaluation questions. This will be the topic of one of my follow-up posts.