**7 min read**
I am re-posting here a blog post I wrote in 2017 (Part 5 in my DAC Evaluation Criteria series). I want to highlight aspects of what I had written at the time, not about the criteria as such, but to reinforce some of the arguments I made about the nature of 'development' and the role of evaluation in it.
It is ironic that evaluation has been unable to prevent - or even just sufficiently critique - some of the most damaging policies and strategies that have originated in both economically rich and poor countries. What could and should evaluators have done to prevent at least some of the destruction wreaked in the 1980s and 1990s by the World Bank and IMF driven Structural Adjustment Programmes (SAPs) that were launched as part of the neoliberal policies forced upon the Global South, often against their will? Would we today have done better than the largely ineffective monitoring systems that failed to provide analyses and syntheses that reflected the reality of the interests and people of that country?
And would we have been able to help foresee and bring the dire negative consequences of the Green Revolution in Asia to light? What values would have underpinned our notion of co-benefits and tradeoffs?
Are we about to repeat the same omissions or mistakes - in essence the result of focusing on project interventions rather than on the larger systems or contexts of which they are part?
If evaluators are unable to prevent such disasters from happening, what good is the work that we do? How can similar situations be prevented in future?
These are some of the questions that should keep us awake at night.
Much is the result of the ever-popular evaluands - 'projects' and 'programmes', isolated initiatives evaluated in isolation, usually with a superficial nod to the dynamic contexts within which they are executed. Some evaluation specialists such as Robert Picciotto have argued for years that we should be in a position to evaluate systems, including those related to trade policies, financial architectures and flows, foreign direct investment, global value chains, and so on. But we have little power to do so, and the number of evaluations with an explicit systems approach is still pitifully small. A lack of insight, resources and/or political will tends to prevent a focus on issues that may have a major influence on the development trajectory of a society or country.
Blue Marble Evaluation now provides us with principles and technical approaches to expand our understanding of the role evaluation can play, yet the political economy of evaluation does not allow us the space to engage. BME type evaluations are still rare, and the burden is on researchers to take on relevant issues. Unfortunately research results can be even more readily dismissed than evaluations; we at least have the advantage that there are intended users lined up to engage with evaluation findings. (This statement is in itself worthy of debate.)
But much depends on our evaluation questions and criteria, and on the flexibility and nuance with which we apply them. What should change if we want our evaluations to support 'development' more effectively?
Our development evaluation questions and criteria depend on what we consider to be 'development' appropriate for this time.
Can we evaluate FOR development that can sustain (endure)?
Our evaluation criteria and questions should compel us to attend to critical aspects that can support and help enable positive development trajectories over the long term at societal or national (or regional) level.
This requires, among others, long-term thinking, linking the micro, meso and macro from the local to global level, as Blue Marble Evaluation principles espouse.
Development - Really? Are you sure?
So in principle we should attend at least to the following:
One: Think carefully about the values, ideologies and/or models within which we evaluate contributions to 'development'. Development is a highly contested concept, with vastly different ideas about how it can be achieved. Evaluators seldom state explicitly whether they are evaluating from a specific stance on development - say human development, human rights or human security, as Shiv Kumar and I pointed out in a book chapter in 2013. When we evaluate portfolios, or single projects, programmes or policy, we almost always fail to make explicit the values, ideologies and models about development that frame what we evaluate, how we assess trade-offs and arrive at summative judgements.
Two: Evaluate for 'development effectiveness', not only 'intervention effectiveness'. Development effectiveness tends to be assessed at country or regional level. The proliferation of global development related indices (and now also the SDG reporting modalities) compels comparison and competition between countries and regions. But not every investment labelled a 'development intervention' automatically contributes to development from a national perspective, even if successfully executed - especially when unrealistic assumptions are made with respect to the 'higher levels' of theories of change, where it is actually impossible to make more than vague statements about the linkages between the intervention objectives and societal or ecosystem impacts.
As a result, we seldom focus on the development trajectory of a country or region.
Three: Avoid the risk of 'ersatz development'. Eminent UK / Korean economist Ha-Joon Chang refers to 'ersatz' development, which he defines as development based upon uncoordinated interventions that do not build on synergies or enable system coherence that facilitates change at a macro level. Such interventions tend to attend to the level of 'community' or 'pilots', without scaling, and strengthen the illusion that "every bit helps". This very common problem continues to be propagated by aid or philanthropic agencies, the private sector or a government. It is rife where governments are dysfunctional, fragile or lacking in confidence or expertise, or where a funder has little interest in engaging with integrated development policies and plans. A 'bottom-up', micro perspective of development tends to prevail, and so do debates about the much-lamented and much-debated micro-macro disconnect, see here and here, for example.
Four: Avoid the risk of ‘development without development’. According to Chang, development without development’ means that interventions focus on enabling conditions such as poverty reduction, individual betterment or meeting basic needs, without a vision of how the country can and will sustain positive economic, socio-cultural and environmental development trajectories in the long term. A positive development trajectory at national or regional level demands economic advancement, where financial flows from within and outside a country are large and sustained enough to fund development activities across sectors in an integrated manner over a prolonged period. It also demands the integration of economic, socio-cultural, environmental and political aspects.
The most successful countries in the Global South over the past few decades have tended to follow well-sequenced policies and strategies, complemented by opportunities to innovate and incentives that allowed for bottom-up experimentation and improvisation. Such efforts have been led by effective government policies and strategy execution in ways that (i) enabled positive impacts to ripple in unexpected and/or sustained ways, and (ii) generated the resources necessary for the country or region to continue to perform and succeed in a sustained manner.
So how can we do better when evaluating 'development'?
Cognisant of the four issues noted above, our evaluation questions and criteria should force us at least to do the following when commissioning or conducting evaluations:
- Focus on synthesis assessments about contributions to development effectiveness, not only on intervention effectiveness; we VERY seldom see this done.
Ensure that we assess the significance of the evaluand at a particular time with reference to the development trajectory of a society or country or region.
Consider whether synergy, complementarity, alignment, harmonisation and/or coherence - all somewhat different concepts - are relevant and if so, have been enhanced.
Evaluate for effective scaling where required, in order to help overcome the frequent misinterpretations of how to scale, and to ensure that any micro-macro disconnect is minimised;
Have a strong focus on trade-offs and the sustainability, or rather durability, of positive development impacts, and the role of dominant resource flows in enabling this (or not).
The bottom line
If we say we do "development evaluation", or better, that we "evaluate for development", each description and use of a set of evaluation criteria have to ensure that some attention is shifted back to the merit of the design and execution approaches, and to preconditions that are essential and sufficient for development success. An obsession with 'Impact' in isolation of everything else can lead to wrong conclusions about the extent to which certain efforts have supported 'development' that is likely to sustain.
Thinking about these issues should challenge us to consider the limits of the value we are in a position to add as evaluators to countries' and regions' development efforts.
And whether we should do more to ensure that we add more value than we have done to date.
Zenda Ofir is an independent South African evaluator at present based near Geneva. She works primarily in Africa and Asia, and advises organisations around the world. She is a former AfrEA President, IOCE and IDEAS Vice-President, AEA Board member, Honorary Professor at Stellenbosch University, Richard von Weizsäcker Fellow, and at present Interim Council Chair of the new International Evaluation Academy.