**6 min read**
Evaluative practice needs to focus on three interconnected, overlapping priorities if we want to help lead action towards a transformed world. I like to call them ‘COVID-Safe’, ‘COVID-Ready’ and ‘COVID-Informed’ Evaluation.
COVID-Safe Evaluation: Coping in this emergency
Papers, blog posts and discussions show that at the moment the global evaluation community’s focus is almost entirely on COVID-Safe Evaluation: what to do to cope – the “how” of maintaining and concluding ongoing evaluations, and keeping evaluation practice active during and just after the pandemic. It focuses on the fine detail of how to adjust methodologies and methods to suit the emergency situations in which we find ourselves.
COVID-Safe Evaluation is important and deserves to be well covered. This guidance by OECD/DAC and UNDP-IEO is a good illustration; salient points from around the world have been captured in this valuable blog post by Karsten Weizenegger; and I also did my own blog summary.
The main challenge for COVID-Safe Evaluation is to ensure that methods and results are credible, useful and nuanced (bringing out subtle, often hidden dimensions) while doing no harm under much more challenging circumstances.
Four areas of increased risk immediately come to mind; there are of course many more:
First, increasing biases – for example, reaching for interviews or surveys only those with phones or access to the Internet; failing to engage with intended beneficiaries of support; and spending less time experiencing the dynamics within families or communities or systems, especially when dealing with power asymmetries and voice.
Second, over-simplifying – for example, putting faith in only GIS and remote sensing to fill important data gaps, yet without the necessary nuances that ‘warm’ and ‘thick’ data provide.
Third, focusing on ‘solutions’ without adjusting for consequences – for example, with increasing focus on the capabilities of big data, giving global tech monopolies the chance to open up economically poor societies to manipulation and exploitation, especially by forces far from home.
Fourth, mismanaging ‘risk’ – for example, by increasing inequalities between so-called ‘international’ and ‘local’ evaluators, where the latter is expected to take risks that the former will not or cannot take.
COVID-Ready Evaluation: Building societal resilience for the next emergency
Dr David Nabarro, highly experienced WHO Special Envoy for COVID-19, urges all countries to work towards becoming COVID-Ready societies. Like many scientists, he believes that people might have to learn to live with this unpredictable and dangerous virus. If a vaccine or cure is not found, “unending lock-downs” will not work. Instead, it will be necessary to go about our daily lives and work, and have social relations, but NOT with a ‘business as usual’ attitude, although the latter is already apparent in countries emerging from lock-downs.
Nabarro notes that being a COVID-Ready country or society will include ongoing tracing, containment and a neighbourhood health watch, even after the first and second waves of outbreak are gone. People in a society have to be willing and full participants in such actions, and communities, public health services and hospitals will have to be highly organised to play their role.
COVID-Ready Evaluation therefore helps to cultivate in societies good sense, good systems, and resilience in the face of crisis. This requires shifting some attention from dealing technically with how evaluative practice should change, to what should be evaluated in support of the ongoing emergency, the immediate aftermath, and the next wave of COVID-19 infections or other pandemic.
This is not only relevant for the Global South, where we are forever the focus of studies and demands to adjust. COVID-19 is clearly a global affair, and economically rich countries have to make sure they are not responsible for a second COVID-19 wave more vicious than the first.
A good example of pivoting to both COVID-Safe and COVID-Ready Evaluation and research is the Research for Effective COVID-19 Responses or RECOVR initiative of IPA. It is a group I do not normally highlight because of their outdated approach to ‘rigor’, and their reductionist view of evaluation and of development. Yet through RECOVR they demonstrate many good elements aimed at mitigating the effects of the pandemic. They intend to inform the immediate response efforts as well as provide support to decision-makers who have to deal with the longer-term fall-out.
It is very unfortunate that they are unlikely to do so from a complex systems understanding of change.
A very useful recent report by UNDESA (with contributions from 43 UN agencies) highlights in this very nice diagram how progress towards all SDGs will be affected by COVID-19. COVID-Ready Evaluation therefore prioritises strategies to counter the loss of income among the most vulnerable families, the lack of remote learning access, disrupted access to electricity, hunger due to disruption of food supplies, and so on. It helps to cultivate resilient individuals, leaders and societal systems that can cope with the next pandemic or similar global crisis. It helps to adjust the mindsets, skills, strategies, institutions and systems of communities, societies and their leaders.
COVID-Ready Evaluation will support intensive efforts to learn from both research and evaluation findings during the first wave of COVID-19. It will require robust studies of the influences on success or failure in shaping COVID-prepared societies. It will demand from evaluators to work closely with research initiatives and make use of research results. And indeed, leading evaluation offices such as UNDP IEO and GEF IEO have already launched useful initiatives to support learning from evaluation during a time of crisis.
Yet this focus, while essential, is not sufficient. It tends to emphasise recovery, preparedness and resilience more or less within the status quo, and in the short- to medium term.
This is why attending to COVID-Informed Evaluation is urgent and necessary.
COVID-Informed Evaluation: Infusing SDG action with systems thinking and transformative development
COVID-19 has made visible in a very striking manner what we already knew – the highly interdependent nature of life and its ecosystems, and the slow-moving crises in the era of the Anthropocene, require us to focus urgently on large systems and transformational change if humanity is to achieve the SDGs at any time. And even more so, with a long-term perspective on both sustainable development and resilient societies.
In fact, we need to help prepare humanity and its societal systems for a very different future.
This is the realm of COVID-Informed Evaluation.
This priority has received limited attention from evaluation professionals and from those financing strategies and initiatives at this time. The most notable contributions have been from those working on the society-environment intersection. But it has to go far beyond this small group. COVID-Informed Evaluation has to be addressed by all influential actors – financiers, commissioners, doers and users of evaluation – with the same urgency and vigour as COVID-safe and COVID-ready evaluation.
I repeat here some of my own COVID-Informed Evaluation priorities that I noted in a summary list and in previous posts in this series:
One, rethinking the value proposition and niche of evaluation in an era defined by so many interconnected pressures, of which COVID-19 is just one – the Anthropocene and its effects on the SDGs, the Fourth Industrial Revolution, massive and increasing inequality, and the geopolitical power shifts that have led to the belligerence of US rhetoric towards China that can easily take the world to war. And then taking steps to ensure that the value proposition is indeed so.
Two, using Blue Marble Evaluation principles, working with sustainable development, large systems transformations, and transformative development aimed at escaping poverty traps.
Three, integrating ‘first principles’ – systems thinking and complexity science as well as understanding of the dispositions of societies – with the value of big, warm and thick data.
Four, paying much more respect to theories, frameworks, models and practices originating from societies with very different value systems, worldviews and narratives. The world cannot afford the danger of a single story – yet it is rife everywhere we look and act.
Five, dealing with the accelerating need for attention for balance – and therefore to choices and trade-offs, including carefully dissecting the easy rhetoric about ‘no-one left behind’.
Six, making connections that lead to both collective action and synergistic portfolio approaches across geographic, sector, stakeholder, disciplinary and ideological silos – including drawing from scientific research in fields such as data sciences, behavioural sciences, artificial intelligence, neurosciences and ecological sciences, and from future sciences that allow for scenario planning and foresight towards policy and strategy options.
In a next set of posts I will focus on these priorities and highlight some of the practical implications of COVID-Informed Evaluation. We need to help ensure that transformations in the world give us butterflies with wings rather than caterpillars with wings.