Osvaldo is one of the most knowledgeable professionals I have ever met - a veritable walking encyclopaedia of all matters related to evaluation. Over the years it has been a feast for me to meet up and occasionally work with him, among others as members of the International Evaluation Advisory Committee of UNDP’s Independent Evaluation Office. In this post he summarises important practical messages from his very useful chapter on ‘Dynamic Evaluations’ in the 2019 book published by IDEAS, Evaluation for Transformational Change.
About Dynamic Evaluation
Dynamic evaluations represent a type of evaluation that aims to contribute to transformational change. In this blog post I consider two challenges: One, how to define ‘transformational change of society’, and how to incorporate it into evaluation practice. And two, to show how an evaluation can become an agent of such transformational change in society – that is, a dynamic evaluation that displays the characteristics that evaluations need to support transformational change.
It is important to realise that dynamic evaluations (DE) are by definition transformative at the level of society, while none of the four types of evaluations generally considered - formative, summative, impact and developmental - are necessarily transformative at this level. The following paragraphs present different aspects of dynamic evaluations with the aim to show that such evaluations can have a role in contributing to transformation at the level of society.
How can evaluation contribute to the transformational change of society?
This needs important shifts in focus. By changing focus from projects and programs to strategies and policies, evaluation can become transformative, dynamic. It is not about forgetting the former, but about taking them into account from the perspective of the latter.
It also means moving away from an emphasis on micro-issues, linearity (including the log-frame) and single method approaches towards macro issues, a complexity lens, and multiple methods.
Key qualities of dynamic evaluations
One, they help establish a relevant evaluation agenda
The Sustainable Development Goals provide a menu of aspirations from which governments and civil society – at both national and subnational level – can choose what is relevant at their level. Dynamic evaluations can, and should help to generate evidence and lessons learned on critical issues such as social innovation, incentives, inequity, climate change, migration, global public goods and bad, biodiversity, waste and plastics, health, and education for all. Evaluators can play the role of facilitators at an early stage, promoting discussion between government and civil society or Parliament on the prioritization of themes for evaluations, taking into account the SDGs.
Likewise, there are global issues for which Blue Marble Evaluation presents an appropriate approach to deal with them.
Two, they make use of multiple methods and techniques
Evaluators should not be locked in single methods or single sources of data, but should be willing to engage with different evaluation methods and techniques for capturing, processing, analysing, and synthesising data. It is unlikely that all evaluators would be familiar with all relevant methods and techniques. Therefore, it would be essential to include in evaluation teams professionals with a variety of experiences and expertise – including evaluation generalists, for example with expertise in systems approach, who may be able to coordinate dynamic evaluations.
The openness and capacity to use different methods and techniques enhance the extent to which evaluation can make a contribution to transformational change of society.
Three, they make sure context matters
For an adequate consideration of context in conducting dynamic evaluations and their synthesis, it is worthwhile to use an adaptation of Realistic (or Realist) Evaluation, based on the triad Context, Interventions, and Results.
Complexity is an aspect of the context that matters for the evaluation of SDGs interventions. An unintended consequence of the 2030 Agenda for Sustainable Development is that each SDG may end up being considered as an entity in itself, pursuing it in isolation from the other SDGs, as if the “S” corresponding to “Sustainable” would be for “Silo”! The intersectoriality of the SDGs and their synergies should be given due importance during their implementation and when evaluation takes place
Four, they deploy Triple Loop Learning
Triple Loop Learning is defined as transformative learning - collectively examining underlying assumptions, leading to change in attitudes and social norms. This is different from Single Loop Learning, which is about instrumental learning (acquiring new knowledge individually) and Double Loop Learning, which is about communicative learning (understanding and interpreting knowledge through interaction with others).
Therefore, it is important to manage expectations concerning transformational learning, given the requirements for triple-loop learning. Dynamic evaluations can contribute to triple-loop learning with an examination of assumptions, including those concerning values , and it can also use findings from behavioural economics.
Five, they encourage policy dialogue as an instrument for the transformational change and institutional development
The shift of focus from projects and programs does not imply that they should no longer be considered as development interventions. However, a transformational evaluation has to take into account the extent to which they are contributing to transformational change and institutional development, for example by opening windows for policy dialogue that may induce policy change and through it transformational change may take place. The figure shows the main direction of causality.
Six, they focus on innovations that can help reduce inequalities
The development experience of China since the late '70s is perhaps the most extraordinary case of transformational change of society. It is a change that consisted of economic growth during decades at growth rates of approximately 10%, with an extraordinary poverty reduction effect but, at the same time, with an increase in inequalities.
One of the ways in which dynamic evaluation can and should play a role in these processes of transformational change of society is by identifying innovations that reduce - or have a potential for reducing - inequalities. That is, inequities-reducing innovations, a sub-set of pro-poor innovations, could pave the way for a transformation that mitigates or eliminates increased inequalities.
The capacity to demand dynamic evaluations
Finally, it is important to develop the capacity to demand dynamic evaluations, which will ensure that these evaluations will be used. This requires that society's decision-makers in government and civil society become aware of the role that evaluation can play in achieving transformational change of their societies. Advocacy for this purpose is an important aspect of the work of evaluation professionals across the world.
We trust that you will help advocate for this type of work to make evaluation more relevant for the era in which we now find ourselves. Let us know whether the qualities of dynamic evaluations resonate with the work you do – or would like to do.
Osvaldo Néstor Feinstein is a professor at the Universidad Complutense de Madrid’s Master in the Evaluation of Programs and Public Policies and member of the Evaluation Advisory Panel of UNDP’s Independent Evaluation Office. He was manager and advisor at the World Bank independent evaluation department, senior evaluator at the International Fund for Agricultural Development (IFAD) and an evaluation consultant for several international, bilateral and national organizations. Author of several publications.