Evaluation for Transformation, Part 2. Interactions between the SDGs – and beyond

**10 min read**

If you read this year only one set of papers useful for SDG related evaluation, let it be this ISC/Nilsson et al. series on the interactions between the SDGs, developed between 2016 and 2018, as well as the subsequent paper on the interactions of SDG 14 by Gerald Singh and colleagues. These papers help to demonstrate the use of systems thinking and complexity concepts in our work (so well articulated in MQP’s Developmental Evaluation), and for planning and implementation or strengthening of interventions, institutions or systems. The content can be used in multiple ways: to prioritise and align policies or strategies, reduce risk of failure, develop better theories of change, work towards transformational change – and to evaluate whether plans and actions have been appropriate, aware of important synergies that can enhance impact, possible preconditions for success, and risks that should be attended to in time.

I wrote already in March 2017 in this blog post about the utility of the ISC/Nilsson et al. paper. I was aware of the series early on as I was at the time working with the person in charge of this initiative on another assignment with ICSU, now the International Science Council (ISC). I have already used or referenced these papers several times in my instruction of students and when doing assignments. My work on SDG 14 brought me in contact with this important related paper by Gerald Singh et al. which advances the initial analysis of interactions between SDG targets to bring new insights about co-benefits, trade-offs and prerequisites.

The analyses encourage us to use frameworks and models in the scientific literature

Please read and use these papers. They are not a panacea, but certainly a very good contribution at this time – also because they highlight two other important issues for evaluation specialists:

One, we need to be very aware of what is being produced in the scientific arena. There are very significant advances that are imperative for evaluation professionals to incorporate into their work, and many of these disciplines are far more advanced in their knowledge of CAS concepts than the evaluation field – although such specialists, who tend to be in the academic sector, are not necessarily aware of the applications in practice, except when they are “pracademics”. Those in my old academic domain – that of ecological sciences, and in my case ecological chemistry – are used to ‘ecosystems’ where such interconnections are part of everything that is done and thought about.

Two, we need to make better use of thoughtful theories, frameworks and models from the literature that we can use in the development of theories of change, and hence in design, implementation and evaluation. As Michael Quinn Patton recently pointed out in an AES keynote: “Developing a theory of change is not about a group of people sharing their ignorance and biases“. Where possible, we need to have solid grounds for TOC logic, beyond the opinion of the few people in the room (At the same time we should also guard against dominant models and frameworks that might not be relevant everywhere in the world).

They can help us to understand ‘synergistic effects’

I have often written, spoken about and considered the importance of synergies, with a focus on synergistic ‘effects’ or interactions. I like this term because I have a background in chemistry where this is a very important concept; others refer for example to reinforcing effects or co-benefits. Synergistic effects emerge when interactions between ‘things’ – usually interventions, their envisaged targets, and/or their outcomes/impacts – result in their becoming more than the sum of their parts. They reinforce one another, leading to greater positive outcomes than would normally have been observed had they been applied in isolation of one another.

Of course, such interactions can also be negative. An example from my chemistry background: We know that people on certain medications are forbidden to eat grapefruit as the latter is known to enhance the effect of many medicines multiple times. This is an example of a negative result on health. We tend to focus on those interactions that give positive results, but this is not always the case. They can constrain or oppose one another – one of the reasons why we should always diligently search for negative consequences or impacts when we evaluate.

Another example from my old field of work: My PhD was on chemical communication in nature (semiochemicals). We found time and again that when we tested the active compounds in isolation, they were much less potent than when they were in the mix together, as secreted by the insect or animal. The impact of the individual compounds was greatly enhanced when they were applied together because of the interactions between them. This is the synergistic effect.

This we also find in social / development contexts. The ISC/Nillsson et al. series of papers highlights this very well, and make it practical to implement In the diagram below, synergistic effects or ‘co-benefits’ (in this case among SDG targets) are indicated by the blue bars in the diagram below. They show which targets are mutually reinforcing or enabling. Constraining or counteracting effects are the negative side of such interactions, shown as red bars in the diagram. This is where negative consequences and outcomes/impacts, as well as trade-offs become important. Such analyses mean we now have a better idea of what to reinforce, what to reduce or eliminate, and where trade-offs will be required.

Key Interactions

Click on the image to see the larger version.

Full explanations are provided in the ISC/Nilsson et al. paper. The papers place the interactions between the SDG targets on a 7-point scale, depending on the extent to which they reinforce, have no effect, or oppose one another. See as example the figures below; they summarise the analysis of the interactions done for SDG 2 targets.

SDG2

Click on the image to see a larger version.

I have always been aware of the importance of trying to find synergies, and acknowledging that trade-offs are essential in development or humanitarian work. Two examples from my professional life.

The AWARD programme:  Here I worked with synergistic effects within a programme, detailing and testing a programme theory of change with the programme team. I worked between 2009 and 2014 as evaluation advisor and internal evaluator with AWARD, a Gates Foundation funded programme that supports professional women scientists in agriculture to develop leadership skills. We found that one of the main reasons for AWARD’s success was the synergistic effects resulting from the eight complementary components of the programme. Skills, awareness and insights that were developed through education, mentoring and exposure activities built on one another and had a chance to be expressed in efforts such as role modelling, participation in leadership activities and conference presentations, developing the fellows’ self-knowledge, confidence, expertise and profile. Qualitative data analysis of the influences on progress and performance showed that this resulted in important reinforcing feedback loops, demonstrating the power of synergies between components within an intervention with multiple dimensions. It highlighted why one dimensional interventions are often quite ineffective.

A UN agency in the Mekong sub-region. Migration to find work is an important dynamic among countries in the Mekong sub-region. In a ‘High Level Evaluation’ assignment I led in 2017 for a UN agency, we found that there were many interventions by several UN and other organisations aimed at improving the conditions and livelihoods of migrant labourers. All related to government priorities in the region and key people were aware of them, but they were often planned and implemented in isolation of one another, missing opportunities where governments as well as initiatives in each country, could work in concert in a coordinated strategy (This is very common, and just one illustration of the detrimental effect of fragmented approaches in the aid world).

In both cases we could have done much more work to understand more deeply, determine more rigorously, and record more systematically such synergistic effects – had resources and time been available. Such information would have been much more useful for future plans and decisions than, say, improving efficiencies in operations.

They can help determine priorities – working with co-benefits, prerequisites and trade-offs

The analyses of interactions between the SDGs facilitate efforts to deal with priorities for action. Gerald Singh and colleagues advanced the initial work by ISC/Nilsson et al. through their subsequent paper on the interactions of SDG 14 (Life Below Water). This brings more depth to the analyses; it engages in detail with co-benefits (synergies), trade-offs and especially, prerequisites for success when planning to achieve set targets. They also highlight the extent to which these relationships are context-dependent. I truly appreciate the value of this work for work given that the oceans, coastal areas, big rivers that flow to the ocean, and all the societies that benefit from these ecosystems provide for massively complex interactions.

SDG

Click on the image to see a larger version.

Wider applications of this type of analysis will benefit from, even demand simulations and the use of modelling. It certainly needs experience to identify and contextualise synergies. But this type of analysis gives more direction for solving complex cross-boundary problems than has been available to date.

A simple thing we can do in everyday practice is to attend vigorously to the factors that influence the changes (or lack thereof) that emerge from interventions. This already helps us understand much more about synergies and constraints to the cultivation of synergy.

They can help us to facilitate transformational change

Dealing with complex adaptive systems (CAS) behaviour is not only about ‘experimentation’ and ‘adaptive management’. If we want to experiment and do adaptive management with awareness of CAS behaviours, we also have to understand what to look for and deal with for best results. Working with synergistic effects and other results of the interactions of goals or targets or interventions when we design, implement, monitor or evaluate can help us do that.

This approach also enables us to work more smartly when developing ‘Theories of Transformational Change’ (ToTC). It is obvious that the more we can enhance synergistic effects, eliminate or reduce constraints and counteracting relationships, and promote and allow for appropriate trade-offs, the better chance we have of getting effective positive change and development.

We can enhance the chance of transformational change when we get powerful networks of individuals and organisations who can (together or separately) encourage and enable synergistic effects – focusing on co-benefits, removing counteracting forces and focusing on appropriate trade-offs across boundaries (geographic, sector, stakeholder, thematic, ideological). This is in large part the focus of the SDG Transformations Forum, which will be the topic of one of my next posts.

The type and extent of synergistic effects depend on context and culture

The extent and manner in which different things interact in a society depend on the environment in which they are embedded. This will differ per societal context and culture. Not all people eating grapefruit when taking the relevant medicine will be affected to the same extent; it depends on the environment created by the body’s DNA and cells, and the dynamics among the chemical interactions that constantly occur in our bodies. That is why not all people get the same side effects when taking a specific medication.

This situation is equally complicated in a social context where many different factors are at play. This is one of the very good reasons why we need to deal effectively with culture and context when we evaluate. Complex adaptive systems behaviour means that the interplay between culture, context and an intervention will lead to different levels and even types of synergistic effects – but they will be there.

Yet we often fail to attend to synergies in evaluative practice

Dealing with synergistic effects is clearly important to get the best results in development, or in humanitarian action. Why then do commissioners and evaluators fail to attend to it?

The fragmentation we find all over the aid world is one reason why aid programmes can be so ineffective, and why we are continuously confronted by the micro-macro paradox. A focus on holistic approaches, creating synergies and so on is not necessarily new. I grew up with ‘integrated development planning” which was very explicitly fashionable in the 1980s. But the SDGs, and these explorations of their interactions, now provide a frame to encourage us to focus on creating synergy in order to solve the complex problems we face in the world.

This means that in our evaluation system we will have to attend to the following:

One, our evaluation criteria should prompt us to attend to these aspects. This is one of the reasons why the DAC criteria require serious review, and why ‘Coherence’ – with a strong emphasis on the interactions between policies, goals, interventions and so on – is critical to include in our sets of evaluation criteria.

Two, we should be prepared to accept that our work as evaluation commissioners and as evaluators is becoming more complicated. To work with synergies requires a focus on qualitative information and informed reasoning. This often goes against the political economy of evaluation, and also asks for improvements in the poor practices we often find when qualitative information is collected and analysed.

Three, we now have much better motivation to work together across disciplinary, sector, stakeholder and ideological boundaries. This is essential when engaging in practice with interactions that stem from complex adaptive systems behaviours. With a sense of urgency, planners, managers, evaluation commissioners and evaluators should become  boundary-spanning leaders. And now we have some more very useful ‘tools’ – these analytical papers – to assist us in the process of spanning boundaries.

Hopefully these papers will inspire us all to be more attentive to those elements of our work that can contribute to the transformational change that our societies and planet need. The work described here is certainly one of these elements. We should not proceed with our practice without doing much more to understand how exactly to evaluate most effectively for the important concepts noted here and, through evaluation,connect to and support much better design and implementation of interventions, institutions and systems.

Share this article

Zenda Ofir

Zenda Ofir is an independent South African evaluator. Based near Geneva, she works across Africa and Asia. A former AfrEA President, IOCE Vice-President and AEA Board member, she is at present IDEAS Vice-President, Lead Steward of the SDG Transformations Forum A&E Working Group and Honorary Professor at Stellenbosch University.

2 Comments

  1. Beautifully and persuasively stated, Zenda. Integrative, holistic thinking and action offers the way forward. Thank you for calling attention to these excellent resources. Here are other examples of integrated, cross-silo approaches:

    Climate Change and Food Systems
    https://futureoffood.org/wp-content/uploads/2017/04/CC-FS-Report-Draft-4.28.17.pdf
    and
    ‘The Economics of Ecosystems and
    Biodiversity for Agriculture and Food’
    file:///C:/Users/Owner/OneDrive/Blue%20Marble/Resources%20for%20the%20book/TEEB%20foundations_vJun4.pdf

    MQP

  2. Dear Michael, thank you for these valuable resources. And for your visionary contributions through Blue Marble Evaluation, Principles-Focused Evaluation and Developmental Evaluation. The evaluation world owes you so much for your intellectual energy and ongoing innovations in our field.

Leave a Reply

Your email address will not be published. Required fields are marked *