With the hiatus in my posts due to a fiendishly busy schedule, I am pleased to have this pertinent guest post by old friends - evaluation luminary Donna Mertens and Stephen Porter who has contributed much to evaluation in Africa. Donna has pioneered transformative evaluation more than a decade ago through a lens on evaluation that has social justice at its core, focusing on the lives and experiences of socially marginalised groups such as women, racial and ethnic minorities, people with disabilities and the poor. ‘Transformative evaluation’ is clearly not only about the evaluation of societal, ecosystems or large systems transformation from local to global level, which is my main focus. But it also has the important dual purpose of using evaluation both to transform individuals and to transform evaluation itself.
“Homer was able to construct (imagine) a closed form because he had a clear idea of the agricultural and warrior culture of his own day. He knew the world he talked about, he knew its laws, causes and effects, and this is why he was able to give it a form.
There is, however, another mode of artistic representation, i.e., when we do not know the boundaries of what we wish to portray…we list its properties – and… the accidental properties of something… are thought to be infinite ” – Umberto Eco The Infinity of Lists
Umberto Eco, in the opening pages to his book The Infinity of Lists provides evaluators with a ticklish challenge: stability supports the definitions of a finite form of list, while instability requires a reforming of an idea with the recognition that the list of its attributes is likely to contain an etcetera. Providing an etcetera is not the laziness of an Evaluator, but a signal that a list needs to be constituted by many hands to make sense of an emerging reality.
Grappling with the instability of our current time is represented in our evaluation literature and indeed Zenda’s blogs, signaling that our lists and checklists need reformulation to guide and shift our practice. The transformative challenge is all around the evaluator. Importantly IDEAS has chosen Evaluation for Transformative Change as the theme for their 2019 Global Assembly. IDEAS did this based on the rationale that:
“transformational change is needed for our societies, economies and our relationship with the environment to become sustainable. On many fronts the world needs transformational change to be able to reach the aspirations expressed in the SDGs and the Paris Agreement: a world free of poverty; leaving no one behind; and ensuring a prosperous and equitable future in diverse and inclusive societies, with economies that increase wealth but not while undermining our food, clean air and living circumstances, with a climate and biodiversity that safeguards the future of humankind. A transformational change is one that reshapes models, policies, structures, practices, culture and management.”
The goal of societal transformation brings with it a challenge for evaluators. How does the role of evaluation need to adapt for the values of the SDGs? What is the list or lists we need? Are lists the appropriate response for evaluators who work in contexts of complexity?
The shift will involve the use of a transformative lens for the evaluation design, implementation and use. Zenda Ofir, in a post in this blog space voiced this challenge as follows: “One can expect that ‘transformation’ is the point of departure for development planning as well as for evaluation in a development context, which now includes both rich and poor countries. Yet when I look for literature on evaluation for transformation it appears to be meagre.”
The response to this challenge in the international development community has focused on systems and complexity theory (see, for example, a recent Webinar recording and Complexity Checklist), without explicitly bringing a transformation lens to evaluation planning, implementation, and use. As a rejoinder we propose that we start to form two lists that challenge the international development evaluation community to configure transformation with an evaluative perspective. These include works by one of the authors of this post (Donna Mertens), and distinguished thinkers who have written about transformation from the perspective of feminist, culturally responsive and Indigenous evaluation.
List 1: What is the form of Transformative Evaluation?
Donna has previously proposed four common themes to transformative evaluation approaches:
- underlying assumptions that rely on ethical stances of inclusion and challenging oppressive structures;
- an entry process into the community that is designed to build trust and make goals and strategies transparent;
- dissemination of findings in ways that encourage the use of result to enhance social justice and human rights; and
- addressing intersectionality when culturally responsive, feminist, equity-focused, and Indigenous theories are relevant in the evaluation context.
Work on evaluation in climate change and sustainability challenge us, for example, a recent New Directions special issue, on adaptive management and from the climate investment fund, to think about how evaluation relates to systemic change, accountability relationships, scale, sustainability...etcetera.
List 2: What resource on transformative forms of evaluation are there?
Existing resources on transformative forms of evaluation include:
The work of Donna Mertens developing the concept of transformative evaluation began in 1998 in her presidential address to the American Evaluation Association; the conference theme was Transforming Society Through Evaluation. Subsequently developing the meaning of transformative evaluation in, for example, three books: Transformative Research and Evaluation, Research and Evaluation in Education and Psychology 5th and Program Evaluation Theory and Practice 2nd ed. Her website provides many resources for evaluators who seek to use a transformative lens
The Center for Culturally Responsive Evaluation and Assessment (CREA), with Stafford Hood and Rodney Hopson as principal faculty, provides many resources that have relevance for international evaluators. Members of CREA have published several books that provide details regarding culturally responsive methodologies, for example, Continuing the Journey to Reposition Culture and Cultural Context in Evaluation Theory and Practice & Tacking Wicked Problems in Complex Ecologies: The Role of Evaluations.
Bagele Chilisa, Professor at the University of Botswana, wrote Indigenous Research Methodologies as a text to guide researchers and evaluators in work conducted in the larger, historical, cultural, and global context, with specific emphasis on historical and cultural traditions of the “third world and indigenous peoples”.
The Maori community in New Zealand have written and presented extensively on the Kaupapa Maori approach to research and evaluation; an approach that prioritizes responsiveness to their culture in the historical context of colonization, for example, Decolonizing Methodologies: Research and Indigenous Peoples.
UN Women have also been at the forefront of advocating for equity-focused, gender responsive evaluation.
Feminist Issues in Evaluation are being discussed within a thematic interest group of the AEA
These two lists are incomplete and can be elaborated to help achieve the SDGs in complex contexts. We accept the challenge, and hope that others in the international development evaluation community will as well, engage in the etcetera, shifting our practices so that evaluation can be substantive in achieving transformation, beyond its instrumental support to projects.
Stephen Porter wrote this blog in his capacity as Research Associate at Wits School of Governance at the University of Witwatersrand. He has worked as an evaluation advisor at the World Bank, Department for International Development, Oxfam Great Britain, and as a Director with Oxfam America. He holds an MPhil in Public Policy from the University of Cape Town and has published nine peer reviews articles on evaluation.
Donna Mertens is Professor Emeritus, Gallaudet University, specializing in research and evaluation methodologies designed to support social, economic, and environmental transformation. Her many books on research, evaluation methods and human rights include Mixed Methods Design in Evaluation; Indigenous Pathways into Social Research, and Transformative Research and Evaluation. She is a former AEA President and founding IOCE Board member as well as the Mixed Methods International Research Association.