**7 min read**
We still have many opportunities to re-imagine evaluation and to combine this with the best evaluation knowledge we already have available. Indigenous evaluators in Latin America, as well as nearly all evaluators in Africa and Asia can still do much to consider what evaluation would have looked like “if invented here” – following on the deeply thoughtful work done by evaluators among the First Nations in North America, African-Americans and indigenous societies in Australasia.
In my experience, such re-imagining of evaluation can best be done if we use as our point of departure one or the integration of both of the following: societal or nation-state cultures that create the dispositions in society with which we have to work, and (ii) the development contexts and priorities in the era in which we now live.
The NICE Framework for Research and Innovation in Evaluation (see diagram in previous post 1) emphasizes the former – thinking anew about the key elements that make up evaluative activities, based on key differences between societal (especially national) cultures – while not ignoring the influence of evolving contexts from local to global levels.
The latter (our current development contexts) can also lead to new evaluation concepts and foci. For example, the 2030 Agenda focuses our attention on systems thinking, complexity and transformation. This is part of the ‘Context’ element of the framework, and can also lead to new insights about evaluation theory and practice, beyond methodology and methods. But the key focus of the framework is the societal dispositions displayed in elements #1-10 in the diagram in my first post on the topic.
Relationship with frameworks that highlight cultural patterns in society
Useful frameworks and models when applying the NICE Framework include Hofstede’s six dimensions of (national) culture; the seven dimensions of Trompenaar and Hampden-Turner (see also see also here for practical applications); Erin Meyer’s Culture Maps and Country Mapping instrument; and information repositories such as the World Values Survey and its Research Paper series.
There are also many other analyses of the differences between societies, among the most prominent those that refer to the differences between indigenous societies and the West. See as one example this illuminating article about evaluation in indigenous contexts by Hayley Marama Cavino. I have a large library of similar articles and books with fascinating analyses that alert us to the importance not only of being ‘culturally competent’, but thinking deeply about what this implies for efforts to re-imagine evaluation and its role in society.
Of course, all formal frameworks are imperfect, and those dealing with culture are always very vulnerable to criticism. Those of Trompenaar and Hofstede have been criticised on the basis that cultures evolve (‘cultural drift’), that the researchers have their own biases which can be reflected in the data collection instruments, that cultural fragmentation is not well accounted for, and that the environments in which data were collected might have their own specific peculiarities.
Certainly, cultural patterns are hard to define in multi-cultural environments, and they evolve ever-faster in a world influenced by increasing interconnections. Still, those of us who have worked across many vastly different societies recognise that many of these dimensions proposed by specialists like Hofstede, Trompenaar and Meyer do persist. The most recent indexes and analyses (see also here) continue to confirm this, and not only in traditional ‘village’ life, but also in modern ‘city’ life in spite of the more dynamic blending of cultures in the latter.
So, cultural patterns should obviously be considered with the necessary care and nuance. But they can provide very useful information for evaluators and evaluation commissioners who have to work at the level of the nation-state or with large swathes of society – and of course, also at community level. It is better than ignoring these obvious differences as we cross borders and boundaries in our work.
The NICE Elements #1-3
|“The West and East Asia differ in fundamental assumptions about the nature of the world, in the focus of attention, in the skills necessary to perceive relationships and to discern objects in a complex environment, in the character of causal attribution, in the tendency to organize the world categorically or relationally, and in the inclination to use rules, including the rules of formal logic.”|
|The Geography of Thought, Richard Nisbett, 2003|
Elements #1-3 of the framework (diagram in previous post 1) draw attention to three broad categories of societal dynamics and dispositions. They will influence how evaluation is done and perceived in different societies.
(Without my intending for it to be so, I recently noticed that they relate well to an analysis of issues faced by Pākehā evaluators highlighted by Rae Torrie et al in this excellent book edited by Stafford Hood, Rodney Hopson and Henry Frierson, some of the pioneers in culturally responsive evaluation. There are also good connections with Karen Kirkhart’s framework for multicultural validity.)
These three elements are intertwined with the other seven elements #4-10 (which can probably also be grouped together into one or more broad categories).
#1. Life within the ‘universe of everything’. I refer to life in the ‘universe of everything’ because it reflects our understanding of, and relationships, with everything around us except other people. So with ourselves and our motivations and existence (in other words, our identity and how we understand ourselves), with the natural world, and with the ecosystems in which we live. Among others, it draws our attention to some of Hofstede’s dimensions such as ‘Uncertainty avoidance’, ‘Long-term orientation’ and ‘Indulgence vs restraint’, and in the case of the Trompenaar framework, dimensions related to ‘Achievement’, ‘Time’ and ‘Direction’ (see also here for details).
Our relationship with self and the universe also shape our spirituality. It is therefore no wonder that indigenous evaluators have been exploring spirituality in evaluative practice – see for example Laura Luo and colleagues from China in the latest volume of NDE, and Vivienne Kennedy, Fiona Cram and her colleagues from New Zealand and the Pacific Continuing the Journey to Reposition Culture and Cultural Context in Evaluation Theory and Practice.
#2. Societal connectedness. This element refers to the structures, institutions and relationships that make up our society. It is the most visible and best understood, and most often addressed in analyses at the intersection of culture, development and evaluation – including in Bagele Chilisa’s key contribution to the Made in Africa Evaluation initiative, which among others highlights the important notion of ‘ubuntu’ in its various forms in African society. Among Hofstede’s dimensions this element refers to the dimensions of ‘Power distance’ (power relations), ‘Individualism’ and ‘Masculinity’, and from Meyer’s culture maps, the levels of trust, decision-making approaches and notions of leadership.
#3. Societal engagement. This element is closely interwoven with societal connectedness, but with a specific focus on how and why societies engage and communicate. Among others, it draws our attention to Meyer’s maps that include ‘communicating’, ‘evaluating‘ and ‘disagreeing’, and to Trompenaar’s dimensions of ‘affective vs neutral’ and ‘specific vs diffuse’. We have societies that prefer documenting as much as possible, and managed by explicit rules and regulations; others prefer verbal interactions and stories, and share understandings in society through implicit rather than explicit ‘rules’. This is also where the implications of language, including terminology and metaphors, are considered.
The NICE Elements #4-8
These five elements speak for themselves. They relate directly to core components of evaluation practice, and provide fruitful ground for analysing differences that can be the basis for adjusted or new approaches to evaluation. It is important for us to understand how different societies attach different meanings to, and deal differently with each of these concepts. For example, we know that different societies have different ways of learning, yet we tend to disregard this in our strategies. It is likely that evaluative thinking differ in terms of its nature and role it plays in society. And different value systems definitely influence how, and on what basis judgments are made. What does all this mean for how we conceptualise and do evaluation?
Two specific examples: First, knowledge systems and understandings of ‘truth’ and ‘evidence’ differ between societies. In some indigenous communities, ‘truth’ and ‘evidence’ are generated by, and from within their society. This means that so-called ‘evidence’ and findings generated by external evaluators (outside that society) will have less credibility and hence less utility for them. ‘Independent evaluation’ will not be accepted here. At another level this element also draws our attention to the inside-out/outside-in notions of an ‘integral evaluator’ proposed by Hazel Symonette.
Second, it is well-known that in certain countries, especially in East and Southeast Asia, explicit negative, potentially embarrassing statements are unacceptable when presenting evaluation findings, conclusions and recommendations. As the well-known term “losing face” indicates, these societies avoid confrontation in conversation and negotiations (clearly very different to many countries in the West). There are deeply philosophical reasons for such behavior in these societies. Conventional evaluation processes – and the values underlying the evaluation findings – have to be rethought under such circumstances. This is just one obvious example; there are many others much less obvious, but no less important.
The NICE Elements #9-10
How change is perceived to happen, as well as understandings of progress and achievement and hence of success of failure, will differ dramatically among societies. For example, many indigenous societies see change as cyclical; they will not feel comfortable with most of the rather linear (even with feedback loops) ‘theories of change’ designed from Western perspectives. Other non-Western societies, for example those in East Asia, have a much better understanding of the complex adaptive systems behaviours of development efforts, and will not appreciate efforts to predict change pathways in detail.
The academic literature also has lovely examples of how “achievement” and “success” can be differently conceptualised in different societies, illustrating the very different evaluation questions that will be posed by indigenous evaluators in societies where relationships and contributions to society matter more than the self-interest and individual performance and capacities so prominent in individualist (mainly Western) societies.
In my third post on the NICE Framework I will give my views on how research and innovation in evaluation, especially in the Global South, can be (and are being) approached in order to accelerate our discovery and development of theories and practices that are more firmly rooted in our rich philosophies and cultures.