Snippet: Dialogue evaluation for independent evaluation

Many years ago, at the beginning of my evaluation career, I read about ‘dialogue evaluation’. It is a simple mechanism that we can use more frequently to mediate between an independent evaluation and self-evaluation – two distinct approaches that many see as incompatible. Dialogue evaluation (see below) can have real benefits in this context, yet it is seldom applied when independent evaluation teams descend on countries or societal groups under evaluation – especially when the evaluation is at a strategic rather than micro (‘community’) level, and a ‘parachute in, parachute out’ evaluation team is used.

Dialogue EvaluationI have not seen it used to refer specifically to the systematic comparison of self-evaluation findings with independent evaluation (emerging) findings, and understanding the reasons for differences. It relates to what is regarded as dialogue in evaluation, or dialogic evaluation, which refers to engagement and interaction between evaluators and stakeholders, ideally but not necessarily to reach some type of consensus (see here and here). This is usually defined as ‘participatory evaluation’, often considered as anathema in independent evaluations.

Jennifer Greene wrote eloquently about dialogue in evaluation in a special edition in the journal Evaluation in 2001. She defines dialogue as “a vision of learning about our differences and moving towards legitimizing and accepting them”. “The purpose of such evaluative dialogue is to enable stakeholders to more deeply understand and respect though not necessarily agree with, one another’s perspectives. Such understandings, in turn, can engender more reciprocal, equitable and caring stakeholder relationships in that context, as embodied in an improved, transformed or even revolutionized evaluand. In these ways, dialogic evaluation constitutes an important democratic activity in society…. Dialogue …. directly engages the moral–ethical and politicized power relationships among stakeholders …. directly invokes the key interests in an evaluation, as it provides a space for the sharing and reciprocal understanding of these interests and for the possible and initial development of some common ways of seeing and acting, some collective perspectives on meaning.”

It has become good practice to ask management or executive teams to do a self-evaluation report aimed specifically at informing independent evaluations. Evaluation teams commissioned to do independent evaluations tend to use such a report as one of many inputs into their process. They seldom engage in a systematic analysis and then one or more systematic conversations – a dialogue that is detailed, respectful and for mutual benefit – about emphases, issues and (emerging) findings that differ between the two evaluative exercises. Too often such engagement is only at the end of a field visit or at the end of the independent evaluation during a ‘stakeholder workshop’ or request for a ‘management response’. This is often too late to analyse each other’s sources of evidence, their different interpretations, and the root causes of such differences. Or at least to develop mutual respect for the differences.

It will not compromise the independence of the evaluation to have such a dialogue with those who did the self-evaluation, and to continue to shape data collection, final conclusions and/or recommendations with these conversations in mind.

When opportunities for mutual learning, respect for alternative evidence or interpretations of evidence, and in-depth understanding get lost, it depletes the quality and credibility of the evaluation. A stronger focus on  more systematic and analytical ‘dialogue evaluation’ in evaluation methodology guidelines can add significant value to the utility and credibility of independent evaluation processes.

Share this Article

Zenda Ofir

Zenda Ofir is a South African evaluation specialist currently based in Switzerland. She is a former AfrEA President, IOCE Vice-President and AEA Board member. She has worked in around 40 countries, primarily in Africa and Asia, and provides evaluation advice to many multilateral and international organisations.

2 Comments

  1. This is an interesting subject for discussion. I also heard as per IFC and Ausaid that monitoring and evaluation is an effective tool to manage the public private dialogue process and to demonstrate its purpose and performance.

    Another effort carried out in 2015 is that IEO completed an evaluation of the self-evaluation conducted by the IMF in order to learn from experience, improve the quality and effectiveness of its work, and enhance transparency.

    In tandem with independent evaluation, self-evaluation contributes to the governance of public institutions and policy makers. What are the challenges in this type of exercise?

    • Isha, thanks for pointing to the IMF example. It will be interesting to analyse the credibility and utility of self-evaluations as it will show the extent to which people are able and willing to assess themselves. My own experience in working with management teams who have done self-evaluations has been very positive (AWARD and IDRC are two examples) – they generally tended to be highly self-critical. But this requires people to have confidence in themselves, and assurance that the organisation will not “get” at them for ‘less than ideal’ performance. This is often not the case in organisational cultures. In my view evaluation should be an integral part of management courses at university to help cultivate evaluative mindsets and the idea that it is integral to management. Self-evaluation has one great selling point – “If we pick up problems ourselves in time we can fix them before the independent evaluators descend on us…”. Finally, in 2009 I did an independent evaluation of a very good IFAD-funded initiative “Managing for Impact” implemented by Wageningen University. Excellent way to integrate evaluation in different forms into management practices. But almost invariably the people involved were middle managers or lower in the hierarchy, typically M&E officers, and could not convince their bosses or the organisation as a whole to start working in this way. In all these types of programmes, if the executives do not buy into self-evaluation, the chance of building such an organisational culture is slim.

Leave a Comment

Your email address will not be published. Required fields are marked *