Updating the DAC Evaluation Criteria, Part 4. Flexible criteria

Our development evaluation criteria should not be determined solely by stakeholder-determined interests, or by pragmatism resulting from constrained resources. Development of nation-states is a complicated matter. If we are serious about ensuring that evaluation plays an effective role in it, there are questions and criteria that we should treat as essential while others can be more flexibly applied.

But how do we determine which evaluation criteria to use under which circumstances? Instead of making some ad hoc choices or mechanically applying a set of pre-determined criteria, we need a credible framework that can direct our understanding in this regard.

Characteristics of a framework for evaluation criteria

Let us consider some of the possible determinants of a framework for one or more sets of revised criteria. There are many possibilities, but based on an analysis of existing challenges related to the DAC criteria (post 2 in this series), at least six imperatives will determine the character of such a set:

  1. Flexibility per context: The set of criteria will enable comparison between evaluations, yet also ensure the necessary flexibility to accommodate different mandates, norms and stakeholder interests. A certain part of the criteria will have to be flexible enough to be tailored to circumstance.
  2. A non-negotiable core set resulting from the nature of development. The set of criteria will ensure that we address critical elements of “sustainable development” – otherwise we will not be evaluating for This demands engagement with the conceptualisation of “development”, and with development as complex adaptive system.
  3. Wide applicability: The set will be suitable for a variety of evaluands, beyond projects and programmes, and across organisational, sector and geographic boundaries.
  4. Nuance: They will have definitions and descriptions that are detailed and hence nuanced enough to improve application and help drive good performance standards.
  5. Supporting conceptualisation, execution and results: They will shift some attention away from an obsession with “impact” towards smarter design and implementation efforts. Evaluation priorities and methodologies will better support intervention and systems design and implementation.
  6. Enabling meaningful synthesis: They will facilitate meaningful synthesis judgments across criteria to get a reasonably holistic picture of the extent to which (sustainable) development or humanitarian action has been improved.

Flexible sets of criteria, tailored to context

As I noted in post 3 in this series, there are two categories of criteria that can be considered as flexible. Which ones are used depends on the context and circumstances.

First, criteria relating to key time-bound mandates, values and norms

This set is determined by important global, regional and/or societal values and norms during a specific period, reflected within an organisational mandate. It will highlight the frameworks or models – and hence the ideologies – about development that a society or organisation and its partners hold dear. It will resonate with global and regional conventions such as the 2030 Agenda for Sustainable Development with its SDGs, and the Agenda 2063 of the African Union.

This set of criteria is only to some extent flexible. There is an imperative to reflect the norms of the day as determined by the mandates and strategies of each organisation or partnership involved in an evaluation.

The following are well-known as well as lesser known examples; the list can be much more extensive

DAC CriteriaAmong the less common criteria:

Inclusivity, Reach and Coverage all relate to the important norm of “no-one left behind” in the 2030 Agenda for Sustainable Development. Though usually treated as the same concept, they can also have slightly different nuances, where for example Reach reflects the boundaries of a targeted group within a country or region (e.g. a major population in need of water and sanitation), while Coverage reflects the extent to which different groupings within the targeted population have been adequately reached (e.g. households in distant villages; the most poverty-stricken households).

Legitimacy is a value-laden concept that relates to the credibility of interventions and processes of, for example, planning or engagement, as seen through the eyes of different stakeholder groupings.

An aspect that is usually neglected by well-meaning outsiders is that different parts of the world are likely to prioritize the criteria in such a set differently, based on their societal philosophies and values – if they are allowed the space to do so. In certain parts of Asia, Africa and among indigenous peoples, the interests of the society as a collective and/or as ecosystem, usually based on intricate webs of relationships, are deemed so important that it equals and might even trump individual rights. Harmony in Society is for example a very important ideal in East Asia, rooted in Confucian philosophy. It seeks to balance and reconcile differences and conflicts through creativity and mutual transformation rather than conflict. Harmony with Nature, with strong roots in Taoism and other indigenous philosophies, is in line with sustainable development as reflected in the SDGs. It is a concept set to become more important across the world. Our existing sets of criteria seldom give prominence to such differences.

Second, criteria relating to stakeholder interests and concerns

These criteria are determined by stakeholder interests and concerns at a particular moment. There is no other imperative to include any of these in a set of criteria, and they are therefore even more flexible than the set based on mandates and norms. Examples drawn from current practice include

DAC Part 4

However, there is in my view also a core set of non-negotiable criteria that needs to be considered. This very important and likely controversial issue will be the focus of my next post.

Share this Article

Zenda Ofir

Zenda Ofir is a South African evaluation specialist currently based in Switzerland. She is a former AfrEA President, IOCE Vice-President and AEA Board member. She has worked in around 40 countries, primarily in Africa and Asia, and provides evaluation advice to many multilateral and international organisations.

7 Comments

  1. Hi Zenda,
    The discussion is getting more interesting. Thanks for the synthesis of context and stakeholders interest domains. Well captured.

    To lend a voice to the context perspective ( EQUITY & GENDER EQUALITY)

    Equity has a positive impact on the construction of a socially fair and democratic society. Unequal opportunities for social groups in society is often a significant factor behind social unrest, which may lead to crime or even violent conflict, with negative effects on the social cohesion of society. Prolonged inequity may lead to the “banalization” of inequity
    Inequity constitutes a violation of human rights and hampers the equitable achievements on Human Development and the Sustainable Development Goals. Without gender equality and equity, achieving the twin goals of “eradicating extreme poverty and boosting shared prosperity by 2030 – will be an impossible dream to attain, If the inclusion agenda is left behind.

    • Yinka
      You make an elegant case for the importance of attending to important norms known to influence development. As you note, much of the dissatisfaction and instability in the world today follows from neglect of issues of inclusion and equity, and the inequalities that are flowing from especially capitalism on steroids. Serious issues for evaluation, and demands much greater attention to the values that underpin our evaluations.

  2. Dear Zenda,
    You have raised the issues on ‘rethinking the DAC evaluation criteria’ so rightly and timely.

    Regarding ‘Sustainability’ issue, as we have witnessed that among SDG documents, there does not appear to be a definition or a conceptual framework explaining the concept of sustainability or the process through which sustainability is to be achieved or progress to be assessed. While it would be very difficult to develop a sustainability-responsive theory of change for achieving the SDGs, it might be possible to do this for individual SDGs (or even for particular sub-goals).
    Again, for example, the current definition of sustainability in DAC Criteria is limited to prospective (likelihood of) sustainability and do not make any reference to retrospective sustainability (how sustainable it has been). Furthermore, it only mentions the need to consider environmental and financial aspects of sustainability, leaving out other essential elements to the sustainability of interventions such as political support, cultural appropriateness, adequacy of technology, and institutional capacity.

    • Bhabatosh

      You are right to point out the importance of sustainability, and the need for nuanced definitions and practices that do justice to the concept. You give good examples. Some agencies – among others IFAD, UNICEF and UNDP – have improved in their evaluation guidelines the descriptions of several of the DAC criteria. You may also want to check out BetterEvaluation, which now has an informative section on how to evaluate sustainability retrospectively.

  3. On reach aspect, the question of hard to reach areas(hard to work in , to stay) in the regions of intervention pose serious challenges to evaluations because even acquiring adequate, accurate and credible data for baseline and process evaluations may be difficult. This will affect the evaluation findings hence decisions made based on them.
    Coverage aspect is hurt by inadequate resources . For instance, imagine a rural electrification programme intended to cover all 162 district local governments of a country. Then because of limited monetary, men, machinery, materials, information resources, only 79 districts are covered.
    On inclusiveness, every intervention for public consumption should have all stakeholders involved at all stages for ownership and sustainability in order to leave no one behind by 2030.

    • Mayie
      Your comments highlight the importance of having such criteria – otherwise one misses important weaknesses in implementation and as a consequence likely also poorer results. It would have been good to have meta-evaluations and meta-analyses for each widely used criterion to see to what extent they are usefully defined and applied.

  4. I suspect that focusing directly on the DAC criteria rather than introducing separately additional criteria. For example, for those aspects of the SDGs that face dynamic complexity, how can the correct DAC criteria be better described so they are useful evaluative concepts? In that description you would of course introduce criteria such as you cite in this post.

    I confess that this exercise is difficult for me because in my practice I respond to demand. I do not see the demand for revised DAC criteria and therefore while I would agree that it may be case where you have to help people see what they need — which is what Steve Jobs brilliantly did with his Apple products. But, I think you have to be much more direct: This DAC criteria should be reformulated in X, Y and Z.

Leave a Comment

Your email address will not be published. Required fields are marked *