Evaluation and the forbidden truth – Part 3

Selective adoption of evaluation recommendations and lessons – There is need for evaluators to define what is meant by “LEARNING” from evaluations. There is an assumption in the evaluation discourse that there is a universal understanding on this. Superficially it may appear as if we all know what learning means.

This assumption leads us to believe that every evaluation has lessons. These assumptions lead many evaluators to believe that evaluations are valuable and should contribute significantly to the development process. Generally, evaluators conclude that it’s lack of adoption of evaluation recommendations and lessons that undermines the value of evaluation. I have been saying the same for the last 22 years. I carried out an analysis on evaluations in a UN agency undertaken some years ago, and over the years the acceptance and adoption of recommendations and lessons by the organization have been very mixed.

Many of you may disagree and treat my suggestion as heresy, but most evaluators are not critical about the quality of their work. They blame everyone else for not “learning” or not utilizing evaluation results. What I am questioning is the assumption that evaluations necessarily produce useful lessons that organizations can learn from. Evaluators need to do much more to analyse the quality and utility of evaluations.

Most evaluations reports have “lessons learned” sections. The analysis I carried out of 55 evaluations revealed that about 50 percent of all the “lessons” in the evaluation reports were not lessons at all.

My analysis concluded that these are often “ …. platitudes borne of a felt need to demonstrate engagement in the knowledge society or simply to satisfy the specified evaluation requirements. The fundamental question is, “What lessons can be learned from non-lessons”?

Before this study I had assumed that the “lessons” from evaluation were all “lessons” and that the organisation should “learn” from these evaluations. I had challenged executives and programme managers to learn from evaluations.

Now I know that the challenge is what evaluations provide as lessons. Can organizations learn anything if evaluations do not provide useful material to learn from?

There is a need for evaluation quality control. The challenge I pose is who will do “quality control” for evaluations? Who will police the police? When lessons are not taken up and recommendations are not implemented, is this evidence of poor quality evaluations or is it evidence of a lack of a culture of organizational learning?

This is a difficult area for evaluators, because we have assumed that lack of learning is always a fault of those who have been evaluated. My study concluded that sometimes – often – it may be the result of evaluators who do not produce useful products that can provide materials for organizational learning.

The current evaluation lessons need scrutiny. May the evaluators of repute be grateful that a window of opportunity has been opened on the “wall of evaluation” and take over the unfinished work and ask the question “where are the evaluation lessons”?

I am open to further learning.