Evaluation and the forbidden truth – Part 1

Having spent 22 years at the helm of the evaluation function in four different UN agencies and a further five years in evaluation consultancy, I find it imperative that I should now have the courage to ask whether managers and organizations utilize evaluation results. In other words, do organizations benefit from evaluations and get value for money?

Evaluations generate data, knowledge and evidence, and identify lessons to be learned as well as good practices that have the potential to make a real difference to the way the UN and other organizations deliver assistance to the world community. The question is, do managers and other stakeholders utilize these evaluation products to the full when making decisions, developing policies, planning and designing future programmes and projects – and do they use these results in taking corrective action on current projects? Can organizations voluntarily learn from the experiences that come out of evaluations – even if the lessons derive from negative findings?

By their nature evaluations advocate for change, but are organizations ready for change? Why is it that the UN organizations, governments and others tend to resist change – even if that change is absolutely necessary for their survival? Why are evaluations even further hampered by the “Lion King” syndrome, where executives only want to hear good news and not bad news? How do evaluators break bad news to the Lion Kings who are not ready to hear such news, who resist change and continue to re-invent failure by committing the same mistakes that are raised by evaluations?

Evaluation is also impacted by the “Green Lawn Syndrome”, where executives only want a “fertilizer recommendation” without wanting to deal with the weeds in the lawn. Do evaluators know how to tell Green Lawn executives that what is needed is not just fertilizer, but a process of removing the weeds and cutting the lawn before the application of fertilizer?

The current training of evaluators appears to seek only to establish evidence in order to speak truth to power. Is this sufficient for evaluation to become an important contributor to developmental change?

But even more importantly: Are evaluators ready to accept that some evaluations are not useful because the quality is not good enough?

I raise these questions based on my observations when I analysed 55 Drugs, Anti-Corruption, Counterterrorism and Money Laundering evaluations implemented between 2003 and 2008.

Illicit drugs are a bad business and an international menace that afflicts producers, nations and consumer nations alike. Traffickers and criminals are the ones who benefit from this illicit trade.

Yet the world wants to hear good news – that the drug problem is under control, that the availability and use of illicit drugs are on the decline, and that the war is being won. But this is not always true. Evaluations often find that the opposite is the case. How then do evaluators break the bad news – that the war on drugs is not being won – to a community that is not ready for such news? How can this news be shared without discouraging those who fund drug eradication programmes, and those who work so hard to fight the drug menace?

Is this the evaluator’s cross or is it a shared burden? Can evaluators remain respected team members of an organization when they are viewed as enemies of “progress”?

Evaluations and evaluators are challenged to the utmost in these conditions. How does an evaluation tell the story of how criminals are winning by pressurising ordinary farmers into producing and trafficking illicit drugs? How do you tell the story of how “bad guys” are winning a war against “good guys” – and remain part of a team in an organization? How does an evaluator walk into a boardroom and look all those executives and managers in the face and tell them that even with their latest efforts and updated statistics the evaluation shows that things are not as good as they think?

It is hard for the evaluators to do this and remain respectable professionals and friends of the organization, even if they are armed with hard evidence and facts.

Whose dilemma is it – that of the evaluators (messengers)? Or of the world community that needs to be healed from expecting too much?

Those who run and manage organizations and sensitive programmes such as drug control, anti-terrorism and organized crime are scared of evaluations because they do not want to hear bad news. After all, no-one really wants to hear bad news.

But is telling the truth to be seen and treated as “bad news”? Programme or project managers all have to defend their programmes and organizations and prove to those who fund them that the resources they receive make a difference. They want to show that the difference is proportional, if not larger than the resources invested in these programmes. Proportionality of results to resources invested in drug or development programmes often proves elusive when evaluations are conducted.

The executives in charge are afraid of losing the already limited and often declining resources. They are afraid because they cannot prove that a further injection of resources will lead to the achievement of better results. That fear often turns into anger, and can be directed at those who have undertaken or managed the evaluations. Sometimes that anger turns into denial and rejection of evaluation results in part or in total.

Once denial and defensiveness set in, the value of an evaluation is lost. The executives throw out the baby with the bath water and shoot the messenger.

In my experience, this means that facts are no longer seen for what they are. Evidence is turned into allegations. Recommendations are rejected. The evaluation report is buried and no action is taken. The situation remains the same. Business continues as usual, no lessons are learnt and hence no improvements are introduced until the next evaluation comes along and makes the same findings, recommendations and conclusions.

So, evaluation becomes a futile exercise, a ritual that keeps evaluators employed but contributes little to development change. This is my 20 years’ story of evaluation.

I know many evaluators tremble even at the thought of saying these things.