Does evidence have a chance amidst the challenges of development?

A fair amount has been written about the apparent gap between researchers and policy-makers when it comes to ‘evidence’ – including the evidence produced through monitoring and evaluation. See for example a related post by Ian Goldman, Fred Carden’s insightful book, Knowledge to Policy, on the use, influence and impact of research in international development, and the dedicated work of organisations such as ODI.

Debates often offer opposing viewpoints. One of my personal favourites is Chris Tyler’s Top 20 things scientists need to know about policy-making response to Sutherland, Spiegelhalter and Burgman’s Top 20 things politicians need to know about science. While such discussions are interesting to observe – preferably from a safe distance – I have been reminded of their debate during a recent programme evaluation that focused on the results of capacity building programmes aimed at strengthening the use of evidence in policy-making. The challenges faced by some participants in this programme again made me wonder whether policy-makers have an interest in evidence, or whether ‘evidence-informed policy-making’ (EIPM) is supported for the sake of appearances only.

Various disciplines increasingly emphasise the importance of basing decisions on ‘evidence’. Scientific management and performance management studies focus on workflow patterns and resource use with the aim of increasing productivity and profits. In the policy sciences, policy process models advocate for a more systematic and rational approach to policy-making. In the social sciences, the need to ‘prove’ the results of social programmes gave birth to the field of evaluation in the 1960s – a field that is now blossoming worldwide.

These developments emphasize the growing need for more rigorous monitoring and evaluation practices that may generate reliable evidence for decision-making.

Davies defines evidence-based policy-making as an approach which ‘helps people make well informed decisions about policies, programmes and projects by putting the best available evidence at the heart of policy development and implementation’ (Davies, 1999). Segone contrasts this with ‘opinion based policy making which relies heavily on either the selective use of evidence […] or on the untested views of individuals or groups, often inspired by ideological standpoints, prejudices or speculative conjecture.’ (see Segone, 2008)

Policy-makers can obtain evidence from various sources, including internal and external monitoring and evaluation systems, academic research or personal experience. However, the evidence has varied credibility levels, depending on the perspective of the policy-maker of the evidence on the source, methods and credibility of the producer of evidence.

The Paris Declaration on Aid Effectiveness sparked an upsurge in national M&E systems that should feed regular information to policy-makers and other decision makers. This is further supported by democratic policy-making processes that encourage participation from citizens and various experts and stakeholders in policy-making.

However, despite increased financial and verbal commitment that recognizes the importance of evidence, we see many decisions that persistently contradict the evidence. This is more prevalent in the development context where inadequate civil society oversight and a lack of transparency in decision-making decrease the need to provide rational justifications for decisions taken.

The development context is also plagued by severe resource constraints that necessitate the selective prioritisation of needs. In these often young democracies, EIPM becomes an idealistic notion. Decisions are strongly influenced by factors like the political willingness of the government elite at a particular point in time, prior political commitments and promises that need to be kept to remain in power, and the power struggle to control resources.

Under these circumstances, the relative power of executive managers and political heads in the inner circle of decision-making supersede the potential influence of more distanced, ‘neutral’ evidence presented in the outer circle. In some cases, contrasting sources of evidence may even become a weapon in the power struggle to support preferred views, pre-determined agendas and desired outcomes.

Observing these policy decisions from a research perspective, the policy-maker seems to regard EIPM as a ‘fashionable’ term to justify existing priorities, with little interest in the evidence itself. Thus, while policy-makers may publicly commit to evidence use, their actual decisions do not reflect this commitment.

While the system is far from perfect, these are also advantages in even the superficial commitment to evidence. To obtain evidence that may support a particular viewpoint, there arises a need for dedicated time and resources within the policy process to allow the various producers of evidence to offer their evidence to the policy-making process.

If sufficient commitment exists this may even become institutionalised in a ‘diagnostic’ policy stage – that by its mere introduction creates opportunity for different sources of evidence and opposing views on the matter to be consideration. It may lead to the appointment of policy advisors that may assist political and administrative heads in their decision making, enabling the introduction of evidence at the level where key policy decisions are taken. It may even result in the (sometime symbolic) signing of MOUs between policy decision-makers and the research institutions that produce evidence, which may create communication channels and platforms for dialogue. These help to bridge the gap between ‘neutral’ evidence producers and the value-laden field of policy-making.

Within each of these instruments or mechanisms, limitations are imposed by the political sphere on the extent to which evidence may inform decisions. This can lead to disillusionment with the system unless researchers constantly acknowledge that policy-making operates in a murky terrain that is different to the notions of a relatively ‘sterile’ environment that influence the work of many researchers. Policy-makers need to find solutions to complex development problems for which no single, perfect or obvious solution exists. The essence of policy-making becomes values clarification, or in David Easton’s classic 1953 definition, ‘the authoritative allocation through the political process, of values to groups or individuals in society’.

This stands in stark contrast to accepted codes for ethical research that expect from researchers to maintain objectivity and integrity when conducting scientific research. The process of value allocation might even mean that even when the policy-maker agrees with the researcher on what needs to be done, the ability of the policy-maker to implement an ideal policy is affected by more pragmatic considerations such as the availability of the financial, institutional, human resources; the available political and managerial buy-in that will enable successful implementation of the policy; and the inevitable trade-offs against other pressing priorities.

While initial commitment towards including evidence in policy-making is often superficial and largely driven by politics or pragmatism, such symbolic commitment becomes the catalyst for the creation of an environment more conducive to EIPM. Dedicated time and resources allow for the creation of evidence; while dialogues with trusted advisors and producers of evidence might direct the policy agenda – if and when it finds accord with the viewpoint and values of the policy-maker.

As Carol Weiss concluded, despite the apparent disregard of evidence, the (research) findings often influence the way people think about a programme, an ‘enlightenment’ where research findings indirectly become a source of ideas, information and orientation to the world rather than a simple ‘recipe’ to solve the problem (http://www.jstor.org/stable/42783234).

With this understanding, available evidence does not necessarily predict the policy decision, but it enables the policy-maker over time to make more informed ‘trade-offs’ and therefore ‘better’ policy decisions.

EIPM is neither a myth nor reality. It becomes an aspiration to constantly produce reliable evidence through dedicated and committed time, resources, networks and symbolic support, while fostering formal and informal links with policy decision-makers to ensure the relevance of produced research evidence. This allows us to bridge the inevitable gap between policy-makers and researchers by ensuring that evidence is available to influence thinking and decisions when the eventual slim window of opportunity – where the political needs and priorities are ready to take up the available evidence – presents itself.