Advocacy Evaluation in the Developing World

Rhonda Schlangen discusses the role of advocacy evaluation in developing countries based on her own experiences working on reproductive health and rights advocacy in Africa. The article does not generalize to all developing countries or to all issues, but is intended as one thread in a larger, longer-term conversation.

The ability of citizens to act in an organized and public way to influence the laws and policies under which they are governed is a fundamental right. This article focuses on advocacy in developing countries where resources are particularly constrained and political space and freedom of speech are often contested. In these environments, advocacy by individuals and nongovernmental (NGO) organizations is often high risk, but when successful, is also high impact. For this reason, advocacy evaluation can be a critical tool to help strengthen, support, and refine those advocacy efforts, and ultimately can help to build civil society.

This article uses two advocacy scenarios to illustrate observations about advocacy and its evaluation in developing countries. The first example illustrates the role of advocacy in a state in northern Nigeria where religious authorities are the arbiters of public policy. The second example uses the case of an advocacy coalition in Kenya to discuss how evaluation could have strengthened its high-stakes efforts to change national reproductive health policy.

NIGERIA: Evaluation that Illuminated the Importance of Advocacy
Nigeria operates under three legal systems—Sharia or Islamic law, constitutional law, and common law. In the religiously conservative North, Sharia law is the main body of civil and criminal law, and is interpreted by religious authorities. Advocacy in this environment is localized, highly contextualized, and constrained by political and religious tensions. While it appears informal because it is often conducted through personal channels and contacts, it is highly sophisticated.

Before an adolescent reproductive health services and education program in part of a predominantly Muslim state could begin, the program director first had to meet with the Imam and other religious authorities to gain their permission to work in the community. Failure to secure this permission was unthinkable. Not only was their assent critical for the staff’s security, but vocal support from the Imam would translate into community members seeking health services. There was little documentation or discussion of this advocacy work, in large part because the project’s donor was primarily interested in supporting health services, not advocacy.

About five years into the project, a formative evaluation was conducted. Because the project was positioned as a health delivery project, the Nigerian academics engaged to conduct the evaluation focused on the delivery of services. However, the evaluation’s design allowed the advocacy work to come to the forefront, and illuminated it as a critical but unfunded project strategy. Use of appreciative inquiry, which engaged a range of community stakeholders beyond health service clients in questions about the project’s achievements, enabled the evaluators to connect the ongoing advocacy with religious and community leaders as critical to the project’s success.

Although a theory of change was not developed and communications strategies were not defined, the project team did have a clearly defined advocacy goal or outcome—maintaining the support of religious leaders and eliciting community members’ support. This was critical in this cultural and political context where laws are defined and interpreted by religious leaders. The “policy” forum was the de facto forum of religious and public opinion. The proxy measure of this outcome was whether the religious leaders and community members permitted the project to operate and community members, particularly women, to seek health services. The program staff used sophisticated and innovative strategies to engage the religious leaders and community leaders, all the while not recognizing what they were doing as “advocacy.” The significance of their advocacy success is best illustrated by one community outreach worker who commented “Before this project, I would have been stoned for talking about condoms. Now, no one will stone you for talking about condoms.”

Ultimately, the evaluation results became a useful tool for dialogue with the donor agency to demonstrate the importance of community level advocacy as a precondition for the health care work they wanted to support.

KENYA: Evaluation’s Potential for Informing Strategy
In many developing countries, development assistance agencies and NGOs fund the work of domestic NGOs to implement projects aimed at promoting health, education, and other aspects of development. In these countries, the evaluation culture has grown up around international agencies’ particular demands for accountability and evaluation approaches. Expectations persist that evaluation of this development work should be shaped around a particular set of methodologies—the “gold standard” of designs that use comparison groups, baseline or endline surveys, and outcomes focused on changes in measureable health or economic indicators. Designs that do not follow this formula are judged as less reliable and thus less desirable. This culture affects advocacy evaluation, which may require unconventional or innovative approaches. In response, national NGOs engaged in advocacy, familiar with acceptable evaluation practices emphasized by their donors, tend to avoid assessments of their effectiveness altogether and focus their limited evaluation budgets instead on methods that assess only whether they have implemented the plan promised to donors, an approach that does not advance their understanding of the advocacy work.

This is what happened to a Kenyan coalition of doctors, nurses, lawyers, and human rights advocates who organized around the need to reverse the dangerous trend of unsafe, illegal abortion in the country. One of the group’s primary efforts was to draft legislation for a sweeping range of reproductive health and rights policies. Promoting policy change on this issue was a high-stakes proposition because the effort took place in the context of a crowded public agenda during a period of particular political and social volatility following 2008 post-election violence.

The group’s donor funding included support for an external, formative evaluation of the coalition’s work. The evaluation focused almost exclusively on the group’s plans and commitments to donors and whether they had been fulfilled in a timely manner. It did not assess progress toward advocacy goals, identify strategic strengths and weaknesses, or otherwise examine whether the coalition’s plan was appropriate for the political and social environment at the time. This was a missed opportunity to use evaluation for strategic learning, especially because the evaluation took place at a critical point just prior to a key policy action. The well-organized, highly functioning advocacy alliance could have used the evaluation to learn about the potential advantages and disadvantages of their approach.

Shortly after the evaluation was conducted, in a disastrous series of events, news of the draft legislation was shared by well-intentioned group members with virulently opposed politicians. Coalition members were subsequently pilloried in the media as “promoting birth control for children.” Unprepared, the group scrambled for a response while their opposition framed the story that played out in the media for days. Politicians and church leaders publicly denounced the draft legislation that had not even been introduced in parliament. Ultimately, the political opportunity was lost, and in such a way that arguably weakened the position of the alliance. The legislation that was to help address the needs of the one in 39 Kenyan women who die of pregnancy-related causes never left the drafting table.

This example illustrates a lost opportunity with advocacy evaluation. While many factors contributed to this loss, a well-designed, well-timed evaluation that strayed from the prescribed traditional focus could have contributed essential learning to the coalition’s efforts. For example, an evaluation that focused on the alliance’s positioning, public perceptions related to key issues, and strategic use of the alliance’s political capital, could have helped it define and organize strategies that would help it move beyond joint action to impact.

Conclusions
While the range of experiences, capacities, and political cultures of NGOs cannot be captured in a simple article, these case examples support several general observations about the potential contribution of evaluation to the advocacy efforts of citizens’ organizations:

• Evaluation is an essential tool for understanding the role of advocacy in societal change. As the Nigeria example illustrates, evaluation can illuminate advocacy and related strategies in supporting program progress or the integral role it plays in creating the necessary conditions for the program to exist. Used in this way, it can both help community NGOs recognize what they are doing as advocacy and help them make the case to donors and supporters for including advocacy in the fabric of program funding.
   
• Advocacy evaluation capacity and tools are necessary to expanding civil society. Organizations with few resources and limited political capital cannot afford to expend efforts on ineffective advocacy. Greater advocacy evaluation capacity, supported by tools that assist in planning and monitoring progress as well as impact, are essential to support the brave and important work of these individual s and organizations. Particularly for young NGOs who are learning to expand political influence, evaluation can help them effectively plan strategies that are consonant with their skills, capacity and resources.
   
• Interim outcomes may be the most important outcomes. In some countries, particularly amongst NGOs who are in the early stages of development, policy change is an unrealistic outcome within a short-term timeframe. For these organizations, developing the contacts and nodes of influence, and gaining a seat at decision making tables, are significant interim advocacy achievements. In some environments, it may take several years of concerted effort just to focus public attention on particular issue, especially if it has a whiff of controversy. The issue of interim outcomes is particularly concerning for young NGOs, who feel pressure from their international donors to produce policy change results. As Regina Thames, a human rights lawyer for the Planned Parenthood Federation of America who works with small NGO organizations throughout Latin America, notes, “Advocacy processes are not valued that much. Donors only care about policy change outcomes.”
   
• Donor flexibility and support is also required. International donors need to allow for evaluation methodologies that are suited to the learning needs of their grantees. Donors must recognize that methodological choices should be driven by the evaluation questions that are asked and the context in which they are asked, rather than assume that one approach is the best in all situations.

While advocacy evaluation in developing countries still suffers from many of the same reputational and methodological limitations that it does in the United States, the innovative, risk-taking nature of many developing country NGOs presents an exciting opportunity to help grow the advocacy evaluation field.

Rhonda Schlangen is an independent evaluation consultant specializing in advocacy evaluation. She emphasizes work with international and U.S. non-governmental organizations and networks to develop practical strategies to evaluate their advocacy for policy change work. Email: rschlangen [at] yahoo [dot] com. Tel: 802 645-9798.

« Back to AEU #8