*




BOND Guidance Notes Series 6

Monitoring and Evaluating Advocacy

This is the fourth in a set of four advocacy guidance notes, which can be read as a series or separately.

The Basics

Monitoring and evaluating advocacy work is relatively new territory for NGOs and donors. A great deal of time is spent modifying other evaluation systems to try and meet the unique and opportunistic characteristics of advocacy. The best attempts are likely to involve you collecting what ever information you can as you go along - orally, scribbling down notes and filing them, talking into a tape etc. so you have evidence to back up your arguments or hunches when you need to. The onus should be on doing it rather than worrying about doing it.

Monitoring is the collection of information about a project over time. It seeks to understand what is happening.

Evaluation is an assessment of the project at one point in time, including the successes and failures. It seeks to understand why what happened, happened.

Why monitor and evaluate your advocacy work?

The initiative to begin monitoring and evaluating your advocacy work may come from a variety of motivations. The reason why you are evaluating your work will determine the techniques you use.

  1. So you can produce credible funding reports. The need to prepare credible reports for funders and managers is the reason most people evaluate their work. You need to show that you used the inputs well to achieve certain outputs. You need to develop reasoned arguments about how undertaking these activities contributes to your long term objectives and to improvements in the lives of certain groups of people. Looking at the effects of your outputs is as near as most people get to evaluating the impact advocacy work has on peoples' lives. You may be able to count the number of messages you've put out, but how do you know whether your messages are being correctly understood? Again evidence recorded along the way can help - has your target audience changed their party line or tone? Are they quoting your arguments in their literature?
  2. To demonstrate to managers, colleagues and Southern partners that advocacy work is a cost-effective way of improving the lives of poor people. How do you follow-up on policy changes, when you have defined them as your short-term objectives, to ensure that they really make a difference in people's lives? This is very hard because of the long term nature of change. It is very difficult to attribute changes in policy or practice to any one organisation or coalition rather than to external factors. Any attempt to understand the affect of policy change on peoples lives (which are also subject to influence by a myriad of factors) can only involve small numbers of people because of the costs involved. The best and most cost-effective way of attempting this is to produce a case study that draws on the experiences of a range of different stakeholders.
  3. To learn from experience. No two-advocacy projects are the same. Deciding what succeeded or failed is often a case of looking at which mix of strategies worked in a given situation or comparing how different strategies influenced a particular target audience in a range of projects. Learning is best done by reflecting regularly on whether you are achieving your outcomes and collecting anecdotal or other evidence to support your assumptions. You also need to monitor the external situation so you can recognise and record the other factors that may have influenced your target audience. This type of reflection is a useful way of fine tuning the next steps in your strategy and building up your advocacy skills. Often it is more useful to focus on why you changed plan than to spend time collecting data on pre-determined indicators.
Inputs, Outputs, Outcomes and Indicators

An essential ingredient of any advocacy strategy is an action plan. This is likely to be a table showing your inputs and outputs;

INPUTS. The resources you will be using - often simply in terms of staff and production costs

OUTPUTS. The activities you will be undertaking.

To build learning, reflection and flexibility into your plans it is useful to add two further columns.

OUTCOMES. The impact you expect to achieve from each of your outputs

INDICATORS. The evidence you will collect to show the outcome has been achieved.

top


Example: Action Plan for the first 6 months of WaterAid's advocacy campaign on Vision 21 and the World Water Forum,

In March 2000, governments, NGOs and research institutions in the water resources sector will gather in The Hague to agree a Vision for the 21st century and a Framework for Action. This will be the first time that targets for water resources will be agreed internationally. Governments are expected to pledge to these targets. WaterAid's campaign aims to ensure the inclusion of WaterAid's position in these agreements and to lead a NGO coalition that will commentate on and monitor the implementation of the delivery of Vision 21 and its Framework for Action.

Input

Output

Outcome

Indicator

Recruit researcher,

research time, money to pay

(first 2 months),

consultation time with internal stakeholders.

Produce WaterAid position statement.

Agreement by all internal stakeholders of WaterAid, including Southern partners on key issues and positions on these issues.

Position statement made public.

OS partners use position statement to assist their own national advocacy work.

Time in meetings and consultations.

Submission to and consultations with WSSCC drafting committee.

Research UK government position on key issues tackled in position statement.

WaterAid position be included in V21 and Framework for Action documents of the WSSCC

PS as basis for lobbying UK government and WB re their positions on water / sanitation provision.

Draft V21 includes WaterAid positions re: key issues which it tackles.

Research time, meetings.

Meetings with academics and other NGOs.

Seminar on WaterAid's water / sanitation position

Revival of NGO-academic network in water / sanitation sector.

Agreement on key issues and position.

Issues and positions being written about in specialist water / sanitation journal.

Strong message at WWF on key issues and position.

(Case study developed by Belinda U. Calaguas, March 1999, as part of the BOND training

What kind of monitoring indicators are there?

The range of indicators is infinite.
Try at least a couple of techniques and aim to monitor the change across a range of your stakeholders. Remember each stakeholder will have different motivations that may affect the data you collect. For instance decision-makers may like to give the impression that they are beyond influence in order to discourage further lobbying. Remember, also that the 'results' of such monitoring will not directly reflect the quality of your work. Within both news and policy environments your ideas and issues will be competing with many other stories. Whether that leaves 'space' for your issue is largely a question of chance.

Monitoring your target

  • Record and observe changes in the rhetoric of your target audience. Keep a file of their statements over time.
  • What are they saying about you and your campaign?
  • Are they moving closer to your position, adapting to or adopting any of your language or philosophy??

    Monitoring your relationships

  • Record the frequency and content of conversations with external sources and target audiences.
  • Are you discussing new ideas? Are you becoming a confidante or a source of information or advice?

    Monitoring the media

  • Count column inches on your issue and the balance of pro and anti comment.
  • Count the number of mentions for your organisation.
  • Analyse whether media is adopting your language.

    Monitoring your reputation

  • Record the sources and numbers of inquiries that you receive as a result of your work.
  • Are you getting to the people you wanted to get to?
  • How and where have they heard of your work?
  • How accurate are their pre-conceptions about you and your work?

    Monitoring public opinion

  • Analyse the popular climate through telephone polling, or through commissioning surveys.



    What should I be trying to evaluate?

    To evaluate the impact of your project you need to be clear about the model or process you are trying to follow and then decide on what information is available to enable you to assess each part of the process. Recent work by the New Economics Foundation suggests you think of the process as an Impact Chain.

    Build awareness > Change policy > Impact on peoples' lives

    In each phase there are policy and grass roots activities, both of which need to be monitored. The relationship between these activities is also important, the more integrated they are, the more successful the project is likely to be. Grass roots activities are likely to involve capacity building activities. For example:

    Group formation > Group activities > Group federation beyond village level > Movement launched which takes on vested interests > Groups of poor are involved in framing legislation and have control over resources

    Policy activities focus more on raising awareness and changing attitudes. For example:

    Heightened awareness about an issue > Contribution to debate > Changed opinions > Changed policy > Policy change implemented > Positive change in peoples' lives.

    Who defines success?

    Different stakeholder's will have different views on what success is, depending on where they are within the impact chain. To get an overview of how successful you were you need to solicit the views of a range of stakeholders i.e: ultimate beneficiaries, local people and their organisations, staff involved, target audience, journalists and outsiders. This can be done using a variety of methods. For example:

  • Surveys can provide an overview of what was achieved
  • Interviews bring in the perspective of different stakeholders
  • PRA techniques such as ranking are useful for assessing the success of grassroots activities
  • Video can be an effective way of keeping the emotion and reality in evaluations, without which spirited campaigns turn into a dry reports
  • Case studies that draw on a range of techniques and that are cross-referenced to avoid bias are a helpful way to provide useful lessons and to present complex material. These can be done for specific projects or institutions or groups of beneficiary's etc.

    Attribution is one of the hardest issues to face in evaluating advocacy work. It is very difficult to know precisely what causes policy changes and precisely what impact those changes have in reality. Many different forces are in play, and NGOs are often amongst the least powerful actors advocating in any situation.

    Questions you might want to ask to help you assess impact and attribution

    AUDIENCE

    CLIENT (i.e. poor people expected to benefit from advocacy project)

    Who was supposed to hear the message?

     

    Who has heard the message?

     

    How did they interpret the message?

     

    How was it different from other messages?

     

    What did they do in response?

     

    Have they heard of the sender?

     

    How do they differentiate the sender from others who might be sending similar messages?

    If clients are not already working with the NGO how are they contacted in order to ensure that the NGO is acting appropriately on their behalf?

     

    To what extent have NGOs who are involved in advocacy projects explained their advocacy activities to poor people they are working with?

     

    Has there been any attempt to get these people to rank advocacy work versus other activities that they might see as more relevant?

     

    What effort has been made to provide feedback to the same people about the results of advocacy work?

     

    To what extent do people feel more confident about their capacity to advocate on their own behalf?

     

    What effort has been made to seek their assessment of results and their confirmation of assumed impact?


    (From a model developed by Rick Davies)

    Top Tips

  • Be clear about your objectives.
  • Record concrete outputs and inputs
  • Identify simple indicators for each output, which do not require huge amounts of time or effort to collect.
  • Track changes in the external environment to see if these are relevant.
  • Systematically collect any concrete evidence especially anecdotal evidence
  • Remember assessments need to provide useful and timely information
  • Crosscheck subjective judgements by asking audience and beneficiaries key questions.
  • Evaluation can mean simply making your own instinctive judgements and thoughts more overt. This can help you to know your own mind and can help you to share information and opinions with colleagues.

    Bibliography

    Baranyi, S., Kibble, S., Kohen, A. and O'Neill, K. (1997) Making solidarity effective: Northern voluntary organisations, policy advocacy and the promotion of peace in Angola and East Timor. London: Catholic Institute of International Relations. (One of the few efforts to grapple with effectiveness - in terms of relative/comparative effectiveness of different campaigns.)

    Covey, J. (1995) "Accountability and effectiveness in NGO policy alliances", pp.167-182 in Edwards, M. and Hulme, D. (eds.) Non-governmental organisations - performance and accountability: Beyond the magic bullet . London: Earthscan. (Early discussion of effectiveness - suggests some avenues for further progress.)

    Fowler, A. (1996) Demonstrating NGO performance: Problems and possibilities. Development in Practice, Vol.6, pp.58-65. (Discussion of some of the problems faced by NGOs in evaluating performance - multiple stakeholders, no bottom line etc.)

    Fowler, A. (1995) Participatory self-assessment of NGO capacity. INTRAC Occasional Papers, Vol.10. (User-friendly guide to evaluation. Contains some useful conceptual frameworks, and ideas about indicators.)

    Fox, J. and Brown, D. (1998) The struggle for accountability: The World Bank, NGOs, and grassroots movements. MIT Press. (Big book, with much information about World Bank - NGOs relations. Contains some discussion of evaluating impact and effectiveness.)

    Herman, R. and Renz, S. (1997) Multiple constituencies and the social construction of nonprofit organization effectiveness. Nonprofit and voluntary sector quarterly, Vol.26, pp.185-206. (Theoretical discussion of what effectiveness might mean for nonprofits - multiple stakeholders etc.)

    Rees, S. (1998) Effective nonprofit advocacy. (One of the few empirical attempts to assess effectiveness. Focus is on lobbying in US domestic politics, but is useful re operationalizing and studying effectiveness.

    Roche, C. and Bush, A. (1997) Assessing the impact of advocacy work. Appropriate Technology, Vol.24, pp.9-15. (Short article exclusively about assessing impact and effectiveness. Argues that evaluating effectiveness is very important and makes some suggestions for progress in this area.)

    Bibliography prepared by Dr. Alan Hudson, March 1999 Acknowledgements

    Information contained within these Guidance Notes was brought together by Megan Lloyd laney, Jane Scobie and Alastair Fraser for the purposes of a training course, 'An Introduction to Advocacy', run through March and April 1999. The course was designed as a participative and discursive process and so a huge debt is owed to all the participants and to the 'resource people' used during the course to present case-studies of innovative work and best practice. These notes draw heavily on unpublished materials from Chris Roche of OXFAM. Thanks to the author for permission to use the information.

    Disclaimer: BOND Guidance Notes aim to encourage good practice through practical advice, however, BOND cannot be held responsible for the outcome of any actions taken as a result of the information contained in the Guidance Notes series.



  • BOND - Networking for International Development | BOND 2005