|
Advocacy Evaluation Session Summaries from the American Evaluation Association’s 2008 Annual Conference
The American Evaluation Association ("AEA") 2008 conference featured a wealth of advocacy evaluation lessons. The Advocacy and Policy Change Topical Interest Group had a major presence at November’s AEA conference, sponsoring fifteen sessions. Funders, advocates, and evaluators talked about their experiences with effective advocacy evaluation approaches and ways to build capacity. We are pleased to bring you the insights presented at nine of those sessions—many thanks to the presenters for sharing their materials with Advocacy Evaluation Update. Session materials are available through the Point K Learning Center (free login required).
Sessions included:
1. Online Evaluation Resources
2. Striving for Alignment: One Funder's Lessons in Supporting Advocacy
3. Complex Challenges in Evaluating Advocacy: Internal Governance Structures and Public Policy Dispute Resolutions
4. Practical Guidance and Tools for Advocacy Evaluation
5. Assessing Foundation Communications: A New Tool for Practitioners
6. Evaluating the Effectiveness of CARE USA's Advocacy to Promote International Health and Development Programs
7. Practical Methodology for Evaluating Advocacy Efforts
8. Expanding Advocacy Capacity: Findings from the Evaluation of The California Endowment Clinic Consortia Policy and Advocacy Program
9. Evaluating Policy Efforts through Systems and Organization Theories
1. Online Evaluation Resources
Johanna Gladfelter Morariu and Susan Hoechstetter
Advocates need tools and resources to help assess their work. This session provided an overview of what’s currently available online, including:
Session materials:
Johanna Gladfelter Morariu’s presentation slides and handout
Sue Hoechstetter’s presentation slides
Johanna Gladfelter Morariu’s presentation of Continuous Progress’ slides
If you have questions about this session, please contact Johanna Gladfelter Morariu (jmorariu [at] innonet [dot] org).
« Back to top
2. Striving for Alignment: One Funder's Lessons in Supporting Advocacy
Ehren Reed and Tanya Beer
Alignment between funders and grantees is very important in advocacy work. It eases the burden of evaluation reporting for grantees, and helps nonprofits and foundations move closer to realizing a common vision for success. This demonstration highlighted the ongoing efforts of The Colorado Trust—with help from its evaluation partner, Innovation Network—to incorporate principles of alignment into its grantmaking approach. The session outlined the unique evaluation methodology Innovation Network has designed for this project. Funder and evaluator both shared lessons learned, including:
- True alignment should bridge not only across grantees working towards the same ends, but across funders as well.
- Striving for alignment requires willingness and creativity on the part of the funder, open lines of communication, and a great deal of trust between the funder and its grantees.
Session materials:
Tanya and Ehren’s presentation slides
If you have questions about this session, please contact Tanya Beer (tanya [at] coloradotrust [dot] org) or Ehren Reed (ereed [at] innonet [dot] org).
« Back to top
3. Complex Challenges in Evaluating Advocacy: Internal Governance Structures and Public Policy Dispute Resolutions
Bonnie Shepard, Maureen Berner, and John Stephens
Bonnie Shepard presented about her experience examining governance issues in coalitions. Governance issues are an essential aspect of many advocacy evaluations. The study describes several aspects of internal governance, such as resolving tension and clarifying decision-making, that play an important role in successful coalition work.
Maureen Berner and John Stephens presented their findings that evaluations of public policy dispute resolution (“PPDR”) and public participation (“PP”) have some similarities, but several important differences, including:
- How agreements are reached;
- How broad the scope of participation is;
- How constituencies are represented; and
- Who has decision-making authority in each context.
Session materials:
Bonnie’s presentation
Maureen and John’s presentation
Maureen and John’s draft paper: Learning From Your Neighbor: The Value of Public Participation Evaluation For Public Policy Dispute Resolution.
If you have questions about this session, please contact Bonnie Shepard (bshepard [at] ssds [dot] net) or John Stephens (stephens [at] sog [dot] unc [dot] edu).
« Back to top
4. Practical Guidance and Tools for Advocacy Evaluation
Anne Gienapp, Tom Kelly, Kendall Guthrie, Jane Reisman, Sarah Stachowiak, Justin Louie, and Catherine Crystal Foster
Anne Gienapp, Kendall Guthrie, and Justin Louie presented lessons and guidance derived from a variety of advocacy capacity building efforts. Support for these efforts was provided by the Annie E. Casey Foundation, the Bill and Melinda Gates Foundation, and The California Endowment.
Key points:
- Tools are not a magic bullet. Successful advocacy evaluation capacity building requires both tools and education—and capacity must be built among grantees, foundation program officers, and evaluators.
- There is a tension between consistent approaches and innovation. There is a real need for efficiency and consistency in language, expectations, and approach. At the same time, the field needs new ideas to thrive.
- Justin and the other evaluators from Blueprint Research & Design focused on relationships. They found that community organizers were more engaged and productive when evaluators gave them concrete examples, templates, and tools that dealt with the kinds of relational dynamics that are most important and familiar to organizers.
- Sustained evaluation requires organizational stability and commitment.
Session materials:
ORS’ Paper: Real-life Lessons Learned and Resources in Building Capacity for Advocacy and Policy Evaluation among KIDS COUNT Grantees.
More information:
About Organizational Research Services (ORS)
ORS report: A Guide to Measuring Advocacy and Policy
About Catherine Crystal Foster
About Blueprint Research & Design
If you have questions about this session, please contact Anne Gienapp (agienapp [at] organizationalresearch [dot] com), Justin Louie (justin [at] blueprintrd [dot] com), or Catherine Crystal Foster (catherine [at] blueprintrd [dot] com).
« Back to top
5. Assessing Foundation Communications: A New Tool for Practitioners
Presenters: Edith Asibey and Justin van Fleet
The presenters unveiled a new guide to help foundations and nonprofits evaluate their communications. The guide, Are We There Yet?, was created by Asibey Consulting for the Communications Network, with support from the David and Lucile Packard Foundation.
Are We There Yet? features nine practical steps that will help users to develop their own communications evaluation strategy. The guide combines elements of planning and evaluation, and it focuses on progress evaluation and the need for course corrections. Several foundations offered examples of evaluation in action as features in the guide, which also offers suggestions on choosing the right evaluation techniques and budgeting for evaluating communications.
More information:
About Asibey Consulting
Are We There Yet?: A Communications Evaluation Guide
If you have questions about this session, please contact Edith Asibey (edith [at] asibey [dot] com).
« Back to top
6. Evaluating the Effectiveness of CARE USA's Advocacy to Promote International Health and Development Programs
Presenters: Carlisle Levine, Julia Coffman, and Edith Asibey
One of CARE USA’s advocacy evaluation challenges is defining what qualifies someone (in this case, any given member of the U.S. Congress) as a “champion” for a particular cause. Once someone is a champion, the challenge becomes how to assess changes in degree—how much more of a champion is that person than they were last year? During the session, participants offered some suggestions:
- Use a participatory approach: Ask U.S. Congressional staffers to help define indicators and develop a scale.
- Consider a functional definition: In community organizing, a leader has been defined as someone who creates followers who become new leaders.
- Be clear about the end goal, since this will help identify the qualities sought in a champion: Is the goal to create long-term support for issues within the U.S. Congress, and/or is the goal to gain support for current legislation?
If you have questions about this session, please contact Carlisle Levine (clevine [at] care [dot] org), Edith Asibey (edith [at] asibey [dot] com), or Julia Coffman (jcoffman [at] evaluationexchange [dot] org).
« Back to top
7. Practical Methodology for Evaluating Advocacy Efforts
Presenters: Annette Gardner, Claire Brindis, Lori Nascimento, Sara Geierstanger, and John Risley
This session featured two presentations about advocacy and policy change evaluation approaches: one on using case studies, and one on “second tier” advocacy efforts.
Using Case Studies to Evaluate Policy and Advocacy: Annette Gardner, Claire Brindis, and Sara Geierstanger are evaluators with the Philip R. Lee Institute for Health Policy Studies at the University of California, San Francisco. They worked with grantees of the California Endowment’s Clinic Consortia Policy and Advocacy Program to develop stories of exemplary practices. These narratives—though focusing on different program areas, such as policy advocacy, developing partnerships, or improving quality—shared crosscutting themes and progress in achieving longer-term outcomes. Next, the evaluators developed in-depth case studies of three different policy-change initiatives, with a focus on how advocacy contributes to policy change. The findings indicate that the program has afforded grantees an opportunity to experiment and develop novel and sustainable solutions based on their communities’ unique needs. Factors found to be important to achieving a policy change included:
- Staff expertise
- Early and frequent participation during the policy process,
- Business acumen,
- Coalition-building and stakeholder mobilization, and
- Making effective use of partnerships with member clinics.
Second Tier Advocacy and Policy Change Evaluation: The Greater Kalamazoo United Way pursues policy advocacy by supporting other advocacy organizations. The presenter argued that for such an approach to be effective, the United Way must devote appropriate resources to the evaluation of the advocacy efforts of their partners. The paper on which this presentation was based proposes a checklist for advocacy evaluation approaches, including output-focused short- and intermediate-term measures and long-term outcomes that are more directly related to policy change.
Session materials:
Paper: Achieving a Policy Change: Key Strategies and Factors for Success.
If you have questions about this session, please contact Annette Gardner (annette [dot] gardner [at] ucsf [dot] edu) or John Risley (jrisley [at] gkuw [dot] org).
« Back to top
8. Expanding Advocacy Capacity: Findings from the Evaluation of The California Endowment Clinic Consortia Policy and Advocacy Program
Presenters: Annette Gardner, Claire Brindis, and Lori Nascimento
In this demonstration, the presenters described the design and results of their six-year evaluation of The California Endowment’s Clinic Consortia Policy and Advocacy Program. The Program funded 19 California clinic consortia (alliances of local clinics, designed to help communities develop healthcare systems that fit their needs). Program grantees undertook policy advocacy activities and worked to improve the financial stability of their member clinics. The Institute for Health Policy Studies at the University of California, San Francisco has evaluated these activities since 2002, using quantitative longitudinal measures and qualitative interviews with grantees, partner organizations, the media, and advocacy targets.
The findings indicate that grantees are achieving not only short-term outcomes, such as increased policymaker awareness of clinic policy issues, but also longer-term outcomes, such as increased funding to clinics and increased access to care.
The evaluation findings also offer broader insight into successful advocacy. The most effective advocacy activities varied from 2002 to 2006. Over time, the grantees became more involved in the policymaking process—for example, helping to draft rules and regulations. In the early years of the grant, the emphasis was on developing or strengthening relationships with policymakers, and as a result many grantees participated on commissions and taskforces. In addition to sharing findings on achievement of outcomes, the presenters also described four data collection tools they used to assess changes in advocacy capacity.
Session materials:
Annette’s presentation slides
If you have questions about this session, please contact Annette Gardner (annette [dot] gardner [at] ucsf [dot] edu).
« Back to top
9. Evaluating Policy Efforts through Systems and Organization Theories
Presenters: Cindy Roper, Mary Kreger, Claire Brindis, Dana Hughes, Simran Sabherwal, Katherine Sargent, Christine MacFarlane, Annalisa Robles, and Marion Standish
In this session, Cindy Roper of Clemson University’s Charles H. Houston Center argued that organizational theory is useful as a way to view accountability and public policy. Organizational theory highlights interactions between program performance and the context in which the program operates. The study of these interactions can provide insight into both the challenges and the accomplishments of program performance.
Session materials:
Cindy’s presentation slides
Paper: Accountability and the No Child Left Behind Act: Implications for Public Policy.
If you have questions about this session, please contact Cindy Roper (cgroper [at] clemson [dot] edu).
« Back to top
« Back to AEU #5
|
