Musings on a Think Tank for Evaluation Capacity Building

by Jennifer Bagnell Stuart

On October 2-3, I participated in a diverse convening of evaluation practitioners and researchers to discuss the merits of “evaluation capacity building,” or ECB.  The two-day “think tank,” held in Minneapolis, was made possible by the National Science Foundation and included enlightened funders from private foundations as well as government agencies, nonprofit/ grantee organizations and national agencies, university based researchers, and independent consultants and consulting firms that provide program evaluation and ECB services. On that last point, we clarified that there is, indeed, a distinction between an evaluator and an “ECB provider.”

So what is capacity building, anyway? Simply put, capacity speaks to an organization’s ability—in terms of resources and know-how—to effectively and strategically run the programs that deliver on its mission. Building organizational capacity can manifest in many different ways, from board development and strategic planning to fundraising and IT.  Given the myriad challenges of running a nonprofit organization, developing the organization’s evaluation capacity frequently does not make it to the top 10 of management concerns.  Presenters Michael Baizerman and Donald Compton offered two definitions for ECB, one of which approximated plain English: “ECB is the intentional work to continuously create and sustain organizational processes that make quality evaluation and its uses routine.”  Nice job.

The stated purpose of the Think Tank was to bring together diverse perspectives and stimulate dialogue about the burning questions and challenges in building evaluation capacity and promoting evaluation use, and to lay the groundwork for articulating opportunities, issues, and next steps to further funder support of evaluation capacity building.  Given that ECB is central to Innovation Network’s mission, the Think Tank was a great opportunity for us to learn more and connect with others who are doing important work in this sub-specialty.
 
The Think Tank organizers kicked things off by presenting a working draft diagram titled “Relationships among Funders, Grantees, and Evaluation” that was used and referenced throughout the convening to provide context for the discussions. The diagram was particularly useful for me in underscoring that nowadays, while “program evaluation” is almost always required in grant-funded work regardless of whether the funder pays for it, “evaluation capacity building” is completely optional and more often a luxury afforded by enlightened funders.  This begs the question, what would happen if ECB were never part of the equation?  Alternatively, what if ECB were required, rather than optional?  Would programs, and the capturing and documentation of their results, be worse off, better off, or neutral?  By the end it was clear to me that developing a theory of change for why we need ECB, and why funders should support it, is necessary to drive the agenda for the Think Tank.

There is a contingent of committed individuals and groups—Innovation Network among them—who believe that the value of evaluation goes far beyond demonstrating results and being accountable.  The representatives who attended the Think Tank appeared fairly united in the view that a strictly “Outcomes” and “Performance Measurement” focus that increasingly dominates the conversation between funders and grantees falls far short of the complexities and nuances of the grantmaking relationship.  It also completely fails to acknowledge the equally pressing need for funders to assess and build their own capacity to evaluate their grantmaking programs and reexamine the demands they place on their grantees.

The format of the Think Tank consisted of presentations on both days, with time for Q&A following the presentations, and several opportunities for small group discussions and networking.  Lots of food for thought for a two-day brainstorming session.   Some highlights:

• A presentation by Margaret Plantz, a tireless advocate for United Way agencies and all frontline organizations, drove home the point that funders should only ask organizations to report data that those organizations will actually use to advance their programmatic work.
  
  
• Tom Kelly of the Annie E. Casey Foundation, Teresa Behrens of the Kellogg Foundation, and Ruth Brousseau, formerly with The California Wellness Foundation, each spoke candidly (bless their hearts!) about the internal dysfunctions and challenges that foundations experience when it comes to dealing with the conflicted issue of evaluation.
  
  
• Ricardo Millet, formerly with the Woods Fund and Kellogg Foundation, delivered a keynote address that identified the cultural and structural challenges of learning from evaluation at the foundation level, program level, and grantee level. Millet spoke with conviction about the need for foundations to recognize that nonprofit organizations and the communities they serve are the source of “authentic knowledge” and as such should be the drivers of change. He nicely crystallized the Think Tank’s theme of locating evaluation capacity building in the context of the foundation-grantee partnership, which centers on tensions and issues of trust, respect and an unequal balance of power.
  

In sum, it was very much a “think tank” in that it is in the formative, idea-development stage.  Things felt a bit muddled at first, but began to take some shape toward the end of the second day when people offered concrete ideas for where the conversation might go.  Some of the main next steps are to write publications, hold another convening to continue the discussion and networking and to start a TIG for ECB at AEA.  [Huh?  Translation: start a Topical Interest Group for Evaluation Capacity Building at the American Evaluation Association Conference this November in Portland. Maybe the next step should be engaging a marketing consultant to help us avoid jargon…]

For more information on evaluation capacity building, visit the Resources section at Point K.