Posts Tagged ‘evaluation’

 

GlobalGiving’s Storytelling Project

Posted by john hecklinger on December 13th, 2010

GlobalGiving has a modest budget and team of around 25 people, all in one room in Washington, DC, but we face challenges similar to those of the largest of institutions involved in philanthropy and international development.  One of the biggest is assessing the impact of what we are doing.  With over 1,000 organizations implementing small projects in over 100 countries, it is impractical for our team to study each project’s impact in scientific detail.

Beyond easy measures of donation flow and reporting compliance, how do we know whether the marketplace we’ve built actually accomplishes something of substance in the world?  Which organizations are doing great, and which are struggling?  How can we celebrate the former and assist the latter?  Does all the work we do in cooperation with our on-the-ground partners all add up to something?  Are we sparking and fostering innovation?  Are the organizations participating in the GlobalGiving marketplace different from other organizations in positive ways?

In cooperation with Rockefeller Foundation, Cognitive Edge, and independent consultant Irene Guijt, GlobalGiving has found an promising way to tackle this problem.  In Kenya, we launched the GlobalGiving Storytelling Project, which asked people to tell stories about community projects and the individuals, organizations, and government entities working to make change happen.   We gathered 2,700 usable stories from individuals primarily in Nairobi and the Rift Valley.

Using the SenseMaker® methodology of capturing people’s stories and asking those people to “tag” their own stories, we are able to see how thousands of stories relate to each other.   We can visualize patterns in the stories that help us understand how people see organizations working in their communities.  Our next step is to make this method available more widely, providing a toolkit that helps our project leaders learn more about how people see them by launching their own storytelling projects.  We want this to be a useful way for GlobalGiving, our partner organizations, and beneficiaries to .

This pilot has huge promise, not just for GlobalGiving but for the philanthropic and development sectors as a whole.  Much has been written about how the lack of quick feedback hinders development work.  See Owen Barder’s recent blog post.  Like marketers of soda or electronic gadgets, how can funders of development initiatives quickly measure performance and make real-time adjustments to meet real needs in efficient ways?  Our pilot is a promising way of establishing meaningful feedback to power this type of real-time learning .

Marc Maxson, GlobalGiving’s chief feedback loop instigator and impact assessment innovator, has pulled together online resources that show how our pilot worked and what we’ve learned.  We will add further resources as our approach evolves.  Our next step is to expand our work in Kenya and to begin working with organizations in Uganda and Tanzania.

Are you looking for a way to learn more about how people view your work?   Are you struggling to find an effective and inexpensive way to evaluate your impact?  Please contact Marc (mmaxson@globalgiving.org) to learn more about getting involved.

Beyond Good Intentions: Getting more visitor postcards from the field

Posted by Marc Maxson on August 28th, 2009

Tori Hogan at the Social Edge blog “beyond good intentions” recently wrote about the problems with getting reliable feedback from projects in international development, and what GlobalGiving has been doing to improve feedback. Here is an excerpt:

“My old stats professor will probably kill me for this, but what ever happened to good old-fashioned gut instincts? Do you trust anecdotal reports by people who have visited field projects and come away with certain perceptions?

I started considering this concept recently when Marc Maxson at Global Giving introduced me to a new project they’re running that allows people who visit any of their countless field sites to submit ‘postcards from the field. These blog-like reports written by non-professionals (mostly by interns and student travelers) have an open-ended framework and are only guided by the question,’”what would you tell your friends about this project?’ According to Marc, ‘gut feelings about recommending a project are broad enough to predict deeper problems.’”

Tori dug through our more than 70 postcards from 2009 and found that this new Visitor Postcards program still needs more volunteers to scale up. I quote again from that post:

“However, as I scrolled through the “postcards” I had a really hard time finding any that were critical of the organizations they highlighted. Are visitors afraid to report on problems or are these organizations actually as perfect as they sound? Marc informed me that, ‘most volunteers try to self-filter and only say good things publicly, but privately send in negative comments.’ Well, it’s not perfect, but at least it’s a start!”

This remains true. The vast majority of comments on projects are positive, although I can think of two organizations who hosted visitors this year that posted negative comments and triggered larger dialogues. But anyone doing a spot check on our postcards is bound to miss them because they are not a significant fraction. :)

Bloggers and the public are our advisors. I personally thank Tori and look forward to doing what I can to get more honest feedback from visitors onto the site. Visitor postcards have played a major role in the outcome of one organization, and we will be presenting a case study on this at the upcoming Skoll “International Social Innovation Research Conference.”

I replied to Tori’s post:

We think visitor postcards have been a major success because more people are getting a first-hand account of what projects look like. And (as our case study will show) it also works to reform a problem project. The power of real-time feedback loops was enough to cause the organization visited to dissolve and reform under new leadership of a group of underserved beneficiaries. This happened in spite of the “self-filtering” problem we discussed.

Visitors often don’t realize that they omit inconsistent (negative) details when they have good rapport with the people the meet. This is human nature, and affects tourists and evaluators alike. I urge you to read “The Surprising Power of Neighborly Advice” from Science Magazine (March 2009), which shows that (a) strangers’ gut feelings are more reliable trust indicators than a set of facts and (b) most people DO NOT BELIEVE THIS even though they act on it.

I cannot underscore strongly enough that getting more people to walk through more  village-level development projects would transform the way that money is spent, for a variety of reasons.

I hope you’ll attend our talk at ISIRC in September, 2009 – Oxford!