Nick Hamlin Posts

Interoperability Judo in the Aid Sector

judo_2_large

Photo Courtesy of Yayasan Damai Olahraga Bali

Ten years ago I nearly set an Italian hotel on fire. I’d plugged an American fan into a European electrical socket, and after about 30 seconds I had a shower of sparks landing on the curtains. What I’d forgotten to account for, of course, was the difference between the 120 volt standard that the fan was expecting and the 240 volts that the outlet was producing. Just because the connection worked physically didn’t mean it would work practically. Contrast this with last month, when I’d brought my laptop and charger on a trip to India but again forgot a power converter. Thankfully, Apple has an elegant solution to the problem of different electrical standards. Rather than trying to convince every country to use the same voltage in its wall sockets, they’ve just built a charger that can handle a range of inputs. They accept the complexity that happens when large groups of people try to collaborate and work with it, not against it. They’ve taken an obstacle and turned it into an opportunity. It’s design judo.

When it comes to financial flows in the aid sector, standards are more complicated than deciding what plug to use. With so many governments, organizations, and companies sending billions of dollars to support global development, communicating the details of these relationships and transactions in a shared framework becomes a herculean task. The International Aid Transparency Initiative (IATI) has made progress in establishing a standard for the sector to describe funding and implementing relationships consistently. The list of 480+ entities that publish IATI-compliant data understates the standard’s reach. Most of the 29 member countries of the Development Assistance Committee (DAC) report IATI data about their aid spending, and the funds sent by these governments represent about 95% of total DAC expenditures. It’s hard to estimate an exact number, but it’s safe to say that the IATI standard describes a significant majority of the world’s aid dollars.

Still, there are some challenges to using IATI-compliant data to get a precise understanding of how the aid sector is actually organized. Despite IATI’s thoroughness, organizations still interpret the requirements differently, leading to the same data fields containing multiple types of information. This can make seemingly simple tasks, like identifying a unique organization consistently, very difficult in practice. Similarly, there aren’t strict validations or requirements preventing organizations from omitting data or inadvertently hiding important outcome data in a pages-long list of transactions. Organizations that don’t share their data are left out entirely, even if they’re mentioned frequently by organizations that do report. All this can make it hard for aid professionals like funders, program implementers, or researchers to extract useful conclusions from IATI data.

So what should the sector do about this? One approach might be to double-down on the rules associated with our data standards and try to force everyone to provide clear, accessible, and organized data. This would be similar to convincing all countries to share the same voltage standard; it’s not a practical option. The alternative is the judo method: work with the challenges inherent in the IATI standard instead of trying to regulate them away. Some friends and I recently tried to do just that as our capstone project for the UC Berkeley Masters of Information and Data Science degree.

The end result is AidSight, a platform that provides easy-to-use tools for the aid sector to search IATI data, explore relationships between organizations (including those that don’t report their data directly), and validate the likely usefulness of their results. For example, imagine you’re an aid agency that needs to report on the current state of the water sector in Ghana. First, AidSight enables you to query all IATI data in plain english instead of a complex requiring search interface or a code-heavy API call. Your results appear as network diagram that maps the relationships between the organizations that meet your search criteria, whether they report to the IATI standard or not. Here’s our result for the Ghanian water sector – note that we’re mapping the just the organizations and relationships, not their real-world locations or relative sizes:

screen-shot-2016-11-08-at-3-34-38-pm

The green dots represent organizations that report data to IATI directly, the red dots are organizations that are implied in the data that other organizations report, and the width of the lines connecting them indicates the strength of the relationship. This approach takes the data reported by 484 organizations and turns it into results for tens of thousands. In this example, there are two “hubs” of reporting organizations on the right side of the map that work with 5-7 non-reporting organizations at varying levels of connection. In contrast, there’s another hub organization (GlobalGiving itself) towards the bottom left that works with many more organizations, but in the same way with all of them. Using this method, users are quickly able to spot the key players in any sector and explore the strength of their collaborations instantly.

Understanding these connections is important, but what if the report needs more granular results? Before downloading and analyzing the raw data, you’d want to know if you’re likely to be able to draw meaningful conclusions from the results we’ve found. To make this easy, AidSight contains a data quality dashboard that uses heuristics to estimate how useful each organization’s data is likely to be and summarizes it with a simple letter grade.

Example AidSight Data Quality Dashboard

Now, anyone at an aid agency can measure IATI data quality with a few clicks and save their data science teams to focus on only the most useful datasets. We can also use this approach to establish valuable benchmarks for the aid sector as a whole. The average grade of C- suggests that there’s lots to be done to improve the quality of development data reporting, but having a framework to measure progress makes it possible to consider how we might get there.

Currently, AidSight is a minimum viable product, so there are many improvements to make. Still, solutions that focus on data interoperability without trying to fight the natural complexity of the aid sector represent exciting opportunities for us to bring enhanced accessibility and understanding to our work in a democratic way. Taking the judo approach to development data means that a growing number of inventive, creative, and driven users will be able to discover new solutions to the aid world’s challenges.


Special thanks to the other members of the AidSight team: Natarajan Krishnaswami, Minhchau Dang, and Glenn Dunmire, as well as Marc Maxmeister for his feedback on this work. Explore IATI data yourself at aidsight.org or download the open source code on Github.

Is Overhead All In Your Head? How Cognitive Psychology (and Font Colors) Can Drive Donations

Nick Hamlin, GlobalGiving’s Senior Business Intelligence Analyst, shares results of a recent experiment on the GlobalGIving website. (Photo courtesy of The Muse)

Nick Hamlin, GlobalGiving’s Senior Business Intelligence Analyst, shares results of a recent experiment on the GlobalGIving website. (Photo courtesy of The Muse)

No one likes worrying about the overhead costs associated with nonprofit work, and rightly so!  For years, overhead ratio has been of the only metrics that donors could use to compare philanthropic choices.  More recently, conversations like The Overhead Myth have pointed out that the world’s best businesses need operating capital to innovate and succeed, so why should nonprofits be any different?  Even though better measures of impact and effectiveness are increasingly available and accepted, a typical donor’s natural reaction when they see a percentage come up in a conversation about nonprofit fees is to interpret it as an overhead ratio. And most donors still don’t like overhead.

For us at GlobalGiving, this presents a challenge.  While we retain a 15% fee on donations through our website, our actual administrative overhead ratio is around 2%. Despite testing several different ways of demonstrating and explaining the difference between our fee and and our overhead, we still get lots of questions about our fees from users who assume that the two are the same. To help fix this, we recently asked ourselves: what if it’s not the explanation text that’s the problem, but how users are experiencing and processing the information it contains?

For inspiration, we turned to the world of cognitive psychology.  In his famous Thinking Fast and Slow, Nobel laureate Daniel Kahneman describes how we all have two systems at work in our brains.  System 1 is our intuitive, quick-reacting, subconscious mind, while System 2 is analytical, logical, and methodical.  He mentions a 2007 study that tried to use the interaction between these two systems to improve scores on the “cognitive reflection test”.  This short quiz consists of questions that seem simple at first, but have a “wrinkle” that makes them more complex than they appear (try them yourself). Half the participants in the study took the test normally, while the other half took the test under a cognitive load, meaning the questions they received were written in a lighter font that made them slightly harder to read. The researchers found that the second group performed much better on the test, presumably because the cognitive load caused their analytical System 2 processes to take over from their more reactionary System 1 minds.  Once in this “more logical” frame of mind, they were much better equipped to tackle the tricky problems.

After reading about this study, I wondered if we could replicate the results on GlobalGiving to help donors process the explanation of our fee and the accompanying invitation to ‘add-on’ to their donation to cover this fee on behalf of their chosen nonprofit. Our hypothesis was that donors usually use System 1 when thinking about our add-on ask; they quickly assume that the 15% represents overhead and they’re less inclined to donate additional funds to cover it. But, if they engage their System 2 mindset that makes them process the text more analytically, hopefully they’ll find the explanation more convincing and be more likely to add-on. To find out if this would work, we planned a simple test in which a subset of users would be randomly chosen to see a slightly modified version of the add-on page during their checkout process.  This page would have exactly the same text, just shown in a slightly lighter font that, we’d hope, would trigger the cognitive load and drive extra add-on contributions.

Users in the control group saw this unmodified add-on prompt.

Users in the control group saw this unmodified add-on prompt.

The test group received this add-on prompt with a decreased font contrast to create cognitive load.

The test group received the second add-on prompt with a decreased font contrast to create cognitive load.

The plan made sense in theory, but we had to be careful as we put it into practice.  First, we needed to make sure that the random assignment process, made possible by our Optimizely A/B testing framework, was running correctly and that all the data we would need to analyze the results was logged properly in our database.  Even more importantly, we have an obligation to our nonprofit partners to make sure we’re doing everything possible to maximize the funds they can raise by offering a seamless website experience for donors.  If this experiment caused users in the treatment group to become less likely to complete their donation, we’d need to know right away so we could stop the test.

We set up a pilot study where we closely monitored whether the cognitive load caused by the change in font color would cause potential donors to leave the checkout process prematurely.  We also kept a close eye on post-donation survey feedback to see if anyone mentioned the changed font color.  Fortunately, there was no difference in donation rates or feedback during this initial test, and we felt comfortable continuing with the larger experiment, which ran two weeks at the end of July (just before the launch of our new website).  In the end, we collected results from about 700 eligible users.

So what did we find? 49.4% of our control group chose to contribute towards the fee, compared to 56.8% of users who saw the lighter font.  This sounds like there’s reason to believe users were engaging their System 2 brains and processing the request for an additional donation. But, it would be premature to declare success without additional analysis.  Specifically, we wanted to make sure there wasn’t another explanation for the difference in add-on rates.

For example, it’s possible that users who were new to GlobalGiving would be less familiar with our fee and therefore less likely to want to add-on to their donation to offset it.  Similarly, donors contributing during a matching campaign might be especially inclined to make sure that the most money possible went to their favorite organization and, as a result, would add-on more often.  So, in our analysis, we statistically controlled for these factors, along with the size and geographic origin of each donation, to get our most pure estimate of the effect of the cognitive load.

The final result was a 7.8 percentage point increase in add-on rates with a P-value of 0.046. This means that we have only a 4.6% chance of seeing results at least as large as these purely by chance.  If we take this increase and estimate what might happen if we made the change on the whole site, we expect we’d see around another $27,000 in additional funding created for our project partners over the course of a year.  That may not sound like much in the context of the $35M+ that will be donated through the site in 2015, but it’s not a bad return for our partners for just changing a font color!

These are exciting results that suggest the possibility of a new way of thinking about how we present our fee, and there’s still plenty of work to be done.  Longer runtimes and larger sample sizes would give us even more confidence in our results and let us explore other potentially important factors, like seasonal effects.  Thinking about how we might integrate these results into our new website also presents opportunities for follow-up experimentation as we continue to Listen, Act, Learn, and Repeat on behalf of our nonprofit partners around the world.

 

Special thanks to my classmates Vincent Chio and Wei Shi in the UC Berkeley Masters of Information and Data Science program for their help with this analysis and to Kevin Conroy for his support throughout the project.