Let’s Be Realistic About Measuring Impact

March 15, 2013 Editor 0

“Measure impact” has become a mantra for creating social change. Claims about making a difference are no longer sufficient; evidence of how much difference you’re making is now required. We should applaud this trend, because results are sometimes ambiguous and claims often go unsubstantiated. But does it really make sense for all mission-driven organizations to measure their long-term impact on society?

Rather than engage in a debate about what one should measure, let’s take a look at what thoughtful organizations actually are measuring. I studied three organizations at the vanguard of performance measurement: Acumen Fund, Robin Hood Foundation, and Millennium Challenge Corporation (MCC). All three provide capital to organizations on the front lines of social change, either as investors or grant makers.

What do these organizations seek to measure? It turns out that they assess different things: Acumen measures immediate outputs, such as mosquito nets made and distributed, Robin Hood focuses on long-term outcomes in the lives of individuals such as gains in future income, and MCC aims for both individual outcomes and broader impacts on society such as reduction in poverty rates.

Let’s start with Acumen Fund, a venture philanthropy firm with a portfolio of over 75 investments in social enterprises in Africa and Asia. Its primary social metric is the number of lives reached in base-of-pyramid markets. This means that for a company that manufactures anti-malarial bed nets, Acumen will count the number of nets manufactured and distributed. For an enterprise that builds toilet and shower facilities in urban slums and business districts, it will tally up the number of times the toilets and showers are used.

But why stop at these output measures? Why not go all the way and measure outcomes like reduction in malaria or improvements in health and environment? Brian Trelstad, Acumen’s former CIO, is blunt about why not: it’s complicated, expensive, and often impractical for early-stage enterprises. Instead, Acumen’s strategy is to review the literature and consult experts to establish a link between a specific output and impact (for instance, how bed net distribution leads to malaria reduction) and to then count the outputs.

Robin Hood, on the other hand, is a grant-making foundation created by hedge fund managers with a penchant for hard numbers. With a mission to fight poverty in New York City, the foundation puts each of its 200-plus grants through a cost-benefit analysis every year. One of its primary metrics is the expected increase in lifetime earnings of its clients. For grants focused on education, for example, its staff first identifies a set of results that can be immediately observed — such as school attendance, standardized test scores, and high school graduation. Then, they search for studies that link those measures to expected lifetime earnings or quality of life. For instance, some longitudinal research suggests that a 10% increase in test scores is correlated with a 4% increase in high school graduation rates, which in turn is associated with $6,500 in increased income per year. Robin Hood uses these figures with caution, employing them as placeholders for estimating benefits until better research comes along.

The MCC’s approach is even more complex as it operates on a 20-year time horizon. The U.S. government agency makes grants to emerging market countries to reduce poverty through economic growth. Its extensive due diligence process — often two years long — first analyzes the barriers to economic growth in the country, and then identifies the sectors where the grants would most likely reduce poverty. For example, the MCC awarded $547 million to the Government of Ghana to build roads and ferries to get farm goods to market. To start, it estimated the number of farmers likely to benefit, and what those benefits would be: reduced cost and time of getting goods to market, access to new markets, and opportunities for wage employment. These data were used to anticipate an economic rate of return, with the primary outcome metric being increases in farmer incomes, along with impact metrics such as a reduction in regional poverty rates.

Once a contract is signed, a monitoring and evaluation process kicks in to allow for mid-course correction. Benefits are expected to begin accruing only after five years, once the infrastructure is built and operating. The MCC plans to eventually conduct long-term evaluations, but has not yet done so.

Notice that none of these three organizations typically measures impact directly. They hypothesize what the outcomes and impacts might be but only in some instances are they able to follow through by commissioning their own research or multi-year evaluations. And these are sophisticated funders and investors who are much better positioned to measure long-term results than the front line organizations that contend with funding shortages and operational challenges every day.

So what does their experience mean for other organizations trying to prove their mettle through measurement? Surely measuring impact matters but we need to be realistic about the constraints. It requires a level of research expertise, commitment to longitudinal study, and allocation of resources that are typically beyond the capabilities of implementing organizations. It is crucial to identify when it makes sense to measure impacts and when it might be best to stick with outputs — especially when an organization’s control over results is limited, and causality remains poorly understood.

Overcoming these obstacles will require investors and front line organizations to make a long-term commitment to research and collaboration. Simply repeating the mantra of measuring impact won’t get us there.

Go to Source

Categories: HBR, Insights

Tags: MCC, Measure impact, Organization, Robin Hood, social