Does philanthropy work?

Can charities accurately measure their effect on society?

On October 19, 1987, the Dow Jones Industrial Average plunged 22.6 percent, a single-day loss unequaled before or since on Wall Street. To most observers, the Black Monday crash came without warning and remained deeply mysterious ever after. The stock market, many concluded, was like the weather: from time to time, an unpredictable convergence of phenomena would have catastrophic effects.

Yet at least one man believed that the crash was coming. In Greenwich, Connecticut, money manager Paul Tudor Jones had been studying the accelerated methods of buying and selling stocks that computers had made possible. The speed of transactions, he felt, had inflated prices, and a major correction was inevitable. Through his hedge fund, Tudor Investment, Jones bet everything on a downward turn. Black Monday tripled his stake, making him one of the country’s richest financiers. In the world of stock-market speculation, a single big win is enough to attract a clientele, whose holdings one can then manage without taking excessive risks, and that’s just what Jones did.

But he didn’t stop there. Jones was born in Tennessee into a religious family; as a young man, instilled with the evangelical values of the South and outraged over the poverty of blacks and working-class whites, he and his congregation had joined civil rights protests and handed out clothing and meals to Tennessee’s poor. Perhaps that background explains why he saw Black Monday as the opening of a new era in which the divide between America’s rich and poor would worsen, threatening the social order. At least part of his prediction proved accurate: while the poor in the United States haven’t gotten poorer since 1987, the distance between their economic position and that of the wealthy has widened.

Jones believed that private philanthropy could help the poor improve their lot and reduce inequality—but only if that philanthropy was effective. Such large foundations as Ford and Rockefeller were fossilized bureaucracies, better at paying their own administrators than at lifting up the worse-off, Jones thought. “Charitable institutions are often content when they build a big budget and spend as much as possible,” he explains 25 years later. “Spending is their criterion of success, which is opposite to how a business operates.” He decided to try to revolutionize social philanthropy by infusing it with methods from the world of finance, seeking to measure the result—the “output” produced—rather than simply track the money spent, the “input.”

Can philanthropy be calculated that way? In the private economy, of course, everything gets measured in dollars. But how can one measure the benefit of a drug addict detoxed or a child properly educated? Economists have long claimed to be able to quantify all human actions—to measure the value of a life, for example, or the added benefit of an additional year of school. Yet no philanthropic institution had ever functioned on such a rigorously quantitative basis. Why did Jones believe that it was necessary? “The resources directed to philanthropy are by definition finite, like all resources,” he argues. “It is legitimate and respectful to donors as well as to recipients to see that these rare resources are directed as a priority to projects that have the greatest measurable results.” Not measuring results could lead to wasting donations.

Moved by the gospel of measurement and seeing no philanthropic organization honoring it fully, Jones created the Robin Hood Foundation in 1987, locating its offices in Harlem. To differentiate their outfit from the thousands of other groups fighting poverty, Robin Hood’s founders—Jones, three fellow financial managers, and a New York–based social-services administrator, all in their thirties—chose not to name it after themselves. (Jones finds it hard to understand the vanity that leads some donors to inscribe their names on museums.) Nor did he wish to create an endowed foundation that would simply spend a fraction of its capital annually—for instance, the minimum 5 percent or so mandated by the IRS. Rather, he wanted to raise a lot of money and distribute it immediately. The foundation has done exactly that since its inception, collecting roughly $150 million per year, mostly from Wall Street, and spending it in New York City—all $150 million of it, every year. Dedicating all funds raised to beneficiaries is rare in the philanthropic world, where staff salaries and overhead frequently take a big cut. Robin Hood can put every cent it raises into its programs because its board of directors pays those administrative costs.

The costs include 25 researchers who work to make Robin Hood’s donations “the most effective in the world,” says David Saltzman, the foundation’s director. They measure effectiveness with a system called RM, for “Relentless Monetization,” which was developed by Michael Weinstein, an economist at the foundation and the coauthor of The Robin Hood Rules for Smart Giving. Most philanthropists let “instinct, passion, and illusion” guide them, says Weinstein; philanthropy needs to become more quantitative. “All philanthropic activity can, in principle, be monetized,” he explains, “because it involves spending by the donor and a measurable benefit for the recipient”—access to a school, or better hygiene, or access to new services. To proceed without quantifiable information, Weinstein contends, might give a donor or an administrator a clear conscience or improve his or her reputation, but it’s ultimately irresponsible, since charitable dollars might not benefit the neediest recipients and might even do damage.

Here’s an example of Robin Hood’s quantitative approach in action. The foundation sets a goal: improving the health of children living in a certain poor New York neighborhood. The most productive intervention, Robin Hood discovers, is to increase the proportion of neighborhood children graduating from high school, which gets a bigger bang for the buck even than getting them to the doctor more regularly. A typical graduate lives 1.8 years longer in good health than a typical nongraduate, research from Columbia University’s Peter Muennig shows. That extra time is worth about $50,000 annually, according to insurance data, so a high school graduation is worth roughly $90,000—that is, $50,000 times 1.8 years. Further, studies show that a high school graduate earns at least $6,500 more per year over the course of his career than a nongraduate does—equivalent to $120,000 if it were spent in advance. So every philanthropic investment that results in the graduation of a student who otherwise wouldn’t have graduated, RM calculates, produces a monetized social benefit of $210,000. Such an investment is therefore a financially positive one if it amounts to less than $210,000. Above that figure, the return on investment is negative.

The difficulty in making this calculation stems from the fact that it adds income, which is easy to measure, to the harder-to-quantify benefits of life and health. But the insurance industry makes this kind of assessment all the time. And Robin Hood’s goal isn’t to arrive at incontestable numbers but rather to present donors with figures that can help guide giving decisions.

The foundation’s approach makes it easier to assess the effectiveness of institutions. If a school receives a grant that helps, say, ten disadvantaged students graduate who’d otherwise drop out, the total social benefit adds up to $2.1 million. One can therefore understand the reasoning behind Robin Hood’s grant to the acclaimed Promise Academy Charter Schools, part of Geoffrey Canada’s Harlem Children’s Zone, a nonprofit that provides extensive services to the poor. Promise Academy graduates all its disadvantaged minority students. Can we be certain that, without this grant, the school wouldn’t have achieved similar results? A major difficulty in determining philanthropic effectiveness lies in comparing action with inaction. In the case of the Promise Academy schools, a comparison, however imperfect, is possible with other schools in the area with students from similar backgrounds; the Promise Academy’s results are indisputably superior.

Robin Hood’s number-based standards often result in rethinking programs. For example, in 2006, it established “Single Stop” offices within the Rikers Island jail complex to educate prisoners about social services available to them at the end of their sentences. The foundation figured that if the newly released could more readily access the services, which included drug-treatment and job-training programs, their transition into society would go more smoothly. The success of the effort, at least on its own terms, appeared incontestable at first. After all, one-third of those entering Rikers had received some form of aid before their first incarceration; after the Robin Hood initiative was up and running, two-thirds of those released accessed services. The foundation attributed this increase to its own work.

But then prison renovations cut in half the space available for Robin Hood’s prison operation, and halved, too, the number of prisoners who could participate in it, whom the foundation now selected by lottery. Yet despite the shrinking of the program, the overall proportion of exiting prisoners accessing social services was the same: two-thirds. Many prisoners had clearly educated themselves about the help that they could get upon release, but the Single Stop office had made no difference. The prisoners who had participated in Robin Hood’s initiative, it turned out, had already known that they could get help outside; those ignorant of the services had never bothered to visit the office. Robin Hood, working with Rikers officials, has now changed the program to focus on that uninformed one-third, many of whom are particularly disorganized young women. “Never fall in love with your philanthropic work, take careful note of numbers, and check for biases that exaggerate the impact of a grant,” says Weinstein, ticking off the foundation’s methods. “Then check again.”

Robin Hood’s approach encourages humility, as another example shows. A correct tax filing can provide rebates and other subsidies to those of modest income but only if they can find a way through the labyrinth of the U.S. tax code. A Robin Hood program accordingly offers free tax-return services to the New York needy—about 50,000 of them yearly, who wind up collecting a total of about $120 million annually from the adjusted returns. Though this looked like a smart philanthropic investment at first, Robin Hood compared those served by its tax program with other groups and discovered that the results weren’t that impressive. Two-thirds of the amount recovered by the adjusted tax returns would have been received anyway, the analysis showed, because most of the Robin Hood clients would have sought help from a relative or used a free Web service to do their taxes. The foundation’s return on investment had thus fallen two-thirds. Nevertheless, Robin Hood continues to support the initiative.

Robin Hood’s evaluation system has its critics. Some are doubtless irritated by its implication that many humanitarian projects are undertaken blindly, based on poor information, and driven more by fashion than by a real need to improve the condition of the poor. Relentless Monetization challenges all that is haphazard, narcissistic, superficial, and even counterproductive in social philanthropy.

But others, such as billionaire philanthropist George Soros, mount more substantive objections. For one thing, he says, even careful measurement can’t protect philanthropy from unintended consequences. Soros cites the Bill and Melinda Gates Foundation’s African vaccination program, which had all sorts of metrics in place, as Laurie Garrett reported several years ago in Foreign Affairs—but nevertheless damaged the existing health infrastructure of African villages by drawing health workers to its own, better-paid initiative.

The director of Soros’s Open Society Foundations, Chris Stone, who long taught nonprofit management at Harvard University, adds that quantification is difficult when the philanthropist lacks solid information about the value of a particular approach: “When we fight poverty, discrimination, addiction, or oppression, we do not know what works and what doesn’t.” The only way to learn is to experiment with innovative approaches, in full awareness of the possibility of failure—which is more likely than success. “How is it possible to quantify what you don’t yet know?” he asks. When Soros took the initiative in the 1980s to back independent journalists in Communist Eastern Europe, his intuition that doing so could have long-term positive effects was impossible to quantify. Yet those journalists eventually played a role in the democratization of their countries and the creation of a free press, Soros points out. Of course, some counting was necessary: he had to decide which journalists would get money, and in which countries. “Yet I did not quantify in a narrow sense,” he says. “I experimented and refined my strategic choices based on my failures and successes.” Soros adds that philanthropists should assess and revise their goals continually: “Freedom of speech, the fruit of hard struggle in the 1990s, has more recently become a tool in the hands of demagogues and enemies of liberal democracy in Hungary and Russia.” The philanthropist who wants to encourage an open society in Eastern Europe must now foster more than free speech alone.

Both Soros and Stone further reproach the quantitative method for ignoring the human dimension of philanthropy. Those who become philanthropists after making fortunes in the highly monetized world of business tend to be attracted by the very different values that should guide the nonprofit sphere, with numbers giving way to love for one’s neighbor and spiritual concerns. To reduce everything to counting, they charge, is to contravene the act of giving. And quantification also fails to take into account the psychological benefit that philanthropic work can bring to its beneficiaries. Soros money underwrites evening classes for schoolchildren in rougher sections of Baltimore and New York with no expectation of improving their academic achievement; the end is simply to bring parents an incalculable sense of security.

The Robin Hood Foundation also supports evening classes in New York but tries to measure their scholastic impact. “We run our shops differently, though we respect one another’s efforts,” Weinstein observes of Soros (who is himself a donor to Robin Hood). In fact, the Open Society Foundations will join Robin Hood, the New York City education department, and the city council in a partnership to advance literacy in the city’s middle schools.

Confronted with Soros’s and Stone’s critique, Weinstein sees less an attack on quantification than an invitation to improve it. Relentless Monetization is an imperfect instrument, he admits, “but who today has anything else to propose in order to ensure the optimal allocation of the scarce resources of philanthropy?” Further, Robin Hood is aware of the need to experiment, but it insists on collecting data on unproved programs, so as to provide a basis for evaluation when particular initiatives come up for grant renewals. Let me confess a personal interest in this question. When I was directing a French humanitarian organization, Action contre la faim (Action Against Hunger), I would have appreciated the help of Weinstein’s methods. The RM system provides a strong incentive to limit waste and can be a check on philanthropic giving that does little actual good or, worse, commits harm. “Interventions should not be funded because they look good, or even because they are good, but because they are the best available options,” Weinstein says.

Still, one can’t easily dismiss Soros’s and Stone’s arguments. It’s even possible that, as Soros suggests, “the quantification of goals discourages philanthropists from fighting certain inequities because they are hard to measure.” If charities do start shifting their priorities toward programs that lend themselves to measurement, something important could get lost in the process.

Guy Sorman, a City Journal contributing editor and French public intellectual, is the author of many books, including Economics Does Not Lie. His article was translated by Alexis Cornel.