Saturday, May 16, 2009

Why Great Nonprofits Isn't So Great

Guidestar recently added a feature to their website which will highlight reviews and stories of nonprofits made on greatnonprofits.org. I've sat back for months now while the funder and nonprofit world got excited about the Great Nonprofits website, but I can't be silent anymore. You may disagree with me, but I really don't think Great Nonprofits is so great, for two reasons.

1. The nonprofits on Great Nonprofits are not representative of the sector
Only nonprofits that have comments or stories posted about them are on the Great Nonprofits site. In other words, there are a vast amount of nonprofits who haven't heard of this site or don't have the capacity to tell their donors or constituents about this site, even though these nonprofits are doing perfectly GREAT work!

2. Comments and stories about a particular nonprofit can skew the public's image of that nonprofit.
Let me explain, ever heard of the website Ratemyprofessors.com? Great Nonprofits is like Rate my professors because when a few people write comments about a nonprofit (or a professor), that is the image that the public sees about that nonprofit (or professor), even if that image is negative or incorrect.

What if someone had a bad experience at a nonprofit and wrote a mean comment, is that representative of all the people who experienced that nonprofit, most likely not. This is so much of an issue that now nonprofits are asking their staff to log on and write nice comments about their organizations so funders and the general public will think they are great!

My advice to nonprofits, funders, and the general public is to invest in researchers to do evaluations that are representative of organizations and the sector. We need more accurate comprehensive views of organizations, not just Harry or Jill's experience with their local soup kitchen.

For example, the special issue Nonprofits and Evaluation: New Directions for Evaluation "explores evaluation through the lens of funders and nonprofits within the field." This issue explains how nonprofits are using evaluation and ways that funders can support evaluation efforts.

12 comments:

Perla Ni said...

Thanks to alerting your readers to the reviews features of GreatNonprofits and now on GuideStar.org. For the first time, the work of a nonprofit - as evaluated by its clients, volunteers, donors, board members or partners - are visible and open for anyone to see.

A quick fact first - except for private foundations, non 501c3s and churches/religious organizations, people can post reviews on nonprofits via GuideStar or GreatNonprofits. That's about 1.1 million nonprofits that are available for review. Some nonprofits have no reviews yet, others already have 70 reviews.

We've heard from many nonprofits that they've used the feedback to make improvements to their programs. We have also heard from nonprofits that this kind of information helps inspire, and inform prospective donors and volunteers. (There are safeguards to minimize the chance of fake reviews.) Nonprofits can respond to reviews easily too by clicking on "comment" next to the review.

Especially in these times, nonprofits want to make their impact visible and concrete in front of an audience of prospective donors and volunteers. This is a free, fast, and simple way for nonprofits - whether you are a large national nonprofit or a small local one - to do that.

Perla

Perla Ni
CEO GreatNonprofits
www.greatnonprofits.org

Sean Stannard-Stockton said...

Have you ever used Yelp? Or read the reviews of a product on Amazon before buying? As a buyer, would you rather that you were not aware of those reviews?

It seems to me that donors are rational people who can use the reviews as one part of their decision making process. If a donor reads one bad review of a nonprofit and decides not to fund them based on only that info, it seems to me that they must not have had much trust in the nonprofit to begin with.

Net-net, I think Great Nonprofits is an outstanding resource providing one type of data that is valuable to donors. In my opinion, the biggest issue with Great Nonprofits is that the reviews are too positive at this point. There are not enough honestly critical reviews.

ecogordo said...

Nonprofits need transparency just like regular companies and government.

With 1 million nonprofits in the world, it is difficult to judge their effectiveness. Are nonprofits being judged properly? How well do they serve peoples needs?

pscrosby said...

GreatNonProfits is a big bold nonprofit experiment...go Perla!

Crowdsourcing community feedback is a good way to gauge online constituent experiences, popularity, and some transparency, but without lots and lots of comments / ratings - like a yelp or citysearch - how can users begin to trust it as more than promotional?

How can people access a 360 view w/out also surfing the data-rich likes of Guidestar or CharityNavigator? And then the 990 financial reporting there is so outta date.

Is there real inside scoop website for npos like good old F*ckedCompany?

Could any online portal ever have it all?

Heather Carpenter said...

My problem is with the people that use charity review sites as a basis for funding nonprofits and do not fund the nonprofits that aren’t on those sites. There are a whole lot of great nonprofits out there that aren’t on Great Nonprofits. I am hoping Great Nonprofits will create a mechanism to reach out to these nonprofits like they did during Hurricane Katrina.

Bob Ottenhoff, GuideStar's president, said...

A few weeks ago we held a seminar with Jacob Harold, program officer at the Hewlett Foundation, on the subject of evaluation. You can hear a copy of the seminar at http://www2.guidestar.org/rxg/news/webinars/webinar-archive.aspx. Jacob reminded us that there are several steps in collecting useful data for measuring performance, a progressive chain of order: inputs, activities, outputs, and ultimately outcomes.

In the nonprofit sector as a whole, we’re familiar with describing our "inputs" (things like grants received, staff hired, volunteer time) and getting pretty good at describing "outputs" (things like number of meals served and people helped). We at GuideStar get most of our data for these two categories from IRS Form 990. But we’ve come to realize that these data points are only part of the picture of a nonprofit – and usually not the most important part. In fact, increasingly donors and reporters are asking us to comment on whether an organization is effective in achieving its mission. That’s why we have launched the GuideStar Exchange program (http://www2.guidestar.org/rxg/update-nonprofit-report/about-the-guidestar-exchange.aspx) and are now collecting nearly 300 fields of data directly from nonprofit organizations. This includes information like the organization’s mission, strategy, goals and measures of success. We also collect financial and organizational metrics.

But our users have helped us to realize there is even more – something that Hewlett calls "stakeholder views" – the opinions of beneficiaries, peers, donors and experts. As one foundation professional put it to me, "I need to learn the facts about that organization, but I also want to know what their community thinks about them and the work they are doing." So GreatNonprofits gives us one more data point to help us evaluate a nonprofit’s work. Just as I would not recommend making a decision based on a few numbers or a single Form 990, I would argue that a GreatNonprofits report in and of itself is inadequate, but we at GuideStar thinks it adds depth and substance to a well-rounded review of a nonprofit’s work.

FsdErin said...

The nonprofit I volunteer with recently mobilized its constituents to write reviews of us on Greatnonprofits for their Green Choice Awards, the effect?

(1) We were able to collect data that from your constituents about their internships with us, how we influenced them, what they liked and disliked, and have that information openly accessible to anyone looking into our organization as ONE means of evaluating us. This information and quotes can also be used in our materials to promote our organization, it saved us labor hours and money.

(2) All of the people who reviewed us were encouraged to spread the word about Great Nonprofits as it not only encouraged them to review us, but showed them a tool they might not have been using.

(3) In a world of complexity any view that only takes into consideration one review, one evaluation (even if it is done by a professional organization), or one use for such a brilliant tool, is too simple to ever capture the whole. This is one more wonderful tool for donors, interns, and volunteers alike.

Congrats Great Nonprofits, I hope you continue to innovate.

Erin Hersey
Foundation for Sustainable Development

wondering said...

I wonder how the overall feedback rating is calculated. I found one NPO that had 1 star and when you read the reviews, one person gave them one star, and one person gave them 5. Are people giving a one star rating being weighted more heavily than those who are happy?

Perla Ni said...

Thanks Erin for sharing your experience! Please let us know if you have suggestions on how we can make it more useful for you.

@Wondering - We use a straight average. You probably were looking at the "Find Reviews" page where the star rating is for the most recent review, and not the average. Just click on "more" next to the blurb review and you'll see the full nonprofit reviews page where it says "Average Rating" up above. We'll improve our user interface on this so that it's not so confusing. Thanks for the feedback!

Perla
CEO GreatNonprofits

Bukie said...

I think GreatNonprofits is actually on to something "Great" here. I recently found them while searching for nonprofit directories and even though they are not simply a directory, I think they offer something valuable to the nonprofit community.
Transparency is the name of the game, and once nonprofits understand this, they will be grateful for the "not so good reviews" if they get any. If you do good work, you should not be worried at all.
I really like the concept and will recommend them to all my nonprofit clients.

Bukie Opanuga
SEONonprofit.com

GayleGifford said...

The shabby and sorry performance demonstrated by Guidestar and GreatNonprofits over Haiti relief that I exposed in my blog post "My worst nightmare is now true: sloppy ratings of nonprofit effectiveness in Haiti" http://bit.ly/6VEae0 shows why Heather is absolutely dead on in her criticism.

Before I spoke up, Guidestar and GreatNonprofits were advising the public on a top ten list of effective relief organizations in Haiti. It was right there on the front page of Guidestar's web site. And the source of that top 10 effectiveness list? The 1 or 2 reviews that GreatNonprofits had received from donors about a handful of organizations. Reviews that in many cases, given these were international organizations, had nothing even to do with their Haitian operations.
Sure, it's nice to know what other donors think of an organization. I give out my recommendations as well. But to pretent that those Amazon style reviews are well informed 3rd party evaluations of NGO effectiveness that have great merit to the public... and to foist them off during a time of such a great disaster, is not only irresponsible, it's downright shameful.
Keep speaking up Heather. I'm right with you.
One has to wonder, who is making dollars from these rating systems?

Johanna said...

Better late than never, eh? (Seeing that this discussion started nine months ago!)

Before jumping in to my comments, I want to first issue a disclaimer: As an evaluator, nonprofit performance is near and dear to my heart. And while I do take issue with GreatNonprofit's approach, I applaud their initiative to push the sector to think and talk about approaches to compare nonprofit performance.

Why I think GreatNonprofits isn't so great:

3) Ill defined rating system:
The data GreatNonprofits uses for comparative purposes is from the five star rating system. To rate an organization, you are asked to award up to five stars for an "overall rating." Since the rating is provided on such a broad item, those that use the aggregate rating of the nonprofit do not know what the rating actually means. It could mean that the organization is perceived as high quality and effective. But we just don't know that for sure. It seems that in the drive to make nonprofits comparable, what we're actually comparing them on has been thoroughly diluted.

4)Satisfaction is used as an inaccurate proxy for quality:
Nonprofits designated as great should be done so based on effectiveness, which in my opinion is the highest measure of quality. Effectiveness goes beyond doing a good job. In the nonprofit sector today, effectiveness means (or should mean) providing the highest quality service possible. Effectiveness means not being satisfied with just doing something, but striving to continually improve. The current GreatNonprofits rating system is more a reflection of customer satisfaction than it is of effectiveness--which, in my book, isn't the advice to be weighed when making a donation decision.

Why GreatNonprofits is great: Like Erin pointed out, GreatNonprofits is providing a valuable service by collecting feedback from a range of individuals who interface with a given nonprofit. I think this is more a service to the nonprofit community, though, than it is to donors.