Wednesday, March 24, 2010

How to be a Critical Consumer of Nonprofit Research

In today's society we want information fast and to the point causing us to make assumptions and generalizations about the sector or society as a whole. Too often this sound bite mentality/culture prevents us learning the real truth behind what we read. Here are examples of three news headlines:
Each of these headlines reported on a different study and made claims about the entire nonprofit sector. It is great that many organizations are producing research about the sector, however, in many cases this research is reported in an inaccurate manner.

One of the first things I learned in my PhD program was how to be a critical consumer of research (thank you Dr. Galloway!). Here are the questions I ask myself when I read any article or research about the nonprofit sector:

1. Who were sampled? (Nonprofits, Executive Directors, Foundations?)
2. Is the sample representative of the nonprofit sector-by organizational type (NTEE Code), budget size, location?
3. What were the sampling procedures?
4. Was the information conveniently gathered or did the researcher attempt to collect data from as many organizations as possible across the sector?

There are a variety of reputable sources that researchers can use to compare their sample to the national nonprofit sector. The National Center for Charitable Statistics produces an nonprofit almanac where the majority of this information is location. Here are some other sources as well.
Now, let's try to be critical consumers of the three studies I listed above:

1. "A Growing Leadership Gap: Need for qualified executives persists even as charities trim staff" Comes from the study: Finding Leaders for America's Nonprofits by the Bridgespan Group

The survey sample included 433 Executive Directors. The Bridgespan Group purposefully sampled organizations with revenues over 1million dollars. I am unsure why they did this since their sample potentially represents only 17.7% of organizations in the sector, therefore the study is not representative of the entire sector. Although the researchers reported NTEE code and region for the sample, it is unclear if this information is representative of the 17.7% of organizations in the sector since no comparisons were made.

2. "Nonprofit Groups Laid Off Employees, Shifted Investments, and Added Governance Steps in 2009" Comes from the study 2009 National Board Governance Survey for Not-for-Profit Organizations by Grant Thorton.

The survey sample included 465 CEO's, CFOs, board members or other executive level staff. Grant Thorton reported that 71% of respondent organizations had revenues of less than 50million. Really? I am unsure why they did not break down the budget sizes into smaller categories. This shows me the sample is heavily skewed towards larger organizations, and like the study above, is not representative of small organizations or the sector as a whole. Additionally, the study does not provide any statistical information about NTEE code or location of respondent organizations.

3. "For Executives, Study Shows Foundations Prefer to Hire from the Outside" Comes from the Study: Benchmarking Diversity: A First Look at New York City Foundations and Nonprofits by the Foundation Center & Career Pathways to Philanthropic Leadership 2009 Baseline Report by the Council on Foundations

I am going to focus on the Career Pathways report since it is a national study. This study includes a combination of interviews with 50 foundation executive, trustees, nonprofit executives, and people connected to the foundation subsector and an analysis of foundation executive appointments from 2004-2008 (400 appointees). I think this report does a nice job of reporting its sampling procedures and limitations of the research. They are clear that the study does not represent the 70k foundations in the USA and the study is skewed towards larger foundations. However, the hiring information is representative of foundations by region.

I recognize that it is difficult to conduct a research study that is representative of the entire sector and I am not necessarily advocating for that. I am advocating for researchers to clearly state their sampling procedures, limitations of the study, and try to see if their data is representative of the sector (and if not -- state that is not). Additionally, since most news headlines generalize research to the entire sector, I want to encourage people to read cautiously and be critical consumers of the research. I would also encourage news sources to state the limitations of the study in the news articles as well.


Anonymous said...

Heather, thanks so much for this great article. It was a great reminder to read research with a critical eye. -ec

James D'Ambrosio said...

Great post on an important subject. Painting with a broad brush often leads to spurious generalizations. I learned the importance of researching a representative sample of the population and looking for statistical significance in graduate school.