Survey research can lead fundraisers to trouble

There's some new research out there that's getting a lot of blog-love. It's the Cygnus 2011 North American Donor Survey.

Beware. It's not the fountain of actionable truth folks make it out to be. If you make major decisions based on the Cygnus survey, you risk real damage to your fundraising program.

(Go here to download the executive summary or purchase the full study.)

The main problem is the limitation inherent in almost all survey research: You learn what people say, which is not necessarily what they do. This is true of other surveys too, but this one seems not to recognize this about itself, and makes bold, sweeping statements that simply can't be supported by facts.

Here are two examples, taken from the executive summary:

Finding: 69% now prefer electronic over print communication.

Problem: This is what people say. Real life reports otherwise most of the time. The response rate of direct mail is consistently several times higher than that of email. In fact, direct mail is more effective at motivating online gifts than email is.

If you took this fact to mean you should shift a lot of your budget from print into electronic media, you'd almost surely be making a large and costly mistake.

Finding: 63% said they do not want to receive token gifts (freemiums/premiums) of any kind.

Problem: If that number is meaningful, why is premium-based fundraising increasing? People say I don't want or need premiums. Their behavior says otherwise. If you're using that approach, you should keep doing so until donor behavior shows you otherwise. If you are not using it, you should consider testing it. It might work. Don't let this study lead you astray on this important question.

There are many, many more findings like these in the executive summary alone. It's a cornucopia of deadly non-facts.

Dubious analysis

The Cygnus research compounds the inherent problems of survey research with sweeping statements that don't square with reality. Like this one:

While it is difficult to pin down the specific thing that causes someone to stop giving, the high cost of fundraising is a major consideration for donors and a reason contributing to donor attrition, which now exceeds 90%.

I don't know if this assertion is "supported" by any of the research, but I can say it contains some serious misinformation. If your organization's donor attrition is above 90%, it's time for you to shut the doors and go home, because you are failing. To claim that it's that bad across the board is wildly out of touch. Just as out of touch as saying the cost of fundraising is a major cause of attrition.

I'm sorry to speak so harshly of the work of fellow fundraising professionals, but this study is likely to cause real harm to organizations that take it as bankable truth. Don't let that happen to you.

Tomorrow: How to make survey research work for you.

Comments

[Harvey McKinnon]

This is absolutely true. Real results- how donors behave, is a lot more valuable than what they say. And that's how we should invest scarce fundraising dollars.

[Teri Morrow]

We recently conducted a member survey and found several areas (including preferred method of communication) that seriously overstate what we see in our reporting.

[Peter Li]

I keep telling myself I'm going to stop reading your blog....

[Jim McLachlan]

There were other things that made this research quite questionable:

- It was an online survey which skews the responses tremendously (Belford and Craver are the most egregious violators).
- 26% of respondents were 65+; now tell me how many donor files are made up of as few as 26% 65+ donors. This fact skews most all of the percentages dealing with changes in the medium giving habits, stated online behavior (65% plan to give an online gift in 2011-we wished), retention, giving amount.

And as you have said so many times observed transactional behavior is really the only meaningful data.

Online fundraising is another arrow in the fundraising quiver that needs attention and development, but with most charities receiving less than 10% of their funds from the medium it cannot be the primary focus except for maybe trendy and or advocacy charities.

Penelope Burk has been talking of the demise of direct response mail and phone since 2003. Not sure why she still does not understand looking at the past 8 years.

[Prof Adrian Sargeant]

I couldn't agree more. An ill informed commentary on basic survey stats has the potential to do immense harm to the sector. The comments in respect of premiums and direct mail are particularly egregious. There are two truly sad things here though. 1) since Penelope is invited to speak on many different conference platforms these 'results' will be aired widely around the sector - and 2) folks will pay for the privilege of being misled. A whopping $75 for the full report??? Better to give the money to a charity - even if they do send you a premium ...

[Jeff Schreifels]

Folks, whatever we can do to STOP the Burke train from rolling over uniformed Development Directors we must do it. I could not agree more with Adrian. She is all over our sector preaching this "gospel" and no one is questioning it. I think we need to have Penelope Burke and Jeff Brooks debate at the next DMA and AFP conference. Now that is something I would attend. Who's with me.

[Pamela Grow]

Thanks Jeff (both Brooks and Schreifels), Harvey, Adrian.

The report states that the study "reached such a large number of donors by partnering with 40 nonprofits - 34 of them American." I'd be really curious to know the size of these nonprofit organizations. My own goal is to empower the small community based nonprofit to market like the nationals - and keep their communities alive. Direct mail is one of my methods of choice.

Thanks to Jeff Schreifels for suggesting this as a topic for our next #smNPchat on Twitter next Friday, May 20 at noon ET. Hope to see some of you there.

[Ken Burnett]

All research is flawed, by definition. Research is like a photograph of a dancer. Inevitably it omits much of the flavour, nuance and pace of what it seeks to understand, so it presents an image that is partial and already dated. Does this mean we shouldn’t do research? Of course not. It is though evidently a misuse of research to use it selectively to make a point. Yet that happens every day. And all should be advised, always, to treat research conclusions with the utmost care.

I’ve not studied this report yet, but before rushing too quickly to condemn or praise it, it’s worth recording that Penelope Burke and her colleagues have done much work that is of great value in our sector, particularly in understanding the huge opportunity that arises when we say thank you to a donor, and how bad, as a sector, we usually are at that. I still quote her three advices for securing the vital second gift as some of the best lines in fundraising.

Thanks too Jeff Brooks, to you, for more brave and thoughtful comment and analysis.

Ken

[Chris Keating]

I just had a look at the Executive Summary and find it difficult to take seriously statements like "79%
expect to give the same or more to charitable causes this year, while only 7% will give less."

I am sure they all intend to help little old ladies across the road, and that very few said they intend to go and kick kittens for a laugh.

However, this is another great example of a meaningless question that simply invites the respondent to say they are a nice person.

Perhaps someone with the full report can tell us whether there is any actually relevant analysis of the context of this question (has the response rate changed from last year?) o whether we should all just save our money.