Testing Tuesday: A weekly series on how to get the most of direct-response fundraising tests
We were looking at some very odd direct mail test results. They seemed to show a result exactly at odds with what experienced people know to be true.
The test showed that an "organizational" approach in the message convincingly out-performed a "personal" approach.
You're probably thinking exactly what I was: What the heck?
I assumed (as did everyone else involved) that "organizational" meant it emphasized the organization's role in the topic at hand, and "personal" meant it emphasized the donor's role.
So for the past two years, the org had been doing "organizational" fundraising. They carefully described how their programs worked. They featured staff members as the amazing people they were. The message was "Donate now so we can do the things we do so well."
It wasn't going well.
The retention rate was low and dropping. New donor campaigns were not sustainable.
They were in a revenue death-spiral.
That's when I asked to look at that old "organization vs. personal" test. Not just the results, but the actual packs.
Here's what I discovered: The two versions were quite similar, best-practice direct mail packs.
- The "organizational" version had a very specific, simple clear offer. It asked the donor to give to make something specific and important happen.
- The "personal" version had a chatty friendly tone, a lot like a letter to a friend -- but no specific offer.
The labels "organizational" and "personal" weren't exactly wrong, but they sure were ambiguous. They had no doubt come from a less-experienced member of the team, and were never checked clarity, and then became received truth.
Those labels were misinterpreted for two years and led the organization to do ineffective fundraising for two years. It cost hundreds of thousands in lost revenue.
We quickly changed the approach. Fundraising results turned around immediately. But the lost donors meant less revenue for the next several years.
Lesson: When you test, have a clear hypothesis. Make sure there is shared clarity of what you're testing and what you call the things you are testing.
Testing is science. You develop a hypothesis (like "a chatty tone might improve results, even without an offer") experiment, and learn. Part of that hypothesis is making sure you label your test in a way that you'll learn what you really learned.
What they learned in this case -- but didn't know it -- was that an offer was more important than a friendly writing tone.
We soon after tested a campaign that had a friendly tone and an offer -- and I bet you can guess what we learned:
Yep. A friendly tone with an offer worked better than either one of those things alone.
Testing is about learning and improving. It takes a methodical, disciplined approach. And clarity about what you're testing. Without that, you are far better off not testing.
