Recently, a client sent us results of a focus group they had conducted, results that included an evaluation of direct mail concepts we’re developing for one of their products. As part of the research, the client interviewed a number of small business decision-makers and asked for their opinions of the various direct mail formats and whether they’d be inclined to read and respond to them.
This reminded me of an article (below) on direct mail and focus groups that I wrote more than 10 years ago. What I said then is true today:
Focus groups are notoriously unreliable when it comes to evaluating direct mail.
That’s not to say that you can’t learn things from focus groups, but (for example) one of the recommendations from this client’s research was to “include clear benefit language on the outer envelope” because that’s what people in the focus group responded to best.
In the real world of B2B marketing, however, envelope teaser copy can easily decrease response. Why? Because people often equate teaser copy with junk mail, and so they never open the envelope in the first place (or their secretary never delivers it to them.) In the focus group, people were presented with the package and told to read it, so that human dynamic wasn’t even present.
Remember: a key factor in direct mail effectiveness is deliverability, not just what gets read once the mailer is in front of the intended recipient.
Why Focus Groups Don’t Work for Direct Marketing
An article that appeared recently in a national direct marketing publication recounted the case study of a database solutions company that used focus groups to evaluate two different creative concepts for an upcoming lead generation mailer.
One concept took a humorous approach; the other was more straightforward and to-the-point. Every one of six different focus groups said that the humorous piece was the best. Wisely, the company decided to test both versions in the mail anyway. The results: The more straightforward mailer out-pulled the humorous version by more than two to one.
The story reminded me of a similar experience we had with a client a few years ago for a campaign that targeted software developers. The client wanted brash, colorful creative to “break through the clutter.” We convinced them to let us develop both a colorful self-mailer and a more sedate letter package in a plain (no teaser copy) #10 envelope.
At final layout, the client took the comps round to all the developers in their organization and asked them which they’d be most likely to respond to. To a man (this was an all-male sample), they chose the self-mailer. In the mail, the letter package out-pulled the self-mailer by a factor of three to one.
Focus groups have tremendous value for many things — notably to help evaluate brand advertising and basic positioning. But for direct mail, they’re notoriously unreliable. The only way to get a true, objective evaluation of a given program is to test it in the field. After all, one of the things that makes direct marketing so valuable — whether print or online — is that it’s measurable.
If you simply must get some indication of a direct mail campaign’s potential merits before committing to even a limited test, then consider using online media — email, PPC, or banner ads, say — to test message, audience, offer or creative. Then use those results to alter your direct mail design if necessary.
Article excerpted from the ebook: “The High Tech Direct Marketing Handbook.” To download a free copy, click here.
Great points, Howard. Focus groups are great for a lot of things, but they’re hardly reliable as harbingers of in-market direct response messages. Even “real time” focus groups via Twitter Search and LinkedIn Answers aren’t going to give you mathematically-relevant data.
Focus groups never give mathematically relevant data. Sadly, they’ve been horribly abused by clients (testing things which aren’t testable – like predicting specific effectiveness of direct mail or traditional advertising) and by agencies (using them to justify any number of frankly silly creative choices).
But, to make this broad claim (that they don’t mix with direct mail) is silly. What direct marketers miss with their focus group phobia are their ability to jump-start the creative process and show us new and exciting insights that drive creative to much more effective heights.
Focus groups are, and always have been, tremendous mediums for digging into depth – investigating those troubling issues. Direct mailers SHOULD be using them to look at what really motivates a consumer about a product enough to get them to respond to direct mail.
They just shouldn’t ever (EVER) be used to vote on creative.
Regards,
Doug Garnett
Thanks for the comment, Doug. I think you and I are singing from the same hymnal. I’m not dismissing focus groups altogether, just the notion of clients using them to pre-judge the performance of one campaign vs. another. However, for understanding core messages, and therefore ultimately what people might respond to best in a direct marketing campaign (as you point out), they absolutely can be a viable part of the direct marketer’s toolkit. Regards,
Howard
Agreed.
I work regularly with an outstanding moderator. Together, our mantra is that we should “never ask consumers anything they can’t really answer”. Because in a research setting, consumers answer every question – whether they know a valid answer or not. (Probably some hold over from the disciplines we learn in school.)
The ONE place where companies want to violate this rule most often are asking consumers to vote on creative. They don’t know a valid answer. Should it be red? The creative guys know that. All a consumer can do with something in from of them is tell you the feelings and ideas it raises for them – at that specific point in time.
It’s our fault if we choose to take this wrong and believe that the answer reveal anything absolute.
Cheers,
Doug
I like the consensus that has developed in this conversation! Focus groups can be very helpful, but only if used properly. Do not employ focus groups in place of testing. However, use focus groups to derive insight into potential future initiatives. To that end, the following are relevant excerpts from a case study that I published quite a few years ago:
“In order to identify multiple types of customers, our cataloger ran a tree analysis off database fields limited to merchandise categories and demographics. These are the most likely to offer clues on how to match the content of a catalog to a given customer’s needs…
“The results our cataloger’s tree analysis [indicated that] female customers generate sales-per-piece-mailed of $2.88, over twice that of the males’ $1.42. However, the analysis reveals a small pocket of male customers who, at $2.95 per piece mailed, are actually more responsive than the average female. These are buyers of jewelry merchandise.
“Our cataloger’s marketing staff, intrigued by this finding, set out to uncover additional clues about the purchase dynamics of this profitable customer segment…[As part of the subsequent investigation,] it commissioned a series of focus groups as well as some comprehensive survey research.
“The focus group and survey research determined that the majority of these individuals are gift-giving husbands. They are what the research company dubbed ‘unimaginative male gift givers.’ These are men who – despite their solid professional success – dread buying birthday, anniversary and holiday gifts for their spouses. They very simply are at a loss for what kinds of presents their wives might find appealing.
“In order to fully leverage these findings, our cataloger convened a task force. Comprised of representatives from Marketing, Creative, and Analytical, the task force’s mandate was to develop a loyalty program to appeal to these ‘unimaginative male gift givers.’ Based upon the insight gained from the research, the loyalty program was built around the following key features:
‘1) Automatic reminders for upcoming birthdays, anniversaries, and other personal milestones. This is a registry based on self-reported information, so that our cataloger can remind its participating male customers of upcoming special events that will require a gift. Based on customer preferences, these reminders can be sent via phone or e-mail.
“2) A consulting service for gift selection. The cataloger staffed its in-bound call center with specially trained ‘gift consultants.’ This service was tied to the existing database, to ensure that duplicate gifts are not recommended.
“3) All gifts are wrapped and then mailed to either the male program participant or – if he’s out of town – his wife. Likewise, a card is included, either blank or fully addressed.
“On the prospecting side, the cataloger’s circulation department began working with its list broker to identify new male-oriented lists to target prospecting catalogs. These catalogs include a description of the loyalty program as well as a form for signing up…
“Over time, our cataloger…was able to drive significant increases in its revenues as well as profits.”
This is one of those questions that by the very act of asking, reveals the lack of understanding of the issue. Direct mail is about numbers. How many you send out, how much it costs and conversion rates. Debra has it right – test, test, test. And testing means pursuing a statistically significant number from an unbiased group – neither condition is met by a ‘focus group’. The very name is misleading. I’ve sat through dozens of these in the past 25 years and have consistently seen marketing departments be swayed, irrationally, by non-significant input from statistically insignificant responses.
Direct mail is direct marketing at its best – get on with it and test!
As any good direct marketer would tell you, testing is the key to a successful direct mail campaign. Having the right balance of test elements allows you to understand what components are working in the focus group, and what needs continued refinement.