← See all of our blog posts

The Hidden Dangers of Review-Mining (and How to Easily Avoid Them)

By Momoko Price

Smart copywriters looooooove online customer reviews.

Why? Because customers have this INCREDIBLE talent for generating highly specific, attention-grabbing statements about products without even thinking about it.

Statements that — with just the tiniest edits — often outperform any “creative” headlines or hooks the world’s top copywriters come up with on their own.

It’s true: online reviews are a goldmine for coming up with powerful and authentic headlines and hooks.

But … can you rely on them ALONE for choosing the right unique value proposition for your business?

Can you pore over 100+ competitor product reviews and come away with a clear understanding of what ONE key message will intrigue your prospects most effectively and sell the value of your product?

Some copywriters might argue yes.

I say no.

Don’t get me wrong: I wish I could say yes, because it would make my job as a copywriter and CRO consultant way, way easier.

I’ve pored over enough user reviews to conclude that while product reviews are ah-mazing for finding killer headlines and product descriptions …

… and equally great for brainstorming a big list of *potential *value propositions to choose from for your product ...

… they’re a pretty unreliable way to actually choose which value proposition is the right one to roll with for your product launch.


Simple. As a source of voice-of-customer data, product reviews are HELLA MESSY … And messy data is unreliable.

Even when review sites enforce a clear question-answer format (the way sites like G2Crowd does), people tend to write long, rambling responses that are incredibly hard to quantify and compare objectively.

Which makes it incredibly hard to quantify which message is the right one.

As an example, let’s look at this sample review of Salesforce:

long rambly salesforce review

Now don’t get me wrong:

This is an amazingly rich source of voice-of-customer data, and there are a bunch of great proto-messages in this review that as a copywriter, I would swipe as headline fodder in a heartbeat.

But if I had to choose which message is the most powerful one out of everything mentioned in this review, I’d be pretty stuck.

Just look at the reviewer’s answer to “What do you like best about the product?” (a question that should theoretically result in a clear, singular, value proposition-focused answer):

It’s a mish-mash of about FOUR possible messages: scalability, customization, integration, and project organization.

.... The customer was asked for the ONE thing he liked best … and he gave FOUR.

(When it comes to online reviews, this happens like 80% of the time.)

And as if picking the best message from one messy, unstructured review was hard enough, now think of how you’d go about picking the top messages from 50 reviews ... or 100.

How would you go about picking the best one out of all of them?

… Now you see the problem.

To get around this, most people — heck, even seasoned copywriters — just focus on grabbing the most colourful, memorable phrases possible that seem like they’re mentioned the most often, then pick a favorite and roll with it.

Personally, I’m not a fan of this approach, because of the very real danger of confirmation bias throwing your messaging research completely off-track.

Thankfully, there is another frequently-overlooked-but totally-worth-the-trouble tool you can use to choose a value proposition based on QUANTIFIED, OBJECTIVE customer feedback — rather than messy, cherry-picked opinions from reviews — even when you don’t have any customers.

It’s not free, but it is pretty darn cheap. And fast. And most importantly: it's relatively conclusive (at least compared to a mountain of rambly reviews).

Pay-per-response surveys.

Pay-per-response surveys are awesome, and with relatively new, affordable survey apps like Pollfish, they are a fantastically fast way to get conclusive, objective feedback from your target market about what your product's value proposition should be.

Why are pay-per-response surveys so awesome? Plenty of reasons, here are just a few:

1. Far less room for misinterpretation/confirmation bias
With pay-per-response surveys, as long as you design your questions intelligently, respondents will give their perspective in the form of clear, singular, categorical answers. Which means no more guesswork about which message is the most important.

2. Ruthlessly honest answers from real prospects Survey respondents have NO idea who you are, which is a good thing. They don’t care if they hurt your feelings, and they aren’t giving their feedback after being prompted by a customer-service representative or a review site curator. So if your idea doesn’t seem relevant to them, they’re gonna to tell you, straight-up.

3. Large sample sizes fast
Survey apps like Pollfish enforce a minimum of 200 responses to ensure that the overall trends in responses you see are reasonably reflective of the general population. And while a sample of 200 is still not huge statistically speaking, it’s much, much more reliable than say, a handful of 1-on-1 interviews or an expensive focus group.

4. At-a-glance trends
Most pay per response tools will summarize your response data for you, with handy graphs you can download and stick into reports and other deliverables as needed. And nothing gets client/boss buy-in on your hypothesis than nice juicy bar graphs showing a clear winner, based on good data.

chandler's boss from friends

Personally, when I’m doing messaging work for a pre-launch startup, I use the following workflow to get the most out of each resource at my disposal (i.e. surveys + reviews):

STEP 1: First, I get insight about which pain points & problems the product is meant to solve from company stakeholders via an internal workshop/discussion. I then grade these pain points according to frequency and urgency.

STEP 2: I then validate how real-life prospects perceive the top-ranked pain points (from Step 1) using a pay-per-response survey — specifically one that includes a well-crafted screener question to filter out irrelevant respondents. I keep my eye out for for standout pain points in the results data, a.k.a pain points that survey respondents themselves rank as the worst and the most desirable to eliminate.

STEP 3: Finally, with the top pain points from Step 2 in mind, I collect bits of copy I can use for headlines and subheads by mining competitor product reviews, focusing specifically on any turns of phrase that accurately and vividly capture the emotion and experience of said pain points.

STEP 4: Last, I edit them into value-proposition-focused headlines to test further via PPC and Facebook ads.

The result? A super-clear, super-fast, research-driven assessment of which value proposition messaging will most likely work best for my client’s upcoming launch.

Curious about exactly what you should ask in your pay-per-response surveys? I’ll be sharing my favorite pre-launch survey template in an upcoming blog post … Stay tuned!


Lifeless value proposition killing your product sales?

Learn how to build higher-converting value propositions in a refreshingly research-driven way. Subscribe to The Kantan Letters to get a copy of my popular ebook, "The Kantan Step-by-Step Guide to Finding High-Converting Value Propositions", and fix your broken value props for good ...