← See all of our blog posts

Yet ANOTHER Danger to Avoid When Review-Mining

By Momoko Price



In today's li'l edition of the Kantan Letters I'm bringing up YET ANOTHER danger to watch out for when mining product reviews (arguably one of the most popular sources of voice-of-customer data for conversion copywriters):

FAKE reviews.

elaine benes from Seinfeld pointing and saying 'fake fake fake' from the fake orgasm episode

Fake product reviews are nothing new, but only recently has it come to my attention just how natural and authentic-sounding fake user-generated content has become on the Web.

Fake (i.e. bot-driven or paid-for) content — especially on social media — has become so widespread and convincing over the last few years that it now plays a major role driving our community discourse and education on key public issues.

Don't kid yourself: Bot-drafted content is WAY harder to detect than you think

Scary? Sure, but until the politicized insanity of 2020's news coverage hit, I'd always just assumed that my bot-detection abilities were up to snuff.

When fellow copywriter Jen Havice sent me a link to Spot The Troll so I could put my fake-content detection skills to the test, I was shocked to discover just how sophisticated bot-generated content has become.

[ If you think you don't have to worry about being duped by a bot yourself, I challenge you to take the test yourself — it only takes 2 minutes and is quite informative! Click here to test your bot-detection skills. ]

After I took the test — and only got 6/8! — I realized two things:

  1. It is extremely likely that over the last year my perception of humanity at large has been significantly manipulated by an army of bots (and so has yours, regardless of how perceptive and well-informed you are).

  2. It is extremely likely that bot and/or paid-for product reviews are significantly manipulating our impressions of products online

Before you mine reviews, clean your sources with these (free & easy) tools

Thankfully there ARE a handful of fake-review-detecting browser extensions out there that you can use to check for fake content before you start mining reviews as a resource:

  • ReviewMeta.com
  • FakeSpot.com

(You can use the websites or just install the browser extension — they're both free.)

Unfortunately, the one thing these tools don't do — as far as I know — is allow you to actually export a product's trustworthy reviews into a separate file for analysis. ReviewMeta lets you see a sample of "good" reviews vs. "bad" ones, but it doesn't let you pull all the good ones to work with locally.

(I'm currently looking into ways to do this. Stay tuned ...)

Using these tools to check how polluted a product's reviews are is a good habit to get into as a copywriter, but it does raise the question of how much you want to be relying on product reviews for directing your messaging and writing your copy to begin with.

Want the cleanest, most accurate VOC research? Use surveys, not reviews.

In the past, I've written blog posts and done talks about the dangers of relying solely on product reviews when writing sales copy.

But now that I'm more aware of the prevalence of fake content online — and more importantly, how ineffective my human brain is at detecting it — I'm even more reticent to say that product reviews on their own are a legitimate source of voice-of-customer data.

At the end of the day, I still strongly believe that customer & product responses to well-designed, properly-targeted, messaging-focused surveys & polls remain THE hands-down most credible source of quantified, usable voice-of-customer copywriting research, both for selecting your value proposition and for drafting sticky, compelling copy. I'd rank recorded and transcribed customer & prospect interviews second, and finally — if you can't get your hands on either of the above — product reviews should really be a last resort source of VOC.

What do you think? Have YOU had much luck writing effective sales copy based on product reviews?

Personally I feel like my confidence in them has really shifted over the last few years.

— Momoko

Lifeless value proposition killing your product sales?

Learn how to build higher-converting value propositions in a refreshingly research-driven way. Subscribe to The Kantan Letters to get a copy of my popular ebook, "The Kantan Step-by-Step Guide to Finding High-Converting Value Propositions", and fix your broken value props for good ...