November 30, 2012

Idea: A Plug-in That Throws Out the High and Low Score Reviews

This summer the New York Times reported that, by one estimate, a third of all consumer reviews online are fake. And 1-star reviews are often misguided or misplaced anger, which can make them pretty much useless. So some suggest that the best thing to do when looking at consumer reviews is ignore all the 5-star and 1-star reviews, and just concentrate on the middle reviews, since that’s where you’ll find the substance. I’ve seen it compared to how some Olympic events throw out the highest and lowest scores to find the true score of a gymnast or diver.

So here’s an idea: how about a browser plug-in that does that automatically? I’d like one that hides the 5-star and 1-star reviews on Amazon, TripAdvisor, and the like (are Apple App Store reviews were available on the web?). But maybe too little data is front-facing, making it impossible without deep access to databases that aren’t publicly available. So I’ll settle for toggles added by the retailers themselves that allow me to turn on and off the high and low reviews.

This is the Best Idea Ever! ★★★★★. I would definitely read this post again!

See also: Product Placement in Amazon Reviews for Related Products


I generally look at the distribution of the ratings on Amazon. I find that to be most telling about the quality of the product over all.

Personally, I think what would be even more useful is a plugin that removed any ratings that didn’t include text descriptions of at least 200 characters (or some other arbitrary number that’s large enough to weed out the “omgee this is awesome” type reviews without stepping on the toes of actual, well thought out, reviews).

There can be really good, legitimate reasons a person would rate something 1 star (maybe the product was COMPLETELY falsely advertised), or 5 star (it really is just that great), but making sure to only count the ones people have put thought into would be really handy.

Any scoring system is of course likely to be subject to abuse by those with axes to grind or competitive products to promote. I tend to favour Jim’s approach in that a well-considered 1 or 5 star review can be worth more than any number of mid-ranking comments. So I like the idea of stripping out one sentence reviews (with the benefit of also removing 200+ character one-sentence reviews). I’d go further and based on keywords ignore any reviews, irrespective of score, containing words like OMG and awesome, and not written in mixed case.

What about reviews weighting on combinations of the age of the reviewer’s account, number and spread of the reviews posted and the number of ‘likes’? Factor in other demographics gathered by major retailers and advertisers, crunch it all together and only show the reviews from your virtual Facebook friends. No system will be perfect, just as long as it is transparent and not skewed by magic page-ranking type algorithms promoting good reviews based on retailer’s profit margin.

On second thoughts, it might be easier to go to a store, play with the product, talk to real friends and make up my own mind.

Ignore high and low scores ★★
Ignore short comments ★★★
Show only weighted comments ★★★★
Don’t shop online ★★★★★

5 star and 1 star reviews can be helpful. I think knowing more about the reviewer is the best direction to go. So having friends or acquaintances reviews available is helpful, as is the idea of seeing a persons other reviews.

I think what Yelp does with this is a pretty good example. If someone gives a bad review of a store, I can look at their other reviews and try to tell if they’re just bat-shit crazy.

“So I’ll settle for toggles added by the retailers themselves that allow me to turn on and off the high and low reviews.”

uhmmmm… considering that a good part of that 30% of fake reviews come, directly or indirectly, from the retailers i don’t see this happening soon.

and even if it did they would simply start flooding the store with 4 stars amazing reviews… ;)

Actually, I make a point of looking at the 1 star reviews, to be aware of any horror stories or important limitations. You can tell which ones are just bitching (or stupid) and which ones are legitimate problems pretty easily. I know that I’ve been glad I read the 1-stars on a product that, it turned out, didn’t do what I thought it would, as others before me had also thought and were disappointed.