Online product reviews are a critical part of my buying process. But how useful, and how legitimate, are the product reviews we see on merchants’ web sites?

I was amazed this past week, after I installed the Nook e-reader software on my Android tablet, to find that the book reviews on the Barnes & Noble web site include massive amounts of spam and graffiti*, as well as anonymous customer ratings without any text. It appears that these reviews aren’t screened by humans (nor even by software), and the result is that the aggregate customer ratings shown for many books are useless.

What I found most upsetting about B&N’s review system was that many books carried 4-star or 4.5-star ratings, but when I viewed the reviews, the first several pages of reviews (those voted as useful by B&N’s customers) are overwhelmingly negative.  That’s because every spam or graffiti review, and every anonymous, wordless review, are each assigned the same weight as a well-written, cogent review.

While Amazon’s rating also appears to be a raw average of the star ratings of all reviewers, its data isn’t corrupted by anonymous or wordless reviews, nor by graffiti or spam (Amazon’s reviews must be at least 20 words long, must use either a real name or a pseudonym, and must be linked to a specific customer account; they’re also screened by humans to weed out spam and graffiti).

Both Amazon’s and B&N’s review systems are vulnerable to “shill reviews” — not only positive reviews posted on behalf of the publisher or author, but also negative reviews posted by folks who simply dislike the author or the theme of the book.

Both Amazon and B&N allow their customers to rate the usefulness of individual reviews (yes or no), and both try to display “useful” reviews first.  On Amazon, reviews are displayed in sequence based on the percentage of useful-vs-not votes (so that a review which is rated useful by 5 out of 6 people is shown before a review which is rated useful by 6 of 11 people).  B&N’s reviews appear to be sequenced based on the raw number of people who found the review “useful” (so a review rated useful by 4 out of 9 people is shown before a review rated as useful by 3 out of 4 people).

I’d recommend to B&N that they allow customers to “opt in” to exclude anonymous or wordless reviews from the ratings they see.

Ideally, any rating should incorporate the “usefulness” data, allocating more “weight” to reviews deemed useful. (Alas, this would likely increase the abuse of that system, too.)

But neither review system is effective for books which haven’t earned more than a few reviews. On either system, search results display a raw rating without showing the number of reviews. Thus, a publisher or author can post a new book, and then someone can post one or two “shill” reviews assigning a 5-star rating, and the book will immediately be displayed with a 5-star rating. If there are 50 five-star “shill” reviews (very few rated as useful by visitors), and 10 well-written, cogent reviews which all assign just one or two stars (all rated as useful by most users), the book’s rating will still be displayed as 4 or 4.5 stars. That’s just ridiculous.

_______

*/ What’s the difference between “graffiti” and “spam” in B&N’s product reviews? I consider a review “spam” if it promotes a web site, a product, a service, or a political candidate or issue. It’s “graffiti” if it’s nonsense, bullying, or hate speech. Both are also usually unrelated to the B&N product page where they appear.

_______

Update (May 21): I just viewed the product page for Toni Morrison’s new novel Home, and here is the full text of all 18 reviews posted on bn.com for the novel:

1. I CANT GET YOUR VOICE OUT OF MY HEAD ALL I HEAR ARE THE MANY ECHOES OF THE DARKEST WORD YOU SAID AND IT WAS DRIVING ME CRAZY I CANT GET THE BEST OF ALL OF THIS BUT IM ALWAYS LOOKING OUT FOR YOU CAUSE YOURE THE ONE I MISD???
2. Hello im josh mother is dee daughters are jenny and eve would u like to b adopted into our family? All are adopted
3. I want Amy! I like her!*goes up to Amy and hugs her*
4. R u my daughter u asked .me if u want to go with them though i umm understand looks sad ~amy
5. *sighs* ill be in my room
6. Good morning.
7. Night
8. Anonymous – No text was provided for this review.
9. Anonymous – No text was provided for this review.
10. Anonymous – No text was provided for this review.
11. Anonymous – No text was provided for this review.
12. Anonymous – No text was provided for this review.
13. Anonymous – No text was provided for this review.
14. Anonymous – No text was provided for this review.
15. Anonymous – No text was provided for this review.
16. Anonymous – No text was provided for this review.
17. Anonymous – No text was provided for this review.
18. Anonymous – No text was provided for this review.

Comments are closed