On Verdict Scales

avatar

On Verdict Scales

I spend a lot of my time thinking about kind of goofy stuff. For example, this article about Rolling Stone re-verdicting many albums over the years got me thinking about verdict scales. I'm reasonably happy with the Maximum PC scale, our average and median scores skew a little high, but that's partly because we tend to review stuff that we think readers are going to be interested in, since we have limited time and pages in the mag. The thing that I don't like is that there's a fairly wide swath of no-man's-land between a 6 verdict and the dreaded deuce that we rarely utilize.

To give you a point of reference, a verdict of 6 represents a product that very few people should spend their money on. The product performs only a limited function, and you'd be interested in purchasing it only if you have need of that one function. We reserve the 2 verdict for products that don't work, that don't have any hope of working, and that you wouldn't want even if they did work. (We also occasionally give 1 verdicts, exclusively to products that have all the qualities of a 2 but also do actual physical harm to the tester or his computer.)

But, I wonder, do we actually need this level of granularity for PC hardware reviews? I'm starting to think that 11 different verdicts is too many for PC hardware. (I'm not even going to talk about the crazy 100 point scale that PC Gamer uses.) I wonder if we could actually mimic the scale that stock analysts use, with five gradations--Strong Buy, Buy, Neutral, Don't Buy, Strong Don't Buy (we'd come up with sassier names for the magazine, never fear). I think that using a less complex scale would allow us to actually use the entire thing, with products more evenly distributed across a typical bell curve. Strong Don't Buy would be reserved for products that don't actually work, while Strong Buy would be reserved for products that would currently be worthy of a Kick Ass award.

I would never suggest ditching verdicts altogether. CGW bravely tried that last year, and they switched back to numerical verdicts within a few months. I know lots of people use verdicts as a substitute for reading a review, which I find overly trusty. When I'm reading verdicted reviews, I use the score partially to determine whether I should read the actual review. I'm much more likely to read an extremely positive or extremely negative review than I am to read a middling review, especially if I'm not interested in making a purchase and am only reading for entertainment. If I'm shelling out my hard-earned cash, then I don't just read the reviews, I actually study them.

What do you guys think? Let me know, by posting in the comments below.

 

[Addition 5/23/07]  For the record, I wouldn't suggest changing to a 1-5 scale. I think that the typcial five-star scale has the same problems our current verdict system has. My suggestion is to actually make the verdicts Strong Don't Buy, Don't Buy, Neutral, Buy, and Strong Buy. 

35

Comments

+ Add a Comment
avatar

vayac

Excellent idea to use something similar to analysts' stock market ratings in your reviews. I would find it much more meaningful as a guide. Thanks.

avatar

statewd

I love the 1 to 10 scale. Unlike the other poster, I really like reading the below 6 reviews since I get to see just how screwed up the product really is. It is entertaining to read a reviewer yelling at a product (constructively).

avatar

Strizza

Ahoy!

If the verdict-O-Meter is below an 7 I just read the pros and cons for a chuckle. Above an 7 gets a fair shake. As it stands Maximum PC seems to have 11 verdicts 1-9, 9kick-ass and 10 kick-ass(I've never seen a 10 verdict without a kick-ass stamp.)

Any verdict 6 or below just isn't a power user item(hoboparade) Verdicts of 7 or 8(Skosh of Cowabunga) verdicts 9(9 kick-ass)or 10 (sell a kidney). Well I'm sure people being paid can come up with cuter names.

In short revamp the verdict-O-Meter and state what to buy and what to forget about.
As for reviews at or below 6 I feel bad just breezing over them knowing how painful it must have been to test the products. Maybe there could be some cross at the end of the mag that pulls from the below 6 reviews(just spit-ballin). Oh what ever happened to 7.1 sound?

avatar

NathanielM

I think the 11 point scale works pretty well. Possible a slight modification in order to scale some reviews down away from the 7-10 range would make the scale function a bit better but I think that a 5 point scale lacks the precision to determine what you really want to buy. I'd assume that a 9, 9KA and 10KA would all fall into the top rating catagory but currently there are some subtle differences between those three ratings that give the reader more information.

avatar

webgrunt

I give this idea a 3. And that's on a 1-10 scale.

avatar

King of The LOSers

i look forward to opening up your mag and going straight to the reviews (right after the rig of the month) and look and see what got a kick ass award (if a product did). If you change the current scale i think you would lose the mag's down to earth....i ask you this....who would say "strong buy" (for the record not me)...but who would say..."Ya that {insert you tech product here} is KICK ASS"...The way it is now it is perfectly fine.

look out for the frying motherboard

King of The LOSers

avatar

soggybomb

I think that the system schools use (A, B, C, D, F) would make things easy. People easily relate to this type of system.

avatar

rcr14

I have spent some time looking at old reviews. Think about your question. Short answer, Don't change it, it works for me.

avatar

Fahzou

I love the 0-10 + Kick Ass award system, I think a lot of people do as well from reading the comments. Don't fix what isn't broken is good advice, but there nothing wrong with adding to an already great system. You could try adding a buy verdict as well, also price should have more impact on review scores . :)

avatar

bcweir

Current 1-10/Kick Ass scale works. Don't change a thing please!

avatar

sc123

If you switch to the "Strong Don't Buy, Don't Buy, Neutral, Buy, and Strong Buy" you run the risk of doing a comparison of say, 10 monitors and having 4 Strong Buys or Neutrals - leaving the reader to wonder which is better for him/her. This is a problem in a magazine that can't give reviews the depth they need to explain how one product may be better or different than another with the same rating.

I like the existing system.
-SC

avatar

Strizza

sc123 made a good point


"If you switch to the "Strong Don't Buy, Don't Buy, Neutral, Buy, and Strong Buy" you run the risk of doing a comparison of say, 10 monitors and having 4 Strong Buys or Neutrals - leaving the reader to wonder which is better for him/her."


If this situation were to arise I would first read the full reviews of the most expensive and least expensive gear in the "buy" and "strong-buy" categories, next I pull out a ruler and measure my wallet, I then match budget with reviews, and in the end I go further into debt.


skosh of cowabunga

avatar

crank

The scale is fine as is. So what if numbers are unused; they are free. I don't read a review if the product gets a score of less than eight anyway. I particularly like comparative reviews.

avatar

arcanus

If a 6 is the point at which very few would buy, it seems wasteful to dedicate 1/2 of your scale to indicating how bad something is. Once an arbitrary point of suckness is reached there's little reason to differentiate between "don't buy", "sucks", "blows", "blows like it's its job", "blows like a champ" and "it bit me".

Re-tooling the scale closer to say 3 "Fulfills a niche need or has large glitches that block usage" as the minimum buy number for either early adopters or niche users to 10 "fully endorsed by the heads of all major churches" seems more useful.

avatar

daveg

I don't understand this "wasteful" talk. Moving from 11 verdict slots to 6 saves you .. what? One bit? One character? I don't see that it makes any difference that some verdicts are practically unused. Let's face it: there just aren't many products that MPC reviews that deserve the low-end ratings, and that's mostly a good thing. If the current scale reflects real-world experience, what's the problem? Cramming those verdicts into one doesn't add any value to the reader. Playing games with scale to achieve a bell curve seems largely arbitrary to me.

avatar

Thoren

Both you and PC Gamer use essentially the same system, except I find theirs more precise... I wouldn't ever trade the "kick ass award" though. I disagree with the fact that people just ignore products in the 4-6 range... I try to read every review, since I like to see how you rate things I've used, and compare your ratings to mine.

Keep up the great work!

avatar

UnlnvlslblE

I think the 1-10 scale is fine. However you need to have a clear, precise explanation of the scale. You can generalize four subsections of the 1-10 scale. (IMO)

1-3 (Don't bother)
1 - Causes harm to the user
2 - Is completely non-functional with the possibility to cause harm
3 - Is completely non-functional

4-6 (Mediocre)
4 - Does one thing well and the rest poor
5 - Does most thing well
6 - Does what it claims to do and nothing more

7-9 (Excellent)
7 - Fills all claims and does it well
8 - Performs same features as competitor but does it with class
9 - Hands down better than any other product in its class

10 - Borg perfection

I think if the scale is well defined and easily accessible there won't be any complaining or second-guessing a verdict. The 1-5 scale gives less flexibility. You may as well use a 1-4 scale based on the 4 sections defined here.

To add in spice to the scale you can throw a Kick-Ass verdict at it as well. I really like the variety of the 1-10. The 1-5 is nice but this is Maximum PC; MORE IS BETTER!

avatar

mingoni

My spin is that 1-5 would actually be more germane. It's more important for you to continue with the great content showing us what features or lack thereof make up an overall rating. Personally, I would rather buy a somewhat dated part that was once rated kick-ass than a currently rated 6 or 7, or even 8. An overall score made up of say five individual scales (objective and subjective) of testing on each product reviewed translated for us into a 1-5 score. If you provide some detail for each of the individual scales we can decipher for a specific need. I agree, who spends time reading the lower rated product reviews especially when the Dog barks on what to stay away from. Personally I would love to see a second tier, or highly rated part that was previously rated Kick-Ass or Best of the Best incuded each month in an ongoing tally w/recent price. Once a year is probably enough for the all-out strictly budget weighted guide.

avatar

Lee_I

Half Life 2 deserved the 11!

avatar

Master123

I like the idea of a 1-5 scale. It certainly gives you more of a feel of how good or bad a product is, sicne 1-10 in my opinion leaves to many questions on whether it is a good idea to buy that product or not. I personally have been a fan of the 1-5 scale. Just make sure you keep the Kick Ass Award!

avatar

ssj4crono

1-10
1-10
1-10
1-10
1-10

avatar

ebayne

I could care less what scale you use, but Kkick Ass" should remain and be reserved for products that are not only rate a 10, (Strong Buy, The Squirrel's Pearls or wahetever the hell you decide to use)but have that certain something that set them apart from the "merely" excellent.

avatar

statewd

I like the 1 to 10 scale that maximum pc currently has because I always consider maximum pc THE first magazine I pick up for reviews. This magazine has much more weight on what I will think is a good product and buy then other magazines like PC Magazine, Computer Shopper, etc.

avatar

dcrail

I have always liked the rating system, it's simple. Me like simple. I don't mind not seeing verdicts in the middle of the scale (if it scores that low it's probably a product that is aimed at a specific customer, who needs/wants the product regardless of the verdict) The only thing I would like to see different is some long-term tests and comparisons. I can't prove it, but my suspicion is that most of us readers can't or won't spend the money to swap out high end parts with every new generation. Seeing what you think of a video card, or processor when it is no longer the cutting edge would give me more insight into my next purchase than reading through old issues for verdicts and reviews.

Keep up the good work.

avatar

William Baldwin

I routinely look to Maximum PC for a superb ratings scale. I, frankly, think the current rating system is great. In all honesty, MPc is the first magazine I come to for a good review.

avatar

blackzarg

I think the 5 verdicts are too broad. PC Magazine has a maximum of 5 stars with their verdicts, but oftentimes you'll see 4.5 stars, or 3.5 stars, which is pretty much the equivalent of a 9 or 7.

avatar

scottww

the scale is perfect, maybe just a harder review

avatar

Kerzon

While I have no issues with your scale method or PC Gamers scale (Some of the worst reviews they had were the funniest reads), But if your thinking of a new system i would probably suggest a star system or in MPC's case some ass-kicking boots.

avatar

VoodooChicken

True story, I left her crying at the altar when she developed this identity crisis a few years ago.

I'm perfectly happy with the current MaxPC scale (PCG scale works fine also). I really don't know anyone that fits their grading scheme to a bell curve, maybe Computer Shopper. I sure don't know anyone that LIKES scales that follow a bell curve. Skewed curves are embedded in us, most likely from grades at school (70 is passing normally; not 50)

----------------------------------------------------------
Look behind you! A THREE-headed monkey!!!!!!!!

avatar

nwasend

I for one like the 1-10 scale, but other users have a point. Way to many 8-9, 6 and kick-ass awards. You either have to tweak the system a bit, or have a more rigorous review to get a better sense of the product.

avatar

robster353

I find the current rating system quite useful, but maybe refining it a bit would eliminate or at least help reduce the no-man's-land syndrome and granularity issues. And I have to agree with you about using the rating as a guideline. To me, the real meat of the review is the text. That's what helps me decide whether or not to buy a product. So as long as you keep the hard-hitting reviews coming, I'm cool with whatever the rating system is. Maybe the 5 could be the kickass and then lower the dreaded 1 to a 0?

avatar

Abstrakt

I personally like the 1-10 scale, because its very easy to relate to, with no need to go inbetween numbers. if you went somewhere, say 1-5, I think you would find yourself having some trouble deciding between whether its a 2 or a 3 or a 3 or 4

on a side note, I dont ever think I saw anything that has gotten a 1... what would a product have to do to get a 0? kill someone?!

I especially like the kickass award as well. that in itself is something I look at when considering something to purchase. I actually was looking for a new LCD monitor, and went to the best of the best section, and what do you know, there was a samsung monitor right on top of the page and it was in my price range! I bought it within a few hours :)

avatar

Scapegoat

I like the idea for you to implement the "strong don't buy" to "strong buy" scale for your reviews in MaximumPC. However, I find your 1 to 10 rating helpful occasionally, and I definitely admire the kickass award. I think it would be cool to keep your 1 to 10 scale, while still implementing at the end of your reviews in highlighted letters (or below the 1 to 10 scale rating) anywhere from "strong don't buy" to "strong buy". So in essence, both included would be good.

Thanks for bringing up this subject, Will.

avatar

ponchato

I think it'd be a great idea. Being able to use the entire scale seems... less wasteful. And, it'd be much simpler, for us less-than-sharp people :D.

avatar

flapjack

I myself happen to really like the 11 different levels. I think that it's interesting to look at the numbers, and I find them more entertaining then a Strong Buy-Strong Don't Buy system. The only thing I think that needs improvement is the fact that you barely use numbers other then a 6, a 9, a 9 kick ass, an 8 or a 3. I have barely ever seen reviews with other numbers.

Log in to MaximumPC directly or log in using Facebook

Forgot your username or password?
Click here for help.

Login with Facebook
Log in using Facebook to share comments and articles easily with your Facebook feed.