Published on August 24th, 2012 | by Andy Astruc10
Review scores are poisonous
Recently, I spectated a conversation between two gamers in the comment section of a review. The game, whose title doesn’t matter, got a 7.9/10. Regardless of how unnervingly specific that score may be, it was the one the writer chose, and they might have had their reasons. The gamers were locked in a heated debate down below, with one trying to convince the other that the review made it sound more like an 8.5. Insults were flung and English was butchered in the name of deciding what a game can do to earn three-fifths of a score point.
Scoring systems infest the gaming industry, roots coiling around every aspect of our wonderful hobby. In a perfect world, they would just be one person’s opinion condensed neatly into tiny digits. A shorthand for the detailed analysis that came before. But they’ve become something more, mutating and growing in power as they feed off their own misplaced sense of importance. Now review scores aren’t just part of the article, they are the article. People will bluster and blow at the very idea of removing them. Readers will scroll down to the bottom of a review and just read the score.
The bloody-minded nonsense at the heart of the machine is that these precious numbers mean absolutely nothing. They are utterly pointless fluff that we invented to keep track of how good the things are that we made compared to other things. Imagine I came up to you in the street and told you that I thought hamburgers were a 9/10.
Medical professionals use a 1-10 scale to help judge how much pain a patient might be feeling, with 1 being almost no pain and 10 being completely unbearable, screaming, vomit-inducing discomfort. The system works because it’s personal. There is no objective measure of a 5 on the scale, only what that person feels. And there’s the rub: scores are perfectly acceptable as a personal indicator, but the average person sees them as the be-all and end-all of analysis. You might be able to argue that a perfect 10 and a miserable 1 could at least be agreed upon, but my idea of a worthless pile of rubbish and yours will be different. Fraser Brown – hirsute and surly reviewer – views a 10/10 as a perfect game, flawless in content and execution. Whereas I see a 10 as a game that did a fantastic job in everything it attempted and was complete entertainment. Neither of us are wrong.
Never mind the fact that you have several writers on the same site working with the same scale but from entirely different perspectives. Even if you think you know how a reviewer feels about their own scoring, it’s still just a number. 5/10 could mean anything from a technically proficient but mediocre game, to a title which was incredibly interesting but badly broken.
And then it gets tricky. Behemoths like Metacritic get involved. The purpose of Metacritic (and other aggregate review sites) is to give an overall view of opinions on a product. Unfortunately, a number stripped of context is nothing but a number. Without the review attached, scores might as well be random words shouted into the aether. To make things worse, Metacritic takes hundreds of these random numbers – some out of 10, some out of five, some on an entirely different scale – and mashes them together with the vim and vigour of a pre-teen girl holding two dolls. Out pops a new number, this time out of 100, and we’re supposed to believe this is an average.
Unfortunately people do believe it, as the Metacritic score is a ubiquitous component of many a large internet discussion on the quality of a video game. This, despite the fact that taking a collection of random opinions from people all over the world, condensing them into tiny numbers and then making those numbers into big numbers is about as representative as deciding where to go on holiday based on how many gold medals a country won at the Olympics.
The logic loops and utter nincompoopery caused by numerical review scores is far from harmless. From a journalistic perspective it can force the people who report a little too close to the people who create. Allegations of review tampering might not be as rife as comment sections may lead you to believe, but it would be naive not to see the potential. And it’s a lot easier – practically and morally – to justify bumping a score up one point for good PR than it is to rewrite a whole review. I personally know writers who have been asked (or forced) to change a review’s score so it “fits” better, whatever that means. Once I discovered a score I gave had been altered post-publication with no warning.
The idea that there is a right and a wrong score to give to a game is dangerous thinking. It leads to organisations and individuals deciding there is some imaginary checklist that represents the perfect game, and that a review is just marking off all the required bells and whistles. Does it have good graphics? Yes. Is there enough genre crossover? No. Can it multiplayer? Yes. Seven out of 10. Critical analysis is equal parts subjective and objective, and to remove the former from the equation is to create mob rule. Without individual opinions, all we have left as our judge is cold, hard calculation and popularity. And you know what’s really popular? Twilight, Call of Duty and ironic beards.
Companies have begun using Metacritic ‘averages’ to dictate their business practices. Obsidian staff were infamously denied their bonus payments for Fallout: New Vegas because the game missed the arbitrary cut-off for review scores. It has an 84. For human beings to get paid, it needed an 85. One percentile point, shaved off by people whose job it is to shove numbers into a blender, decided the amount of money people deserved to be paid for their work. It’s been pointed out that this was the contract the employees signed, as if that somehow absolves Bethesda of being complete lunatics. They are using random numbers to dictate policy on creative output. Sales numbers I’d have no issue with; at least they make some sense.
More recently, Irrational Games were found to be using Metacritic scores as a job requirement. A listing on Gamasutra – which has since been amended – asked for applicants to have “Credit on at least one game with an 85+ average Metacritic review score.” There’s that number again: 85. Somewhere in the clandestine offices of Games Industry Headquarters, someone decided that only games that score an 85 or higher are worthwhile. Never mind that 85 is far, far above average. And never mind that your work on the art design, or the music, or the character models, might have been the second coming of Robot Jesus, we’ve used science to come up with an objective measure of your talent based on hundreds of dissenting opinions and a collection of flimsy numerals.
Those on the business side can at least claim to have a solid reason for their kowtowing to the golden calf of review scores. They want to make money, and be sure of making more money in the future. Scores are a deeply flawed but simple way to judge how well a publisher’s product has gone. What really busts my noodle is that the people playing the games go right along with it. Consumers, whose only interest should be getting to play good games, will regularly take up arms and fight for titles to get the scores they deserve. Video game PR departments have somehow managed to poison the minds of gamers so completely that they will passionately and humourlessly debate the meaning of the number six.
There is a large segment of the gaming population that genuinely believes that a game can objectively be deserving of an 8/10 score. That every journalist and gamer should agree on exactly how good it is, what is wrong with it and why. That terrifies me. One of the better aspects of gaming as a culture is that it isn’t really a culture at all. Gamers come from all walks of life, and the wrenching dread in the pit of my stomach when someone argues about the number at the end of a review is the same feeling I get when people wax idiotic about casuals ‘ruining’ the industry.
And it’s all an argument about numbers. Not the content of the review. Not the underlying problems that might be present, or what it means to the industry as a whole, or any of a thousand different discussions one might have about interactive entertainment consumption. Numbers. Disagreement is a healthy jumping-off point for conversation, but shouting about the validity of decimals makes us all look like children.
We shouldn’t need to detach ourselves from a game to evaluate it; these are not history essays, they are video games. In order to competently talk about them we need to play. Experience. Immerse ourselves. And that is incompatible with a system that rewards passionless analysis. Gamers and journalists alike can so easily forget what actually makes a game worthwhile. I would rather play a broken game that is fun, exciting, entertaining and interesting over one that is technically proficient and lacking in other, more subtle, areas. It would behoove us to step back from our established traditions and ask who exactly they are benefiting. And random number assignments aren’t doing anyone any favours.