The Parthenon in Athens, Greece. Photo courtesy of caribb.

Are reviews primarily a consumer guide, or should they serve another purpose? Do review scores deter intelligent discussion of videogames? Is the presence or absence of a review score the only difference between a reviewer and a critic? What is the role of the reviewer when the Internet is democratizing published opinion? How should reviews and reviewers evolve in light of the emergence and growth of Flash games, small games, indie games and user-generated games?

These questions and more were on the mind of N'Gai Croal, John Davison and Shawn Elliott last summer when they decided to expand their conversation to a number of noted reviewers, writers, bloggers and journalists for a published email symposium on game reviews. (See below for the full list of participants.) The planned list of topics include Review Scores; Review Policy, Practice and Ethics; Reader Backlash; Reviews in the Age of Social media; Reviews in the Mainstream Media; Casual, Indie, and User-Generated Games; Reviews vs. Criticism; and Evolving the Review.

The topic for Round 1, which will be published here in installments over the next several days, is Review Scores. Yesterday we published Part I, today we continue with Part II.

Participants

• Leigh Alexander, Gamasutra/Sexy Videogameland/Variety
• Harry Allen, Media Assassin
• Robert Ashley, freelancer
• Tom Chick, freelancer
• N'Gai Croal, Level Up/Newsweek
• John Davison, What They Play
• Shawn Elliott, 2K Boston
• Jeff Gerstmann, Giant Bomb
• Kieron Gillen, Rock, Paper, Shotgun
• Dan Hsu, Sore Thumbs Blog
• Francesca Reyes, Official Xbox Magazine
• Stephen Totilo, MTV News

***

Robert Ashley, freelancer: I don't advocate putting an end to scoring. Scores seem to be the one thing that today's online audience can easily form a conversation around, and I think, ideally, a review should be like a conversation between reader and critic. I just think the incredible seriousness surrounding scores (born in no small part, I'm guessing, from the fact that retailers stock their shelves based on review scores, ratcheting up pressure on critics to treat scores like jury sentences) is irritating. Handing out a mediocre score to a mega-hyped game can brand you a heretic (a Crispin Boyer, if you will) when you're just trying to be honest about your reaction.

***

Stephen Totilo, MTV News Multiplayer: Who is actually upset about review scores?

Offended publishers who wanted a 9? Stressed developers whose bonuses depend or ability to get another deal require them to get at least an 8? Superfans whose feelings you hurt by giving the game they are going to buy anyway a 7?

Before we do any more debating about the merits of putting a number on a score I want to know who cares.

Have any of you come across gamers who won't buy a game they were curious about because you gave it a 7 and not a 9? And, if so, would they have made a different decision if your review didn't include a score? Was it review scores that did in Too Human or put Wii Music slow out the gate?

Do scores ever really hurt or help games? Or are we just debating the best way to describe a game's quality, be it through numbers, words or faces in various stages of excitement?

***

Leigh Alexander, Gamasutra/Sexy Videogameland/Variety: Backing Stephen on this. Scores do stress the reviewer, but the stress doesn't come from any source that really counts--excepting maybe the anxiety I get sometimes knowing people's jobs depend on, say, Metacritic, and hoping that I was as thorough and fair as I could possibly be.

Like some others who've chimed in, I write the text first, and then see what score the text supports. In a way, I'm not assigning the number as a be-all measure of the game itself, but as a shorthand for my evaluation, and I think that's the function for which scores are most useful--as Jeff said, a single value that makes sense from a macro viewpoint for people who don't read text. There are a lot of those people (which makes me wonder if we shouldn't be trying to create more accessible, readable text, but that's probably a whole 'nother issue).

Finally, because I talked about preconceptions, I just wanted to clarify that while I may sometimes go in with ideas, hopes, dare I say biases, I never go in with a number in mind. The number is the last thing I come up with.

***

Dan "Shoe" Hsu, Sore Thumbs: To Stephen: Yes, yes and yes. They all care. But in the end, that review and score aren’t for them anyways -- they’re for your “normal” audience--so it really shouldn’t matter how bunched up their panties get.

At EGM, we’ve had plenty of readers who told us they would not even consider a game purchase if the reviews didn’t average a certain score they had set in their minds. Now, this “certain score” is usually a moving target--higher if it’s a game that reader wasn’t originally interested in, lower if he already had that title on his Amazon wishlist. This overreliance and faith in this one rating (scored by someone you probably don’t know intimately well) may seem silly, but I’m absolutely with Jeff and John on this: Despite complaints from a vocal minority, the vast majority of readers really want that number, letter, or direction the thumb’s pointing. It’s ingrained in society and it’s pointless and stubborn to fight it. People don’t always have time to read a 2000-word, well-crafted review to get inside the brain of the reviewer. For most folks in this short-attention-span world, that “4 out of 10” usually says more than enough.

Are we off-topic here, by the way? To answer question #2 above, yes, I sometimes change my score after I write the text. I do it like Robert initially: I score with my gut. But then while writing a review, I get to reflect upon my play time, think back to my progress 10, 20, or 30 hours ago, check back on my notes from last week, etc....and then I might adjust my score (usually by one increment up or down) based on my experience in its entirety. This may seem obvious--to score the whole game, not just how it finishes--but I also have that same short-attention-span problem. And this helps me keep too much emotion out of the process, too, so a game’s high-note finish doesn’t unduly inflate the score.

***

Francesca Reyes, Official Xbox Magazine: I can’t tell you how much I personally hate coming up with scores. It’s putting a quantitative label on something that’s qualitative. But that said, it’s a necessary and completely understandable function of game reviews in enthusiast pubs. (Hell, when I read movie reviews or book reviews, I do the same thing--look at the score to justify whatever half-ass, cobbled together pre-opinion I may or may not publicly admit to having.) The reviews I’m used to writing aren’t criticism in the sense that you’re a “reader” and you’re “reading” a game like you would text, or a movement, or even a movie--they’re practical forms of consumer advice. So, I agree with Jeff in that we’re here to inform our audience if a game is worth X amount of money based on whether it delivers, and to what extent, on its back-of-the-box promises, as well as how it handles as a game. And that demands a score, really. A signpost for what the review contains.
 
I’ve been lucky that all the pubs I’ve worked for fall in the realm of “enthusiast,” so in a lot of ways, you’re writing for gamers like yourself when you score a product. You’re also using a set of established criteria, depending on your publication’s ranking scale. If a 7 in a certain magazine means “average,” while a 5/5 stars in another means “must-have,” etc., this is what you’re working off of.
 
I remember writing a review of one game across multiple publications. One had a 100-point scale, another had a four-star scale, and yet another had something else. The thing that sticks out in my mind is that the 100 point one was for Ultra Gameplayers and the star-scale was Next Generation. UGP was your regular “review as a gamer.” Next Gen was “review for innovation and uniqueness, like ‘does this push the boundaries of its genre, etc.?’”. One game, reviewed against two very different sets of criteria. Interesting contrast. 
 
So yeah, it’s totally ideal and utopian to think that you can sum up a game in a one-word or one-sentence definition from a pre-existing list of rankings, but this ain’t science. It’s voodoo magic in a lot of ways, no matter how hard we try to justify what the numbers, letters, or stars mean.
 
I try not to go into a review with a preconceived score in my head, but like Shawn said early in the thread--the result of months of pre-release hype or non-hype may or may not play a role in expectations from the reviewer and the reader. You may never really know. It’s just human nature, really, and publishers know this. But the trap is that we all play a *** ton of games, right? But our readers do not. Yeah, some of them play a lot of games that we haven’t. Some of them may play as many as or more than we have. But that’s the small portion of our audiences. Most of them pick and choose what they buy and we have to understand that spending their cash may rely heavily on what reviewers say.  You have to respect that and go in to a game with the same expectations that someone without the months of exposure to a title might have.
 
Maybe that’s idealistic to expect this of writers who are supposed to be “experts” on their field (how do you become an expert in a field or medium if you’re not exposed to everything it has to offer, right?), but when it comes to reviews--in a lot of ways the boss is your reader and you have to kind of get in their skin. Am I always successful at doing this? Hell no. But it’s what I always aim for.
 
As for the actual process of coming up with a score--sometimes you just know, based on games you played before or a gut feeling when you’re playing it. I sometimes wait to put the score in the review until after I’ve written it so I can step back and get some perspective. If I have the luxury of time (shyeah), then I can let it sit for a bit and return to it for another pass to see if the score still holds. I like how Shoe mentioned the “high-note finish” and I agree. Games are experiences and once you see the entire narrative a developer has to tell you, there’s a sense of accomplishment that sometimes make you review through rose-colored glasses.

***

N'Gai Croal, Level Up/Newsweek: Robert, I’ll raise my hand and say that I do advocate putting an end to scoring. You wrote, "Scores seem to be the one thing that today's online audience can easily form a conversation around, and I think, ideally, a review should be like a conversation between reader and critic." Yes, a review should be a conversation between reader and writer. But what kind of conversation do review scores foster? Judging by the Metal Gear Solid 4 example Kieron cited above, not much. (We’ll get into that more when we tackle Reader Backlash.)

Jeff, you wrote, "scores are primarily there to serve as shorthand for folks that won't or can't read the full review." Unless I'm missing something, that's quite an indictment of a portion of your audience. Why would you write for people who won't or can't read an entire review? You also say that scores "assist average, everyday people in their purchasing decisions." I don’t object to heds and deks; pros and cons; bullet points; final words; buy, try, fry; and other forms of giving readers a succinct take on a reviewer’s opinion about a game’s value. Heck, you could just tell them how much you think the game is worth. And like you, Fran, I firmly support the consumer guide function of game reviews. But anyone who won't or can't read an entire review isn't making an informed decision by looking at a single review score, so I’m not convinced that the reason review scores are “necessary” is to genuinely inform those consumers who can only focus on a letter grade, a number or a star rating.

In a different context above, Shawn talked about intentional fallacy. I think that's what's happening with review scores in the age of the Internet. Those of you who assign scores intend them to perform a certain function, but in the real world, the use to which they're being put by the most vocal portion of your readership is pernicious. Scores help bring out the worst in readers. They shut down conversations; foster silly debates; and they encourage meaningless comparisons. For too many readers, the very presence of scores turns the text of a review into a sideshow for the main event: this number, those stars, the orientation of that thumb. The text becomes a caption and the score becomes the photograph, en route to becoming the final cog in the Gamer Metrics-Metacritic-GameStop machine. Why would any of us want to perpetuate that?

This may seem easy for me to say from the perch of a blog and a magazine that isn’t dependent on gamers for its survival. But many of us work for or were previously employed by outlets that have been struggling. None of us are safe. Newsweek had a round of buyouts this year, the third in my nearly 14-year tenure at the magazine. Time Inc recently had layoffs. MTV had layoffs last week. Ziff-Davis closed Games For Windows magazine. The Tribune Company just declared bankruptcy. For years, magazines and newspapers have been moving towards bigger photos, more charts and even “charticles.” The Associated Press has imposed a 500-word limit on its entertainment writers. All of this devalues the importance of the word. Review scores are yet another signal to your readers that your words don’t matter.

If we don’t think scores are genuinely meaningful--there may be a robust defense of the inherent value of review scores; of the 6.5 versus the 7.0 and the 82 as compared to the 89, but no one has offered it yet--why do we continue this charade? Shoe, you wrote that the desire for scores is "ingrained in society and it’s pointless and stubborn to fight it" and that "For most folks in this short-attention-span world, that '4 out of 10' usually says more than enough." Apparently it’s not enough for most folks that we slit our own throats; we’re expected to provide the knives as well. I won’t pretend that yanking review scores will bring an age of genteel conversation or Socratic debate to the intertubes. But if message boards must be clogged with pointless argument, I’d rather it be fuelled by the words you wrote rather than the numbers you assigned.

***

Kieron Gillen, Rock, Paper, Shotgun: Since you ask for a justification for scores and the fetishized graduations of hundred-points scale, N'Gai, let me give it try. I'm from the wanky English tradition, and was anti-scores for most of my adolescence for the obvious reasons. They're stupid and putting any kind of scale over a subjective experience was ridiculous. For game reviews, at the current point in the medium, I've come around to them on solely utilitarian grounds.

For scores, per se: reviews exist fundamentally as buying guides, the proverbial ***-filter. They can do other things, but the reason to exist is to spend other people's money. When I score, for the pure consumer magazine (as opposed to one with a more critical leaning) I score for one thing only: should you buy it. For those less-prominent games, a high score is a direct tool to even make them read it--World of Goo and Braid getting a string of 10s is an obvious flare for ATTENTION! in a way which just giving them a glowing review isn't. If reviews are a *** filter, the mark is a very blunt tool for achieving that. That, to many gamers, is just what they want. In a real way, N'Gai, that's the job.

(This will come up later, but the thing about classical game reviews is that it isn't like film or music reviewing, but a hybrid between some stuff that's purely subjective and some stuff that's objective. You don't get a band's new CD which doesn't work on most CD players. You don't get movies which freeze randomly. For a review, that stuff genuinely matters and we're betraying our readers and being deeply disingenuous if we pretend otherwise. Are there other forms of games writing for other readers? Hell, yeah. But dismantling the review isn't the solution. We should just go and build something else.)

And the hundred-point scale? Perversely, what I most like about it is actually its weakness. It's inherently ludicrous. Who can tell the difference between 83 and 86 percent? No-one. In other words, its subjectivity is totally clear. The 10 or 5 point scale has a way of actually tricking people into thinking there's some science at work. The hundred-point scale--and calling it "percentage" scale is another thing that's deeply deceptive--is very silly. The fact it's a rough tool rather than a scientific implement is blatant. If you try to argue a few percentage scores you look as if you're suffering from some obsessive-compulsive disorder in a way you don't necessarily look like if you try to argue between a 7 and a 10. 100-point scales, treated correctly, are an ideal way to both act as a shorthand for the review, and simultaneously make it clear the mark isn't the review.

***

Shawn Elliott, 2K Boston: To add to N'Gai's thoughts....

Some of us suggest that our audiences sees scores as buyers' advice. Actual sales rarely correlate with review scores in cases where games are not also heavily hyped and marketed. Increasingly, gamers pre-order games prior to the publication of reviews. Interactive demos allow our audiences to decide for themselves whether or not a game will be worth their dollars. In addition, word of mouth and message board discussions inform our potential audiences' purchasing decisions with an intimacy and directness that we cannot provide. Finally, review aggregation sites such as Metacritic mute the bias of individual reviewers and provide a bigger picture.

I suspect these circumstances suggest that our self-perception is, well--a throwback to a time when magazines and websites were gaming's gatekeepers.

And yet we have John's anecdote about the angry reactions of some Computer Gaming World readers when the magazine dropped its scores. Robert introduced the idea that "review scores have one use: driving traffic from message boards and social networks to your site and giving those people an excuse to argue out their fan beefs in the comments section." Jeff countered that ratings "are primarily there to serve as shorthand for folks that won't or can't read the full review," which prompted N'Gai to ask why anyone would want to write for such an audience. Maybe our audiences aren't a homogeneous monolith--not in the sense that different readers look to a Level Up or Gamespot for different reasons, but that we (and our bottom lines) want or need different readers look to a single site or magazine for many reasons. Is this part of the problem?

***

Stephen Totilo, MTV News: Shawn concluded his note with "Is this part of the problem?" I ask again, as I did in my previous note (with slightly different phrasing), what's the problem? Who or what are review scores hurting? N'Gai makes a passionate argument against the damage he sees review scores doing to the discourse about games on some message boards and comment threads. And he pitches a convincing case that such damage obscures the value of the scorers' words. All told, though, that doesn't seem like a whole lot of pain.

But, again, what's the problem? Are quality games not being appreciated because of the existence of review scores? Are quality critics not being read because of the proliferation of scores? Are talented game creators losing their jobs because of review scores? If yes to any of those questions, then would the abolition of scores remedy those situations? If not, I see no more reason for Giant Bomb and IGN to ditch scores than I see them needing to have their reviewers append to their reviews drawings of whatever flower the game they just played makes them think of. Whatever info the readers find useful and edifying, you know?

To the score haters, though, I direct you to Kotaku's reviews for support to your arguments. The mad bloggers there found a way to write reviews that don't use scores but can still somehow be comprehended in the time it takes to tie one's shoes. See their Far Cry 2 review, and, aside from too many puns, it gets the job done.

***

John Davison, What They Play: Kieron's point that game reviews have to be a hybrid between purely subjective commentary and some stuff that's objective is an extremely important one. The innovation and creativity of games design (or lack thereof) is more than just artistic expression, and an assessment of the overall experience is often incomplete without some mention of the mechanics. Criticism of games reviews often focuses on the fact that we spend too much time on this stuff, but it is often a more important consideration than the "art". I think we'd all love to spend more time on digging into what a game is trying to "say"--but we're often faced with impenetrable control mechanisms, or distracting technological problems that cannot go unmentioned. There are also still an awful lot of games, particularly in the current climate of "casual" style games on DS and Wii, that are practically *all* mechanics, and an intelligent discussion of how well they work is all that's needed.

Take a game like Prince of Persia, though. While there's much to be said about the art style, the way the narrative unfolds, and the co-dependence of the lead characters--a review of the "experience" cannot ignore the fact that the game has some nuts and bolts mechanical issues. It has a tendency to play itself, for a start. The controls are simplified to the point that it can be distracting to an experienced player, and that there are some unavoidable issues with the camera. They're boring topics compared to something that, say, focuses on an emotional response to the player's growing bond with Elika, but I think there's an expectation from our audiences for us to convey and assess the overall experience, not just the "heart."

Shawn, your point about the "homogeneous monolith" is a really important one. Times *have* changed since the days of magazines and websites being gaming's gatekeepers, and while we all tend to identify with our own outlets, and maintaining a sense of community that is distinct, we all have a part to play in the broader scheme of things. Given that Google is the window through which the world views the vast majority of content, there's less and less loyalty to specific outlets, and instead people are simply looking for opinions on specific topics. For this growing audience, a score is still important for exactly the reasons that Robert and Jeff have mentioned. It's shorthand. It's a hook. It's a way to get people to look at our reviews. We are all, after all, businesses. We need traffic. We need people to come to our sites and read our stuff.

N'Gai, you asked "Why would you write for people who won't or can't read an entire review?" I think that's oversimplifying. Often, our audiences (both the hardcore, and the not-so-hardcore) go trawling for review scores purely for validation of their taste. An abundance of high scores serves that purpose, and makes them feel vindicated whether it's because they've made a pre-order, or are simply fans of the franchise.

***

Robert Ashley, freelancer: I wouldn't want to separate "art" from game mechanics and interactive feel. Atmosphere and storytelling have been a fixation for games critics lately, but the connection between the player and the machine--the mysteriously engrossing feeling that makes people scrunch their faces up and open their mouths like zombies--is the real "art" of gaming. Technology and all its various failures play a huge role in that feeling. Anyone can tell when something isn't running smoothly, but I wouldn't call these observations entirely objective. Badly behaving cameras can drive some people insane. Other people not so much. The same thing applies to many common technical problems. The question is, did you have that great zen feeling of being inside and part of the game, or did technical problems and poor interface constantly break the spell? Making a game feel like Super Mario Galaxy isn't a simple question of technology and time. There's something special going on there.

Next: Tom Chick steps into the ring with a jeremiad on scoring. Leigh Alexander defends editors' involvement in assigning scores. Shawn Elliott points out the pitfalls in that practice. And much more. P.S. If you absolutely, positively can't wait to read the rest of Round 1, Shawn Elliott has posted the entire transcript--all 16,000 words of it--on his blog, here.