Are
reviews primarily a consumer guide, or should they serve another
purpose? Do review scores deter intelligent discussion of videogames?
Is the presence or absence of a review score the only difference
between a reviewer and a critic? What is the role of the reviewer when
the Internet is democratizing published opinion? How should reviews and
reviewers evolve in light of the emergence and growth of Flash games,
small games, indie games and user-generated games?
These
questions
and more were on the mind of N'Gai Croal, John Davison and
Shawn Elliott last summer when they decided to expand their
conversation to a number of noted reviewers, writers, bloggers and
reporters for a published email symposium on game reviews. (See below
for the full list of participants.) The planned list of topics include
Review Scores; Review Policy, Practice and Ethics; Reader Backlash;
Reviews in the Age of Social media; Reviews in the Mainstream Media;
Casual, Indie, and User-Generated Games; Reviews vs. Criticism; and
Evolving the Review.
The
topic for Round 1, which will be published here in installments over
the next several days, is Review Scores. Last week we published Part I and Part II; today we continue with Part III.
Participants
- Leigh Alexander, Gamasutra/Sexy Videogameland/Variety
- Harry Allen, Media Assassin
- Robert Ashley, freelancer
- Tom Chick, freelancer
- N'Gai Croal, Level Up/Newsweek
- John Davison, What They Play
- Shawn Elliott, 2K Boston
- Jeff Gerstmann, Giant Bomb
- Kieron Gillen, Rock, Paper, Shotgun
- Dan Hsu, Sore Thumbs Blog
- Francesca Reyes, Official Xbox Magazine
- Stephen Totilo, MTV News
***
Tom Chick, freelancer:
On the subject of review scores and expectations, I have a lot on my
mind when I review a game. I'm crammed full of preconceptions,
expectations, prejudices, hopes, and fears. I call it "context" and
it's probably the single most valuable thing I have to offer as a guy
who writes about games. I've got thirty years of it under my belt. I
don't let loose of it when I play a game, and I certainly don't let
loose of it when I write about a game. I write as a hobbyist, and I
write mostly for other hobbyists. We know how Spore was hyped, we're
aware that Haze was from the Timesplitters guys, we've heard about the
problems with Killzone, and we remember Trespasser. It's how we talk.
When I have my druthers (i.e. when I'm not writing for a wider
audience), it's exactly the sort of stuff that goes into what I write.
And, yeah, it figures into whatever number or letter I have to slap
onto a review. Like any gamer worth his salt, I have the bias of
experience.
As for how much and when I think about scores, I
think about them all the time. I think about how much I hate them and
how much damage they do to the state of videogame discourse. Scores are
an end run around saying anything meaningful. I hate when someone says
(almost always on the internet), "I liked your review, but I would have
given it an 8 instead of a 7". Because that's an unborn conversation
that will never happen. If I didn't have to come up with that
insufferable 7, the comment would have had to go as follows: "I liked
your review, but I disagree with what you said about it being too hard"
or "I liked your review, but I disagree with what you said about the
graphics being too much like Fable" or "I liked your review but I
disagree with what you said about the ending feeling out of place" or
even "I liked your review, but I liked the game more than you did".
Those are all starters for at least a line of thought and at best a
conversation, and in either instance, we can both be the wiser for it.
Review
scores are for the lazy, the unengaged, and the inarticulate. They're
for stickers on boxes and press releases. They're understood
differently by different people, and they're applied differently by
different publications. They're an attempt to inject some sort of
science into someplace it doesn't belong and the sad irony is that they
mean nothing. I don't know if games are art, but so long as we're
branding numbers into their flanks, they're certainly consumer products.
Now
I have the luxury of saying all this, because unlike some of you (well,
John and Dan, at one point), I don't have to run a magazine or website.
To folks dealing with lazy, unengaged, and inarticulate readers, I
don't envy you your job selling stuff to them.
***
N'Gai Croal, Level Up/Newsweek:
Harry, you came up as a journalist and critic alongside hip-hop. You
wrote for The Source, with its famous one-to-five-mics rating system
and all the arguments--inside the magazine and among its readers--that
were stemmed from its reviews. You also worked for the Village Voice,
an outlet that didn't apply scores to any of its reviews--with the
notable exception of Robert Christgau's monthly Consumer Guide, which
introduced letter grades to music criticism--yet fueled legendary
debates nonetheless. You worked for Rockstar Games. You're even writing
a book about architecture in computer and videogames. (I bow.)
What
do you make of all of this? There's nothing new under the sun, so you
must have been in or around discussions like the one we're having right
here--does it bring back any memories you can share? Are Jeff and
Francesca correct when they say that scores are a legitimate part of
consumer reviews, regardless of how some readers may respond to them?
Are Tom and I right to argue that review scores help engender the
all-too-often juvenile discourse that surrounds videogames? Is Stephen
onto something when he suggests that the case against scores is
tenuous, and that as long as some readers find scores edifying,
reviewers who choose to score games shouldn't agonize terribly over
doing so? Is John right that Rolling Stone's editors, not its writers,
assign the star ratings? (I had to try, man.) And finally, what's it
like being on the receiving end, watching scores and reviews trickle
in, and--tell the truth, now--which mattered more?
***
Shawn Elliott, 2K Boston:
Before we finish this section with a final question on the real
consequences that review scores can carry (an elliptical response to
Stephen's repeated "What's the problem?"), I want to take a moment to
address any unanswered inquiries. Note that, for the moment, I'm
withholding my thoughts on the intersection of mechanics, meaning, and
the stories that games try to tell (the short, unsubstantiated version
is that I agree with Robert that it's largely fruitless to look at
these in isolation).
Kieron commented that he thinks
"AAA-popular-sequels tend to start with 9/10 and lose marks, while
games with less expectations start with 5/10 and have to gain them."
This strikes me as especially true for enthusiasts. Anecdotally, it
also seems as though critics are more inclined to take the gloves off
with less-anticipated, lightly marketed games or, conversely, to
forgive their faults. In addition, Leigh wondered about the wisdom in
bringing an awareness of a game's budget to our analyses. Thoughts?
And
then there's John's Rolling Stone rumor. What do we make of a critic
submitting copy and his or her editor supplying the score? It's an
interesting thought experiment. I do think that the policy would
encourage writers to explain themselves more carefully, and that it
might erase much of the discrepancy in words and numbers, but what
ramifications would it carry?
***
Jeff Gerstmann, Giant Bomb:
I find Kieron's comment about big budget games catching a bit of a
break in reviews to be possible in isolated cases, but I don't think
it's the norm at most publications. I've also seen the opposite, where
the big budget game gets trashed for not living up to the
insurmountable mountain of marketing hype while the low-budget indie
darling catches a break because it was made by a team of five people or
something. And I've certainly seen little, unmarketed games get
absolutely thrashed in reviews. Sometimes it seems like the reviewer is
doing this because it's "safe" to do so, like it was part of some sort
of "see? We totally use our entire 1-10 scale" chest pounding. This is
why it's important for reviewers to have an editor (or editors) that
can keep them in check and ask questions about a review and its score
before it gets published, especially at outlets where that one review
is meant to represent the entire publication's view on a game.
While
I've never been a party to anything quite like this Rolling Stone
rumor, I've spent a great deal of time as an assigning editor for
reviews. The most important part of that position is working with the
authors on clarity, to make sure they actually mean what they say, and
that they aren't coming off as more positive or negative than they
intend. Sometimes that involves changing the score to make it match the
words more closely. This would happen most often with freelancers, as
they can't really be expected to be experts of how one publication's
scoring system differs from another. But except in extremely rare cases
where the reviewer wasn't available at post time, those changes were
made after discussing it both with the internal staff (as part of a
review vetting process) and with the author. I wouldn't want to do
things the way Rolling Stone supposedly does them, but that might say
more about my faith in most freelancers than it does about the policy
itself.
Lastly, I really don't think a game's budget matters when
reviewing a game. High-budget console games and mid-budget console
games cost the same $60. The only dollar amount that matters is the
retail price. While bad games certainly don't get dramatically better
as their retail price drops, it's a lot easier to overlook some of a
game's flaws if you're getting it for $20 instead of $60. But at the
same time, it's probably fine to mention a game's budget in passing.
There's a big difference between a brief mention of the budget and
using it as the centerpiece of your entire article.
***
Leigh Alexander, Gamasutra/Sexy Videogameland/Variety:
I'm super eager for Shawn's discussion point on the impact of the
scores, and I don't want to drag everyone down a side trail, but wanted
to note something on the Rolling Stone rumor.
No idea what they
do at Rolling Stone, but I've actually been in situations more than
once where the final star ranking or number was suggested by my editor.
It's not as sinister as one might imagine--it isn't as if a person who
didn't play the game is independently applying a number with no input
from me. In fact, it's more of a collaborative review process between
the editor and the writer to be sure that the final rating really does
correlate to the text as it's written. Limits in what you can express
within a word count, which I'm sure Rolling Stone is constrained by,
can make it useful to have two pairs of eyes on the situation.
In
fact--and I think this is especially true for those who write for more
mainstream print--many reviewers in that context want to avoid the
wham-boom core market number hysteria. They want to write articles,
they prefer to write crit, and don't want to calculate numerics. They
may be non-traditional reviewers (as I'd assume Rolling Stone's are).
In that case, especially where Metacritic is involved, in my experience
an editor may volunteer to apply a score that's correlative with the
review text simply to offer an option for those writers who don't want
anything to do with the numbers game.
In those cases I have
always elected to have input because I feel comfortable that way--but
keep in mind that scores can cause headaches and PR arguments for
reviewers, and in that case, an editor stepping up and saying "point
them to me if they hate the number" may be one way of allowing the
writer to do his or her own most honest work with impunity. When an
editor plays a role in the score, he or she's essentially "backing" the
writer's text by shouldering all those burdens a number can provoke.
It's a pleasant reversal from the nightmare scenarios we hear about
where writers catch flack from their bosses or even get canned because
some publisher flipped their lid.
Again, I don't even know anyone
who writes for Rolling Stone. I wish I did, because I'd tell them to
write about good music again. But my suspicion is that scenario doesn't
so much involve some sinister overlord stealing scoring power from the
reviewer, rather an editor who wants to make room for the writer to do
what the writer's being paid to do--write well and thoroughly on a
title.
***
Shawn Elliott, 2K Boston: The editor in
me empathizes with Jeff, especially in cases where a green writer's
copy is in apparent conflict with the rating he or she attaches to it.
Discussion is helpful here, as the writer may reveal anecdotes and a
level of analysis lacking in their initial draft. How many of us
remember the teachers who told us they wanted to read in our revisions
whatever it was we'd just said?
The writer in me, however, is
just as wary of knee-jerk editing, and editors who've played -- and
have formed their own opinions of -- the games they've assigned to
freelancers. Here, for example, we find text apparently taken from PC Gamer UK's Call of Duty 4 review
transported to GamesRadar's 360 review page where a presumably miserly
British 8.5 becomes a prodigal American 10. Bylines differ, too, as
though we're to believe that two writers, an ocean apart, have arrived
at the same sequence of words and altogether different assessments of
their meaning (the editor's note states that the GamesRadar score was
once a 9, which still doesn't explain discrepancy). That Francis's copy
came first is fairly obvious in lines like, "If the whole game had been
like that, or even just as inventive throughout, you'd find a frankly
silly score at the end of this review. Instead it's a more restrained
one[....]" My suspicion is that a comatose editor couldn't be bothered
to read let alone edit the original. Frankly, that's a *** insult
to writers and readers if there ever was one. It says, 'Hey, fill some
space for the cretins naive enough to not notice. We'll handle the hard
work."
Hopefully, this is an anomaly. The idea of dismissing the
Rolling Stone idea for it is as dumb as it would be to eliminate
governorship because of Blagojevich, but, again, we're all human.
***
N'Gai Croal, Level Up/Newsweek: Regarding
the Rolling Stone rumor, I'm think it's legitimate for reviewers and
editors to discuss scores before assigning them, but I'm wary of
editors assigning scores all by their lonesome. This point was driven
home for me when, in the wake of Jeff's firing from GameSpot following his Kane & Lynch review, Tor Thorson issued a statement
explaining that "The copy was adjusted several days following its
publication so that it better meshed with its score, which remained
unchanged." Even given the force majeure of this situation, the
question that immediately popped into my head was this: if changes had
to be made at all, why wasn't the score changed to better mesh with the
copy? If there are going to be scores, the individual reviewer should
always be intimately involved in that process.
Next: Shoe
tees off on exclusive reviews. Stephen responds with Just Say No.
Kieron explains why exclusive reviews matter tremendously in the U.K.
And Harry makes his long-awaited debut. P.S. If you
absolutely, positively
can't wait to read the rest of Round 1, Shawn Elliott has posted the
entire transcript--all 16,000 words of it--on his blog, here.
