Objection: The Fault, Dear GameSetWatch, Is Not In Our Metacritics, But In Ourselves

re: Objection: The Fault, Dear GameSetWatch, Is Not In Our Metacritics, But In Ourselves

joeboy101 — Tue, 08 Jan 2008 14:15:22 GMT

N'Gai -

Honestly, this is a problem for any media being reviewed, be it music, movies, or games. And I do agree with you that taking the score out of its context cheapens and homogenizies the effort put in in providing quality reviews compare to crap, pointless reviews. But keep in mind that a dearth of compilation sites for consumer reviews is out there from Rotten Tomatoes to Consumer Reports, its not like Metacritic is doing anything new here.

What they do have a point on, is Reviewer's Bias. Similar to what you have when you get a FPS freak to review an RPG or a lover of classical music to review Kanye West's new album. It could be reviewed in a basic sense, but not to the same degree as others could. And the stilited perspective could cause a significant drop in score, soley because it belongs to a particular genre the reviewer doesn't care for.

That's why many of the popular Wii games are getting low scores, you got one person reviewing group games and that's what they are. Group and children's games. A 30 year old reviewer who made their bones with Half-Life and Baldur's Gate is going to be insulted by Carnival Games. But get an 8 year old with his parents and that game could be as good as gold. It seems like many games being reviewed for the Wii are getting low marks because of genre, not because they are poor games.

And as to reviewer score, though the scores cheapen and simplify a review, it is a poor reviewer who does not accurately portray their opinions with whatever scale they use to rate a game.

re: Objection: The Fault, Dear GameSetWatch, Is Not In Our Metacritics, But In Ourselves

Mitch Krpata — Tue, 08 Jan 2008 14:28:38 GMT

Those of us who write reviews are always going to appeal to people to read the text, man, just read it! It isn't going to happen. In my wildest dreams I've considering asking my editors to drop the scores entirely, but the truth is people really like quick and dirty ratings. Even if they do read the text, they still want that score, because it's a quick and easy reference.

And aggregate sites like Metacritic actually tend to be fairly accurate when comparing similar games. Consider an 88 for Ratchet and Clank Future and a 79 for Heavenly Sword. Sounds about right, doesn't it? Not everybody would agree, but enough people would that I think the usefulness is self-evident.

I am glad that you call out the consumer reports aspect of reviewing for what it is, which is a cheap cop-out. Nothing bugs me more than reading a review that trashes a game, and then goes for the deathbed repentance in the last paragraph by saying something like, "But fans of [some similar game] are sure to love this one!" It's a wishy-washy way to cover one's rear. I'd like to see more critics with the courage to condemn. Frankly, I'm suprised they've been willing to dump so heavily on these Wii minigames. Maybe it's because those aren't the games that get advertised on big sites, but that couldn't be...

Still, I would argue that it's not simply a matter of opinion that has driven down the aggregate scores for so many of these mini-game collections. It's a function of how game reviewers do their jobs. I don't know how it works for full-time reviewers at the big sites -- maybe they can more easily enlist some help -- but as a freelancer my process usually is this: Receive a game. Plow through as much as possible. File a review, usually in less than a week. There's not much space in there to get a group of people together who desperately want to play Mario Party 8. You could argue that this is unfair to these types of games, and I wouldn't disagree. I'm not sure what the solution would be, except perhaps not to review them at all. But that doesn't sound right either.

Mitch Krpata

http://insultswordfighting.blogspot.com

re: Objection: The Fault, Dear GameSetWatch, Is Not In Our Metacritics, But In Ourselves

SpaceShot — Tue, 08 Jan 2008 16:37:26 GMT

Unfortunately, Mom rushing around for a child's birthday or for Christmas has no idea what to make of a review. It's doubtful she'll even look for a score. She's just going to look for Pokemon, Mario, or Nintendo emblazoned on the front. "Wii Sports gets my kids off the couch!", she thinks, even though she is disappointed later when they are sitting on the couch playing it or abandon it for Metroid Prime or somthing.

The biggest disconnect in games on the Wii is that, in many cases, the target audience is not able to afford the games. Their parents buy the games. That is changing, and the Wii has made inroads amongst couples who want a party game machine when their friends come over.

Reviews are pretty pointless except for the gamer like myself who has taken the time to read them and consider the past known biases of the reviewer. "Ah yes, but he didn't like the same thing about Crackdown and I loved Crackdown. My 360 LiveEye shows me I played it for 108 hours, so I imagine I will like Assassin's Creed, as well."

That type of critical thinking...

re: Objection: The Fault, Dear GameSetWatch, Is Not In Our Metacritics, But In Ourselves

N'Gai Croal — Tue, 08 Jan 2008 19:14:54 GMT

@joeboy101: You're right that this afflicts many media, though older media--books, theater, dance, art exhibits, etc.--tend to go unscored by most of their reviewers. Where you may have a point--and review aggregation sites could stand to improve--is that reviewers' biases, or preferences, should be presented in an easy to digest form. The Movie Compatibility Test app on Facebook presents a good model for how sites like Metacritic could help their users understand how closely a particular reviewer's tastes--and even the entire aggregate's--matches their own.

@Mitch Krpata: I've never liked assigning scores, even when I was writing movie and music reviews for my college paper. As to the "accuracy" of Metacritic, my guess is that it does better at gauging how core gamers will feel about core games than casual gamers will feel about casual games because--surprise--the majority of recognized game reviewers are themselves core gamers. My beef is that the Metacritic average shouldn't be seen as a proxy for the review text. Some qualitative analysis is called for before they can indict core game reviewers for missing the boat, especially since reviews are inherently subjective.

@SpaceShot: You're right that moms (and dads) may not have as much time as they'd like to do their homework on game reviews. I see two solutions: One, Nintendo needs to let publishers offer downloadable demos of Wii games, as Microsoft and Sony do already. Two, motivated casual gamers should blog their own reviews. Core reviewers shouldn't have to change their tastes to suit a hypothetical casual or child gamer. Casuals and children should offer up their own reviews, which like-minded people can then seek out.

re: Objection: The Fault, Dear GameSetWatch, Is Not In Our Metacritics, But In Ourselves

ferricide — Wed, 09 Jan 2008 00:04:37 GMT

two things --

i'd guess that most parents buy the games there kids are asking for, not researching which games are better. i'd imagine the number one factor in which wii games sell well is marketing, followed maybe secondarily by how appealing it looks oh the shelf (to either kids or adults.)

another thing about core gamers reviewing non-core games ... they may be reviewing casual games but they're doing it for core gamers. the 1UP review of sonic & mario is relevant to its audience because that audience still likes sonic and mario, despite those games having massive success outside of the core market. so, naturally, the 1UP audience would be curious to see if the game were worth their time or simply a waste of it. seems relevant to me.

and i hate to sound like a crotchety old reviewer, but we also have to face up to the fact that for most people, mediocre is often good enough. the sad truth is that sonic & mario may well be totally mediocre. the truth may also be that kids find it fun enough to play for a long period of time. i know that i loved a lot of games that had deficits in their execution as a kid. they had something about them that made me love them anyway. but any reviewer who has been doing this long enough to know that games you're positive are middle of the road with no particularly redeeming feature can still engender love in some of your readers (who then write incredibly pissed-off emails in defense of the game.)

doesn't mean you should stop being critical; just means that's how it is.

as was pointed out by simon, i believe, in the original interchange, film critics' best picks and the box office charts bear very little resemblence to each other...

re: Objection: The Fault, Dear GameSetWatch, Is Not In Our Metacritics, But In Ourselves

Pyrmont — Wed, 09 Jan 2008 15:01:33 GMT

I don't see the link in N'Gai's piece, but for those that are interested the Game Set Watch entry being referred to is at http://www.gamesetwatch.com/2008/01/gamesetchat_how_do_wii_judge_f.php. Oh, and while I'm being the link bunny here's Simon's reply: http://www.gamesetwatch.com/2008/01/opinion_casual_games_reviews.php.

re: Objection: The Fault, Dear GameSetWatch, Is Not In Our Metacritics, But In Ourselves

SpaceShot — Wed, 09 Jan 2008 18:35:52 GMT

Good point, N'Gai regarding mediums that aren't scored. Movies are scored by some reviewers but the best review for any movie is the "#1 Movie In America" after opening weekend. Games seem to rely on that score, and worse now the metacritic score. It may be a function of more gamers being computer-savvy and therefore web-savvy and therefore attracted to developing a mashup that combines all these scores into the most meaningless score we can possibly derive. :)

re: Objection: The Fault, Dear GameSetWatch, Is Not In Our Metacritics, But In Ourselves

SuperEffective — Wed, 09 Jan 2008 19:54:03 GMT

Consider: if game of the year 2007 is a meaningful exercise for the casual consumer (meaning they pick up Time magazine for a list of sure-fire purchases that they can't go wrong with, which is exactly what they'd find for books or movies), why did Peggle not make anyone's list? Why is Peggle not even reviewed by the major sites?

Metacritic only exposes the problem: reviewers are gamers, and the gaming commentariat seems to have developed a horribly condescending attitude towards simple games that anyone can pick up. They also give very short shrift to sports games (for jocks!) and Nintendogs (for girls!).

The discussion is so narrow that it feels a little claustrophobic at times.

re: Objection: The Fault, Dear GameSetWatch, Is Not In Our Metacritics, But In Ourselves

TheHangdMan — Thu, 10 Jan 2008 03:13:58 GMT

@ SuperEffective

"Metacritic only exposes the problem: reviewers are gamers, and the gaming commentariat seems to have developed a horribly condescending attitude towards simple games that anyone can pick up. They also give very short shrift to sports games (for jocks!) and Nintendogs (for girls!)."

One name comes to mind here (one that pops up with some frequency on this blog) , Roger Ebert. If anything, this disconnect between the reviewers and the buying public means we are finally approaching the same ground as movies and TV.

There are plenty of movies and tv shows that are critically reviewed, and yet flops with the general public. Shoot, it's been happening with games arlready (think Psychonauts). There is a disconnect because we are gamers. Most reviews are written by gamers. The people playing 'Peggle' aren't really gamers (like you or me), they're just people playing games. I guess what I'm trying to differentiate is the fact that most of the "hardcore" gamers out there today along with their reviewers are like the movie connoisseurs and movie critics today, (with all the snobbery implied). This is a good thing.

Now you maybe asking yourself "What do you mean Twelve? How is this a good thing?" It means that much in the manner that not everyone agrees with Ebert's analysis and methods of analyzing movies, someone else comes along to suit their needs (like the Movie Mom). It's capitalism at work (or at least it will be), if a reviewer tends to be biased against your genre of preference, you find one that takes a more objective look at your genre, or one who is biased in favor of it. You may not see them yet, but I'm pretty sure there are or will be causal game reviews by casual gamers, which better reflect the product.

In the meantime, I (from my high and mighty Throne of Judgement +3) shall continue to treat casual games like the Adam Sandler/Ben Stiller movies they are.

Cheers,

from the least understood card in the deck.

re: Objection: The Fault, Dear GameSetWatch, Is Not In Our Metacritics, But In Ourselves

SuperEffective — Thu, 10 Jan 2008 15:24:34 GMT

Hanged Man: the critic-audience disparity in games is *much* wider than with books or movies. You are pretty much guaranteed to wring some enjoyment out of the NYT's 'best of 2007' list if you go out to 'There will be blood' or pick up 'Out Stealing Horses'. You don't need special training to enjoy these. They may skew a little highbrow, but I'm convinced that the average person isn't going to be disappointed if they follow the reviewers' advice.

But Bioshock? It's a great game in many ways, but 99% of America is going to have a better time with Peggle. By almost every definition you can think of, that makes Peggle a much better game. Ideally, both are making a top 5 list in some way, but the exclusion of the title that most people are going to prefer really tells you a lot about where we stand today.

re: Objection: The Fault, Dear GameSetWatch, Is Not In Our Metacritics, But In Ourselves

trip1ex — Thu, 10 Jan 2008 19:19:35 GMT

REviews aren't perfect. Never will be. The best advice for games like MP8 and Carnival GAmes is to ignore the reviews.

And really go read reviews on Amazon.com for those games instead to get a more mainstream opinion.

re: Objection: The Fault, Dear GameSetWatch, Is Not In Our Metacritics, But In Ourselves

TheHangdMan — Fri, 11 Jan 2008 03:30:57 GMT

@SuperEffective

Thank you for bringing up "There Will be Blood". I'll stay away from the book lists, which I'm sure has some lovely reads, but I tend to stay away from writing in general (whole different topic of discussion) Now, back to TWBB, an excellent movie in my opinion, but let's take a quick look at the box office totals for a moment. A quick look at http://movies.yahoo.com/mv/boxoffice/ will show that TWBB is way back in 18th place, behind movies such as 'Alvin and the Chipmunks', 'P.S. I Love You' and 'Aliens vs. Predator - Requiem'. Just because these movies are ahead in popularity (I.e. money spent) doesn't make them betterm movies. I seriourly doubt that TWBB will ever approach the amount of money that the chipmunk movie will ever make, but that is very much besides the point.

Causal games are very similar to movies like Alvin and One Missed Call, they are things you can devote little time and effort to and give you a couple hours of entertainment in return. That's great. I'm glad that we have little time wasters like that, it's why I signed up for Kongregate in the first place. But..... and this is quite a big but, these movies and games are not superior to their *ahem* as you said 'highbrow' counterparts like 'No Country for Old Men', 'There Will Be Blood', or 'Bioshock' and 'Portal'.

One other note that people should pay attention to is the game rating sliding scale of doom. It seems that if a game gets under a 9 or 90% it is complete and utter crap. This is more of a mindset of the readers than the reviewers themselves. The 56% and 62% ratings of Carnival Games and Mario Party 8 are more akin to two/two and a half star ratings than honest to god flunkings.

When the chips are all down and the votes are counted, I'm pretty sure that Peggle will not be on anyone's top 5 must play games of all time, and National Treasure won't be picking up any awards anytime soon either. (and Razzies don't count)

Until later mi amigos,

hang in there.

re: Objection: The Fault, Dear GameSetWatch, Is Not In Our Metacritics, But In Ourselves

eurotransient — Fri, 11 Jan 2008 17:04:14 GMT

It always seems like I jump into these great discussions far too late, but what the heck, consider my hat thrown.

I have to say that comparing a particular level of quality in a film to its analogous level of quality game is a slippery slope. The problem lies in that while we all tend to look at box office income for films as a definite measure of how a movie is accepted by mainstream audience, I think that misses the basic point. With film, over time we are not limited by the way we choose to engage the product. We might miss it in the movie theater, but catch it on DVD. We might skip the DVD and pull it up on demand or catch it on HBO when it debuts there. The point is that despite the popularity of a film at the time of its release, it has a life that extends beyond that by the very fact that it exists in a medium that is available in an almost constant stream, whether you want it there or not.

Games, on the other hand, require a level of commitment. A console must be purchased; games must be purchased. Compare the relative costs of experiencing "There Will Be Blood" and experiencing "Bioshock". For the average, non-gaming consumer, this immediately precludes them from engaging in the "Bioshock" experience. I'm not sure what, if any, solution can exist for this. Even if you want to look less "hardcore" examples, the barrier is still greater to experience "Carnival Games" than it is to experience "Alvin & The Chipmunks".

The Wii does change things, somewhat, in that it suddenly opens up the potential buying public to a much larger degree. Nonetheless, just like the average person going to the theater to see "Alvin & The Chipmunks" isn't particularly interested in how well reviewed the film is, this new demographic of Wii owners, at some point, is not going to be particularly interested in what the gaming press thinks of "Carnival Games". Maybe at this very moment, given how new this expanded gaming demographic is, there's a greater sense of trepidation for, say, your grandmother to figure out what she wants to play. Ultimately, this feeling of uncertainty will wear off and your grandmother is going to realize that asking her grandson (who has logged roughly 7,000 hours online in Halo 3) what he thinks of "Carnival Games 3" makes about as much sense as asking Fred Phelps what he thinks of the movie "Shortbus".