Metacritic is all WET: Just Sayin' Edition
Last week I bought WET because it looked like fun and I wanted to play the game. After playing half way through the game I found it didn't only look like fun, but it delivered on its promise. It was what I wanted Stranglehold to be and what Gungrave never delivered. Lot's of mindless, shooting fun. Sometimes that's just what I want in a game. But apparently, most critics did not share my view. When I first looked on metacritic the score was in the deadly sixties, but has since moved up into the safe haven of mediocrity found in the seventies. Not quite green banded goodness, but not bearing the red mark of humiliation. I wish metacritic didn't matter, but unfortunately, developers' livelihood is based on this hopelessly useless, conflicted, arbitrary measurement system even though more and more and more people are realizing marketing and word of mouth are more significant factors in the purchasing decision than a Metacritic score. If you really want to compare apples to apples in admittedly anecdotal but still compelling example, Rock Band 2 for the 360 scored a 92 to Guitar Hero World Tour's 85 last year. But Guitar Hero, with a 40% larger marketing spend, outsold Rock Band 2, the placebo, by a wide margin. Publishers still use these numbers as gating to signing developers and not only are they useless, they are shoddily calculated numbers based on arbitrarily assembled numbers.
Out of curiosity, I decided to see who was responsible for raising the score and what they had to say about the game. They were courageously disagreeing with gaming stalwarts like IGN at 66, Gametrailers at 63 and sponsor and influence free Giant Bomb at 60. It looked like it was Armchairempire.com. I say looked like because they were noted as giving the highest score the game:
But when I clicked through to the review, I saw they didn't:
Now, I am sure this is a careless error, but how dare you be so careless when developers' livelihoods are at stake. Didn't anyone at metacritic feel the need to confirm the numbers posted on the site, or is the move from 75 to 88, part of metacritic's "weighted average" calculation, described as:
The METASCORE is considered a weighted average because we assign more significance, or weight, to some critics and publications than we do to others, based on the overall stature and quality of those critics and publications. In addition, for music and movies, we also normalize the resulting scores (akin to "grading on a curve" in college), which prevents scores from clumping together.
Metacritic does acknowledge scores are misreported and suggests a solution:
Q: Hey, I AM Manohla Dargis, and you said I gave the movie an 80, when really I gave it a 90. What gives?
A: Now, if you are indeed the critic who wrote the review, and disagree with one of our scores, please let us know and we'll change it.
This does happen from time to time, and many of the critics included on this site (such as Ms. Dargis) do indeed check their reviews (as well as those of their colleagues) on metacritic.com.
Are you serious? You can't be bothered to confirm you are accurately transcribing numbers and it is up to the critics to fact check?
But shoddy journalism is not my only concern. It is the sites' holding itself out as objective when it is really a conflict laden subjective aggregation of a limited set of already subjective market data. Now, of course the fact that CBS owns of Metacritic and Les Moonves, President and CEO of CBS Corporation is on the board of Zenimax, parent company of WET's publisher wouldn't influence metacritic to hunt for some favorable reviews and maybe fudge some of the weighting or even a number, but a purportedly objective site should not be in a position where it must explain why not. When I was in law school, they taught us to avoid impropriety, but the appearance of impropriety. Avoidance is simple. You disclose. In this regard, I renew my suggestion from an earlier post and I offer a disclaimer:
We are affiliated with a studio, record company, television networks and game companies, in fact we are better connected in entertainment than CAA and WME combined. We can probably get Les Moonves on a conference call. To give you a better idea, here is a partial list of our family members:
- CBS Television
- CBS Records
- MTV
- MTV Games
- Harmonix
- VH1
- Nickelodeon
- Paramount Pictures
- Paramount Television
- Paramount Digital Entertainment
- Dreamworks Animation
- BET
- Spike
and of course our distant cousin, Bethesda Softworks. We do our best to avoid influence from our parent and siblings, but the significant subjective component in our scores makes it kind of hard.
This has been a public service message. Thank you and goodnight.
Oh, and by the way, pick up WET. It really is a good game.
Comments
Metacritic does not list any game reviews that do not have scores attached, so the Manohla Dargis question isn't applicable. The mistake on the Armchair Empire review is unfortunate, but they are not asking critics to fact-check their scores.
You are absolutely right about the critic being from the New York Times, but I did not read the response the same way as you did. However, the same faq we both reference contradicts your statement about the use of reviews without scores:
"For each review found, we will take the score given by the critic and convert it to a 0-100 point scale. (For those critics who do not provide a score, we'll assign a score from 0-100 based on the general impression given by the review.). These individual critic scores are then averaged together to come up with an overall score."
Not sure what to tell you about the Dargis entry in the FAQ. Your interpretation is simply incorrect. If you read the entry above the question you cited, it's clear that the discussion is about reviews that are not scored in the original.
The other quote you cite does not contradict what I said at all. I said that Metacritic does not list any game reviews that do not have scores in the original source. This is a fact. They list film reviews, TV reviews, etc., that do not have scores, but they do not do the same for game reviews.
The only exception to this was the inclusion of reviews from Variety and a few from the New York Times, in which the reviewers did not publish numerical scores but did provide them directly to Metacritic. A little more info on that:
http://forums.metacritic.com/eve/forums/a/tpc/f/367108/m/2180059775
If you don't believe that Metacritic exclusively uses scored reviews in its game listings, I invite you to find some score-free reviews in the site's Games section. Happy hunting.
This is why your response to the Dargis entry in the FAQ was wrong -- because you are trying to compare processes between sections of the site that work differently. The Dargis stuff doesn't apply to games. It's about judgment calls in assigned Metacritic scores.
Metacritic doesn't make judgment calls with game scores. They simply screwed up on the Armchair Empire review. I'm sure if you emailed Marc Doyle or posted on the Metacritic forum, he would hasten to correct it.
Thank you again for your input and your post. However, again your cite supports the point of my post. On the forum, Mr. Doyle wrote (I added the emphasis):
In other sections of Metacritic, we regularly track publications that do not assign a score to their own reviews (LA Times, NY Times, Hollywood Reporter, Variety), and we've always estimated the scores based on the impression gleaned from the review. I've only very rarely taken this approach in the Games section of Metacritic because gamers and the games industry are so sensitive to our scoring system and process. However, I did track the New York Times game reviews for years - a site which had no scores. In that case, the lead critic from the Times, Charles Herold, emailed me his unpublished scores every time the NYT published a review and I used those numbers on Metacritic. (See the scoreless New York Times review of The Orange Box by Charles Herold which appears on Metacritic's Orange Box page with an 88 score) My argument is that if the score comes from the horse's mouth, from the critic him or herself, so that I know the critic's intent definitively, I am satisfied that I can maintain 100% accuracy. (Incidentally, we stopped tracking the NY Times when they laid off their veteran lead critic a few months back.)
He has used non scored reviews in games and the decision was completely subjective. The point of my post, and the ones I linked to, is metacritic is subjective an arbitrary. He decided at times to use the reviews, and at other times not to use the reviews. Why were they included when they were? Just in case we are not sure about the subjectivity, Mr. Doyle confirmed it when he further wrote in the same post you cited:
And a further note: I receive hundreds of requests from publications asking to be included in Metacritic's process every year. I take each of them seriously and evaluate them using a host of metrics and standards. I've been doing this since 1999, and I'd doubt that anyone has read many more game reviews than I have over the same period. I would suggest that I'm competent to make such decisions, and I am the last and only word on which publications make it to Metacritic.
I see the pages and pages of faq's about how it is measured, but tell me again what his credentials are to be the guy who "knows." Seems to me, the system might be a little better if there was a metacommittee determining the critical composition of metacritic. After all, doesn't the underlying reason for metacritic lie in the wisdom of crowds?