An exclusive gaming industry community targeted
to, and designed for Professionals, Businesses
and Students in the sectors and industries
of Gaming, New Media and the Web, all closely
related with it's Business and Industry.
A Rich content driven service including articles,
contributed discussion, news, reviews, networking, downloads,
and debate.
We strive to cater for cultural influencers,
technology decision makers, early adopters and business leaders in the gaming industry.
A medium to share your or contribute your ideas,
experiences, questions and point of view or network
with other colleagues here at iVirtua Community.
It’s fair to say that the founders of Metacritic never foresawit generating the attention it has attracted. Intended as a way ofseeing at a glance whether a game was worth buying, it’s now used asa measure of game quality by the largest publishers, developers andretailers.
John Riccitiello has used its scores todefine EA’s business strategy to analysts; Steam prominently displaysthem on its product pages; developer Frontier uses them for salesforecasting.
And this simple set of numbers is deemedresponsible for many industry ills, from over-examination of reviewscores to influencing developer royalties. “I’ve heard thatpublishers will try to put a step in royalty levels depending onMetacritic scores, or some sort of Metacritic-related compensationstructure to a deal,” says Andy Eades, development director atRelentless.
Metacritic is still edited by just one man,Marc Doyle. But his focus remains very much on the reason why it wasestablished in the first place. “I really see myself as a kind ofgatekeeper to tell people that these are the games you should be payingattention to,” he declares. His role is to gather scores andcomments for every game released in the US, choosing whichpublications are included and concocting the formula that combinesthem into a single number.
A night owl, he works into thesmall hours from his office in Los Angeles. And though it’s nowowned by CNET, Metacritic is still his baby, Doyle co-founding itafter studying at USC Law School. There he met Jason Dietz,who came up with the concept and name in 1999. They launched itin 2001 and sold it in 2005.
Metacritic isn’t the onlyinternet game review score aggregator. The other major site is GameRankings, also owned by CNET. Doyle and GR’s editor, Lee Alessi,“talk to each other,” but have different methodologies. GR’sscores are based on averages, while Metacritic weighs publications’scores differently, depending on Doyle’s opinion of their prestigeand quality. But he won’t reveal how.
Both work on the sameprinciple, however: consistently include enough reviews fromenough publications across enough games and the results will smoothout. “A big game – one of the GTAs – I know Edge is going toreview it, and I know an easy grader will too, and so the biggames will get the same treatment,” Doyle explains. “If I includeall your reviews and all theirs, it all works out.”
Certainly,viewed broadly, the games at the top of the scale are generally thebest games around, and the bottom ones certainly aren’t. Thechallenge for Doyle [pictured above] – and the main source ofcontroversy – is in his selection of publications. The originalbasis was: “Who is the most credible, who has the best reputation,the best analysis?”
But now, he says: “It’s essentiallyabout whether gamers are going to them because they’re reliable foradvice on what games they should buy. I really don’t have to do toomuch research because they just come to me. I check out their scoringmethodology, send out a questionnaire asking when they launched,how many reviews they cover a week, total reviews done, aboutstaff – all the things I’ve learned over the years that I have todo.”
Clearly, much credence is placed on metascores, buttheir use as a metric for business decisions also depends on whetherthere really is a causal relationship between scores and sales.“There’s anecdotal evidence both ways,” says Doyle. “I know thatcertain publishers have done very comprehensive studies and they’vebeen able to highlight certain types of games and certain types ofgenres for which predictability will be much higher – racing,sports and certain types of action games, certain types offranchises. Others you just don’t know, like why did the Ben 10game sell through the roof? I don’t know. It’s not sopredictable, it’s not scientific or perfect.”
Activisionhas made such studies. Executive VP of publishing Robin Kaminsky saidat the 2008 DICE conference that higher-quality games, based onscores from Game Rankings, on average sell more, and that for everyfive points above 80, on average, sales double. But she notedthat many games buck this trend, and that the largest publishershave found that the greatest sales growth tends to occur in gamesscoring in the region of 70 compared to those scoring 80 ormore.
She also presented 18 products achieving scores of90 or more in 2008 and 2007. Only two were projected to sellover seven million copies, while seven sold less than a million.Overall, 12 out of the 18 sold less than two million, afigure that marks a rough break-even point for a triple-A game. Inother words, there is a correlation but quality does not assuresuccess.
And yet Metacritic is still gaining in stature, apoint illustrated by the fact that Doyle is receiving increasingcorrespondence from publishers. “If I’m missing something,publishers contact me and ask whether there’s a bias or a systematicissue – ‘Why are you covering this publication and not another?’”
Indeed,many PRs are strongly affected by a greater use of metascores aswell. Certainly, we’ve spoken with a PR for a major publisher whoexpressed huge frustration with the stress their companies place ongetting the best they can. “PRs who haven’t been in the industry verylong will get angry when certain new publications that I know haven’tearned their reputation aren’t included, or some local daily paper,”says Doyle.
“I say, ‘Guys, they haven’t made it yet’ –I try to be as kind as I can because I know sometimes they haveclauses in their contracts that make them affected by metascores interms of bonuses or penalties.”
The practice could be subtlychanging the relationship between developers and publishers, too.“You really want a producer to focus on doing everything that’s rightfor the game, not to be focused unduly with the review score it’sgoing to achieve,” says Splash Damage’s Paul Wedgwood [above].
“Ifhis bonus is wound up at a score of 70 or 80, he might betempted to err on the side of caution rather than taking risks andpushing for an 85 or a 90. Look at projects like BioShock,for example – on paper that isn’t something any sane producer wouldtake on, but it’s obviously well justified by its review scores.”
Furtherto this, many developers of games for broader markets feel thatmetascores are unrepresentative of their work. “If you look at familygames and kids’ games, they consistently score as many as tenreview points lower for, dare I say, what’s similar quality,”says Frontier’s David Braben. “It’s actually really hard to make areally
good kids’ game.”
For Wedgwood, developers canbe more directly affected by a poor metascore. “The negative sideis if developers are penalised for achieving low scores despite nothaving control over the resources and schedule for the project.”
Doylethinks so too: “If they’re having to achieve a certain metascore withthe same budget, that’s disturbing.” But it depends on therelationship between developer and publisher as well, as Wedgwoodsays: “Obviously, if the developer is wholly or partially funding agame or has a strong relationship with the publisher and can determinewhen it’s going to be released and how much it’s going to cost tomake, it’s their responsibility. And I think in that situationit’s quite common for a publisher to have an expectation for quality.”
Indeed,Wedgwood is a proponent of the idea that publishers should offer abonus related to earning certain metascores: “I think that reallyshows confidence from the publisher, saying ‘irrespective ofwhether or not this is a commercial success we’re going to pay you abonus just for achieving a certain review score’ – that’s a realincentive.”
But Braben [above] argues that developers have hadincentives all along. “Think of sales as a great big glorifiedMetacritic,” says Braben. “There’s been a lot of earnest talk aboutusing Metacritic and Game Rankings to incentivise, but the one reallyobvious way of incentivising things is royalties. EA has giveninterviews in which it mentioned average Metacritic scores as being ametric of the quality of its games. The problem is, why is itquoting that and not sales success?”
Eades agrees,reminding us that game companies are businesses: “There’s no pointin getting nine out of ten, ten out of ten and then not sellingenough products to justify a sequel.”
Which brings us back to the fact that Metacritic was only ever meant to be a general
guideto what to play for a game-buying public. For as long as everyone inthe videogame industry remembers that at the heart of it are one man’sdecisions, it could have value as just one of many other ways ofmeasuring a game’s overall success.
And perhaps, among allthe fears that a new focus on quality by publishers has emerged, thisis a change for the better. Wedgwood certainly agrees: “Wouldn’teverybody rather be working for a publisher that’s more concerned aboutquality?
Digg Topic
Tag topic on On del.icio.us
Technorati Search
Post to Slashdot
