The Critic and the Cloud

With GPS records, computer stats and timestamps in hand, they will eventually strike. They’ll comb through the complaints and the cries until they find a mismatch, and then they’ll issue a statement.

The critic never drove to that point, they’ll say. He did not traverse the highway mentioned in the third paragraph.

And he most certainly did not toggle the hardest difficulty setting.

What a terrifying back-and-forth between a New York Times car critic and the spurned CEO of Tesla Motors this week. As each side argues over a negative car review’s truths, the Internet watches with popcorn in hand, gobbling up the “gotcha” attempts while picking sides. But I’m not enjoying it.

All I can think is, God, what if that ever happened to me? And it very well could. Really, Tesla’s show of defiance against the enthusiast press has shown a lot of manufacturers’ hands. Product makers already have the data – mountains of it – for exactly this kind of pushback…perhaps none more than video game producers.

———

I am, among other things, a guy who gets paid to review games. Sweet gig, right? I swim in new titles, typically before they reach store shelves, and as far as other people can tell, all I do is describe how awesome my playtime was.

Far be it from me to complain about my line of work. It is indeed “awesome” (sub “bitchin’” or “righteous” as needed), and I have busted my ass for over a decade to call it a bona fide job. But sometimes, a game to review lights a bundle of dynamite. It might arrive barely a week before launch, especially if my outlet hasn’t been deemed a priority by a publisher, and yet I’m expected by editors to pound out a proper, top-to-bottom review within a brief “launch window.”

For perspective, consider film critics, who only get one viewing and a few days to toss their thoughts in the dryer; even that can feel like too little time to make sense of 94 minutes of Bradley Cooper. Comparatively, an average game requires 8 hours, at the very least. That number typically climbs, whether thanks to optional content, multiplayer testing or non-content disruptions like frustrating challenges or required grinds.

None of that should shock any reader of gaming site Unwinnable – nor should the fact that critics deal with those situations in wildly different ways. Those ways often speak to their critical voice.

In some cases, my style of review would be decimated by an angry, data-wielding CEO. Here’s an example: in February 2012, I reviewed quest game Kingdoms of Amalur: Reckoning without beating it.

Reckoning MapEA could very well have dug up my Xbox Live data and responded to my review with the following truths: I only unlocked 24% of the achievements, and I had a six-day gap between some of those achievements. Over a year since the game launched, I can’t even check to see how much I leveled my character or my total progress in the game, as I gave the game away weeks after my review ran. But by my recollection, I got through roughly half of the story content – and nearly all of the side quest content up to that point – at about 30 hours of play.

———

No great expose came and went; no blog post lifted my career and tossed it upon a decimated trailer park. Possibly because EA didn’t bother checking critics’ playthrough stats, or because my review was largely positive.

But what if someone attacked me with a data-driven gotcha? Never mind that I believe quest games should be consumed slowly, the way busy adults (my target reader and expected demographic at that outlet) play them, rather than gorged upon; never mind that I lost a lot of time to side quests, which, as I said in my review, Amalur practically forces its warriors to complete, lest they fall behind in the main questline; and most importantly, never mind that 30 hours made abundantly clear both the game’s action mechanics and its storytelling techniques.

Instead, my review would be bludgeoned with a “did not finish!” asterisk.

Truth be told, my personal bias tilted a little weaker than my review’s score, 8/10, may have illustrated. Which is why I hate numbers in reviews. Indeed, Amalur was an 8/10 experience: its huge, beautiful world exploded with a smooth combat system, satisfying upgrades and a more-than-serviceable plot. The game fulfilled the cost-to-content equation, and with polish. But badly-paced side missions interrupted the story’s flow, and the game’s strides had already been topped by better quest games only months earlier. Those are truths reflected in my descriptions and final judgments – and the reasons I didn’t really go back to the game after reviewing.

None of that nuance exists in the cloud, and I’m terrified that I will one day ever have to defend a review; I’ve always believed that my byline is defense enough, which is why I very rarely reply to comments. (Just to cover all of my bases: I fully stand behind my Amalur review, and in no way have I ever produced an insufficient, misleading review.) But rest assured, before long, a game critic’s feet will be put to the fire either by a publisher or a sleuthing fan. An achievement or a leaderboard score somehow won’t match the text, and the pitchforks will come out alongside accusations of an agenda.

Unless such controversy proves that a critic flat-out lied – for example, criticizing a specific moment without proof that he/she got that far – I’ll rush to his/her side with a giant shield. Critics deserve space to play games at a reasonable pace. To not reach the very end of a massive game (or unlock multiple endings), so long as the criticisms don’t hinge on its total narrative arc. To not always test the easiest or hardest difficulties whenever they prove irrelevant. Quite frankly, to quit playing a game early if its mechanics have been laid bare and its frustrations proven wholly debilitating. So long as those choices match the typical audience of an outlet, and are addressed by the text (which is tough to do with an itty bitty word count), then they’re valid.

This will only get worse as the world of games and software shifts to an always-online, cloud-dependent model. If the code lives on a server, so will the subjective experience of a pre-release critic. Game companies have already trumpeted what they can do with anonymous usage data, from pushing in-game ads to “perfecting” motion-control add-ons. Their mountains of data will only grow, and while the average fan may merely feel creeped out by the trend, the critic-publisher relationship will only grow further strained by it – unless game makers learn early on to let go.

It’s not just producers who should cool their heels on wielding data, Tesla-style. Magazine and website editors may soon also be tempted to peruse their critics’ achievement lists before printing a review. It’s not a far-fetched idea. Modern, online games, more than any media, leave a breadcrumb trail of their players’ progress, and fact-checkers never want to be caught with their pants down.

To them, I insist, stick to the critics’ words. A lousy critic will run full-speed through a game to reach what seems like the ending, ticking every achievement along the way. A real critic will drive a game the way his/her readers would – poking their heads in every corner, spending more energy on the moments that prove the most exciting and revealing — then leave the place behind once it’s been set on fire, 100% completion be damned.

———

You can find Sam Machkovech on Twitter, giant shield in hand, @SamRed.

Subscribe to Unwinnable Weekly
Categories:
Commentary, Gaming, Sam Machkovech, Technology
Unwinnable On The Web:
  • felix

    Not sure what's the point of this. Achievements are unimportant? Who cares? If I ever saw someone complain about stats and achievements, I knew he'd be just some idiot gamer. Why not write about interesting stuff (and maybe with some good hooks, because the point of this was really elusive and turned out very thin).

  • UnSubject

    You obviously missed (or didn't remember, or whatever) the storm around Eurogamer's review of Darkfall where the devs dug into server logs and claimed the reviewer only spent 2 hours playing.
    http://www.eurogamer.net/articles/editors-blog-da

    Less well known was the claim that Jim Sterling hadn't played through FFXII as much as he said he had:
    http://www.dualshockers.com/2010/03/17/what-exact

    So yes, the data exists to mine and call out those who review games.

  • Rob Haines

    I think the most salient lesson to come out of this whole Tesla controversy is that when you use data like this to try to prove a point, neither side comes out of it looking good. From a position of mostly ignorance about the whole debacle, my assumption is that the critic was probably trying to angle the story in a negative fashion, and that Tesla are presenting data in a manner which naturally biases towards their company interests.

    Both of which are equally dishonest in my eyes, and neither the NYT or Tesla are benefiting by having their names dragged through the mud in the fallout. Let game developers, publishers & critics in equal measure take note.

  • zpoc

    i dont think there's anything wrong with stating within the review or as a footnote 'i played xx amount of hours, and completed roughly xx% of the game which i felt gave me an adequate grasp of what the game has to offer' (much like you did here). some review outlets have mandates that the review see the ending of the game (which is not the same as 100 percenting a game), but if who you're writing for doesn't care, more power to you. if they DO, then it's your professional obligation to do so. and i'm not sure why you even bring up achievements – no one's ever cared whether a reviewer maxed out their gamerscore on a particular game, and no one ever will.

    and anyway, your imagined 'gotcha asterisk' isn't really the same as the tesla situation, unless you went out of your way to lie and say that you did in fact complete the game when you didn't. 30 hrs is a lot of time to invest, and being honest about that time should be an ok thing to do. my harshest reaction might be that perhaps you should think about what kind of games you'd bother reviewing at all – an epic 60 hr RPG is maybe something you just shouldn't review if your schedule/patience doesn't allow for it.

  • chen-chen

    Regarding "EA didn’t bother checking critics’ playthrough stats": you are severely overestimating how much access game makers have to individual players' info. It's not that they didn't bother: they can't. As much as it may seem like one big corporation sometimes, EA (or Activision or whoever) are not Microsoft (or Sony or Nintendo). Each company jealously guards the data it collects about its users.

  • http://www.cosmosonline.ru/?C=S;O=A cosmos game

    Great issues altogether, you just gained a emblem new reader. What might you suggest about your publish that you made a few days ago? Any certain?