Game UI testing: How do you go about it?

JoePie

Hi all.

I'm working as QA for a games company in Dublin, and one of the reasons I was brought in was to look at bringing in an automated QA framework.

I'm in two minds about it as I'm coming from a traditional software background where automated testing using the likes of FitNesse or Selenium made perfect sense for GUI testing.

We're working on mobile games, primarily iOS, so that further enhances the problem. I've been looking at a lot of test driven dev tools like Cucumber (more specifically an off-shoot called Calabash which is set up for iOS), and Sikuli, which is a sort of half recorder/ half python scripted tool for UI testing.

Obviously, a lot of these tools will work fine with static apps, or even static-ish games like a Football Manager type or text based game, or something where a lot of input can be hardcoded.

Most games however are a lot more dynamic in their nature in terms of visuals, and as art assets get changed or new gameplay goes in, tests can be thrown way off, so maintenance requirements can be high, and obviously grow the more tests you add.

I'm trying out a couple of these different tools sort of "after the fact" if you get me. As in, on a near completed game, when ideally, these tools would have been embedded from the start of dev, but wasn't an option with this game since no one was looking at it and I didn't join until late on.

Obviously, we go through the standard manual regression stuff and the developers have a good suite of unit tests built up.

My opinion is that we don't need automated testing in this scenario. Obviously the upsides are code coverage, cut down on regression time, spot any technical issues early. But the downsides far outweigh them: visual layers, game balancing, behaviour testing, heuristic analysis, things like that. So it seems like we would be creating more work for ourselves in the long run.

So how does anyone else go about testing their games, if they do it at all?

EDIT:
I should note that I've spent a lot of time in the last few days reading lots of for/against posts and discussions around the internet and while I am in favour of automated testing, I just can't justify it in this particular case. For the UI I mean.

RedXIV

Hi JoePie,

I too have worked in Games QA and currently working outside it but would encompass a lot of UI QA in my current role. Automated testing is great for lots of aspects of testing but UI testing is NOT one of them. The point of UI testing is to experience the application as an individual, not a machine. Otherwise what you're doing is Load testing for lots of users, or End to End testing which indeed could be automated.

The User Interface testing isn't just to test the flow of the expected use cases, its to predict how users may respond to different aspects of the GUI and to see where potential improvements could be made. For this reason, UI testing is usually done near the very end of the development cycle as its not about fixing bugs, but more about the polish to go on the game.

Hopefully this helps a bit

JoePie

Certainly valid points.

Like, I've done plenty of UI automation in other jobs, but they were static business apps. Whereas game UI's seem like they would benefit the most from something like automated UI testing, but it's the last thing I would want to set it up for.

Think I'll mess around with a few more tools, just so I can strengthen my position of "THIS WON'T WORK" and if I find the holy grail, I'll be sure to share it. Don't hold your breath though.