AUTOMATED TESTING ROUNDTABLES

GDC 2024

The Automated Testing Roundtables are back again at GDC 2024!

Each session focuses on a different discipline, but cut across team sizes, budgets, genres, and platforms. We don't care if you're a iOS-only one-person indie or a console juggernaut, we just want to hear your voice!

Resources

Check out some of the talks around GDC 2024 that we recommend, and some of the vault talks we've enjoyed in the past. Henry Golding and friends also maintain a Trello of useful resources:

Automated Testing Resources

Community

If you'd like to keep in touch all year round, why not join our Discord?

Join the Discord!

Process Roundtable

Wednesday 10.30-11.30am, Room 308 South

Beyond the software that we use to test our games are the people that write the tests and the workflows they use to reinforce them. How does the process work in your studio? Are the engineers responsible for automated tests, or QA? Are QA silo'd or integrated? What could be better?

Legacy Roundtable

Thursday 4-5pm, Room 312 South

As much as we would like otherwise, automated testing is still a rarity in our industry. Do you have a culture or a codebase that doesn't currently embrace automated testing? Are you having trouble getting people onboard, or maybe it was very easy and you want to share why? Grand failures are just as interesting as successes!

Implementation Roundtable

Friday 11.30am-12.30pm, Room 312 South

Software automation has many faces, from locally-run functional and unit tests, to server-driven continuous integration, and even external device farms. How do you do automated testing in your team? What works well for you? More interestingly, what didn't work and why?


More automated testing talks at GDC 2024

Here's a selection of talks that we think might be interesting to Automated Testing Roundtable attendees. Add them to your schedule now!

If you've spotted any that you think would be interesting to roundtable attendees, let us know in the discord.

Testing Empowered: Integrating ML-Based Playtesting in a Team with Limited QA Capacity

Sun Jeong

All Access Pass, Core Pass

Wednesday, 11.30am-12pm, Room 2010 West

A guide to automated exploratory playtesting for small- to medium-sized teams

Automated Testing of Shader Code

Keith Stockdale

All Access Pass, Core Pass

Wednesday, 2-3pm, Room 3014 West

More details from the ubertesters at Rare, this time on shaders - always a difficult thing to test

Taming the Beast: Leveraging Machine Learning to Curb the Cost of Continuous Integration

Sarra Habchi

All Access Pass, Summits Pass

Tuesday, 2.10-2.40pm, Room 2010 West

Sarra promises to explain how you can use machine learning to shorten build times on your continuous integration server

QA Bots for Everyone

Christoffer Holmgård

All Access Pass, Core Pass, Summits Pass, Expo Pass, Audio Pass, Friday Only Expo Pass, Indie Games Summit Pass

Friday, 11.30am-12pm, Room 2024 West

A sponsored talk from modl.ai, but includes a deep dive into the bot implementation for Salt Sea Chronicles, a narrative game and therefore outside the comfort zone of tools like this. Will be interesting to see how they handle it.


Interesting talks to watch right now

Here are some that we've enjoyed in previous years. If you know one that we haven't got listed here, from any year, let us know by dropping by the discord.

Fixing Bugs by Cloning Them in 'The Last Clockwinder'

Matthew Blair, GDC 2023

Play on the Vault FREE!

Automated input replay can be a valuable tool in building some types of tests. This talk explains some practical steps to implement this in Unity.

TestMonkey: Automated Testing At Santa Monica Studio

Ben Hines, GDC 2023

Play on the Vault

A deep dive into Sony Santa Monica's testing framework, that can handle Visual Tests, Smoke Tests, SAN / ASAN Tests, Determinism, Gameplay, and Code Integration tests.

AI for Testing at EA: From 'Star Wars' to 'Apex' and Beyond

Jonas Gillberg, GDC 2023

Play on the Vault

Follow a simple testing AI for Battlefield V as it evolves into a useful tool for research and testing across EA

Engine Optimization Powered by Big Data

Vladimir Katchadourian, Vincent Leclercq, GDC 2023

Play on the Vault

If performance is a pillar of your game, why not track it with the same automated tools you use for testing? In this talk, Ubisoft explains their approach to aggregating and analysing performance data.

AI Summit: Where The $@*&% Are Your Tests?!

Kevin Dill, GDC 2021

Play on the Vault FREE!

Pracical lessons on unit testing AI from frequent AI Summit speaker Kevin Dill. This will be sure to have useful concrete takeaways.

AI Summit: Automated Game Testing Using a Numeric Domain Independent AI Planner

Bram Ridder, GDC 2021

Play on the Vault FREE!

This approach to testing with AI looks to be using more classical AI techniques than the usual machine learning approach. It'll be interesting to see how this compares to ML.

Lessons Learned in Adapting the 'Sea of Thieves' Automated Testing Methodology to 'Minecraft'

Henry Golding, GDC 2021

Play on the Vault FREE!

Sea of Thieves has been making waves(!) the last few years with its focus on automation and deployment. This talk on how similar approaches were tried at sister studio Mojang is sure to be insightful.

'Final Fantasy VII' Remake: Automating Quality Assurance and the Tools for the Future

Fabien Gravot, GDC 2021

Play on the Vault FREE!

Gravot will present a testing AI being used to explore FFIV Remake with greatly reduced setup costs compared to regular exploratory QA tests.

From 0-1000: A Test Driven Approach to Tools Development

David Paris, GDC 2020

Play on the Vault FREE!

How did Playground Games introduce a testing culture to an already-shipped game, and why? What did they learn, and how do they plan to improve?

Building An Intelligent Game Testing System in Netease MMORPG Game

Yingfeng Chen, Xiao Li, GDC 2020

Play on the Vault FREE!

This session will introduce new ML and AI techniques that have been used in the testing of the Netease MMORPG game - "Justice", which helps to build an intelligent testing system for three typical game testing scenarios: quests regression testing, class balancing testing and dungeons testing.

AI for Testing: The Development of Bots that Play 'Battlefield V'

Jonas Gillberg, GDC 2019

Play on the Vault FREE!

Bots designed specifically to help testing can be incredibly valuable. This talk provides some useful insight on how to develop and manage the workflow of such a system in a large team.

Automated Testing: Using AI Controlled Players to Test 'The Division'

Jose Paredes, Pete Jones, GDC 2019

Play on the Vault FREE!

Bots seem to be a theme of GDC 2019! This is an hour session, compared to the 30min for the Battlefield session listed above, so looks to go into more detail. We'd recommend seeing both if you can!

Automated Testing of Gameplay Features in 'Sea of Thieves'

Robert Masella (@ZipLockBagMan), GDC 2019

Play on the Vault FREE!

See Robert describe how Rare used multiple tiers of tests in unreal engine to have the confidence to ship Sea of Thieves updates every week. Also features the most astounding bug count graph in GDC history.

Level Up Your Jenkins Reporting

Adrian Yu, GDC 2019

Play on the Vault FREE!

A talk by veteran roundtabler Adrian Yu on how EA uses Jenkins for continuous integration and deployment - vital tools in the belt of any serious automated tester. His talk is more than just a dive into Jenkins, as he promises to explain how they manage spam and notifications to make them truely useful, rather than noisy.

A QA Perspective on Live Game Production

Michael Vork Larsen, GDC 2019

Play on the Vault FREE!

How RETO MOTO maintain stability in their live service game by involving QA early in an aglie process, including using automated testing. Heros & Generals is a game that's been running for a while, so this promises to have some great insights.

Tools to Reduce Open Bug Count at Media Molecule

Amy Phillips (@amy_phillips), GDC 2018

Play on the Vault FREE!

How to improve the stability of your game by linking tools together and optimizing the path that bugs take between being introduced and being closed.

Automated Testing and Profiling for Call of Duty

Jan van Valburg, GDC 2018

Play on the Vault FREE!

You will learn about the inner workings of Call of Duty's automated building and testing tooling, it's various unique features, and how its ideas might be applied to your own projects.

Assassin's Creed Origins: Monitoring and Validation of World Design Data

Nicholas Routhier, GDC 2018

Play on the Vault FREE!

Daily automated testing of world design data is a game changer. Attendees will learn how it helps world builders in their daily tasks, allows for more iteration time, provides data for evidence based decision making and makes for a much smoother debug phase.

Adopting Continuous Delivery At Rare

Jafar Soltani, GDC 2018

Play on the Vault FREE!

You will learn about the benefits of adopting "continuous delivery" in AAA video games, what challenges you'd face and what practices you can use.

Practical Unit Tests

Andrew Fray, GDC 2014

Play on the Vault FREE!

A failure study of Fray's first attempt at unit testing gameplay code, what went wrong, and how you can do better.

Notes from previous years

Notes from Process, 2018

Notes from Legacy, 2018

2021: Process, Legacy, Implementation (Thanks to Sean Sweeny)


If you have any questions or just want to say hi: @tenpn