Automated Testing Roundtables 2018 Day 2 Notes

Notes from 2018 Day 2: Legacy

with thanks to Nicholas Dunning

First look at the problem you are trying the address before you dive in
Start simple, and use the team to start to build the focus
Use simple record and reply to start on a functional level
Focus on new code and start with unit tests
Sikuli is a good tool to start with when you don’t have hooks into the old system
https://sikulix-2014.readthedocs.io/en/latest/index.html
Check if the business can afford to dig deep into this type of testing
Make sure no one else is already doing this in another way
Could be a manual approach which can be automated
Start with code which has little or no dependencies
Code coverage is also a good tool as there are many off-the-shelf solutions

Old store the failed test results
Storing just a time series can be useful
Smaller tables hidden behind views can help with search and query time
Smaller items can be kept for a long time, whereas larger items could be deleted after a reasonable amount of time
Companies are using Machine Learning to detect the most useful information, would be worth to search this industry
Raw data is only kept for a short time
Once the build has been analysed, the remote store of raw data is deleted
Devs need to pull this locally, so they should have a local copy anyway
Compression of large files is key
Use a combination of Hot, Warm, and Cold storage based on the data type
If you have incremental builds, only store the output of the failing increment (Uber)
Book: Designing data intensive applications

For games, it’s tough to find something off-the-shelf
Loading times and network rates mean it is tough to test without hooks and active feedback loops
Mods written to hack the game can actually be used at testing tools
Check out AutoPlay AI, they are working on something for this so check them out

Code coverage is just a metric, so it shouldn’t be at the “cover-all” solution for how well your code is actually tested
High-coverage doesn’t mean you are testing well, but it’s at lest a good initial indicator
Sometimes it can lead to a false sense of security
Bug tracking often starts in the low-coverage areas which might be the wrong place to start
Low coverage is bad but striving for a high number might not be the ultimate goal
Tough when you have engine-specific API which cannot be called in tests
Look at how often the code is changing, as this is actually a good place to start with finding bugs
If a certain function is constantly changing, it might be prone to errors
Try to reduce the complexity of the function instead of striving for high coverage

Separate the test results and the test framework to help find the cause of the problem
Focus on the infrastructure to increase confidence in the system
If you are thinking of a rerun policy, be careful
Tests should be kept small
If your unit test is big, it’s probably not actually a unit test
Mocks are good to keep the scope of a test small
Keep parameter count of methods low in order to reduce the amount of build-up and tear-down needed
Don’t spam people with auto-generated build information, because they will ignore a lot of information
It might not be the developers who don’t know how to test, but actually the testing strategy may be inclusive to wiring flakey tests
Educate people to know why granular testing is beneficial
It’s often seen as a negative to write lots of small tests
Andrew's free vault talk covers spotting bad patterns in a poor unit test suite and fixing them.

The developer responsible for the broken build is sent an email
Lots of information on how to reproduce the problem
Also sent to QA team
Use pattern analysis to automate the feedback process and recuse the manual overhead
Use pre-push hooks in git so that it doesn’t get merged
If you broke it, you fix it
Helps to reduce context switching