Brute force bug fixing

If you played the 2018 demo, you’ll probably have experienced a couple crashes or physics turning off suddenly.

Unfortunately I missed these bugs while crunching to get the demo out before the year’s end. Planning a lot more carefully this year to avoid a similar situation.

Usually when fixing bugs, we first try to “repro” (reproduce – steps required to make the bug happen). Once we’ve reliably found a way to repro the bug, we can find what code is causing it by debugging or looking through error logs. Then after applying a fix, we try to repro it again – if it doesn’t show up, the bug is fixed.

Physics bugging out was somewhat easy to reproduce and fix. It turned out armour pickups were being spawned with invalid transforms which would cause the game’s physics and collisions to behave really strangely (clipping, freezing or sometimes just crashing). The engine never reported any errors though which made repro a bit difficult, but eventually it was fixed. Will also be writing my own spawning function which will check for invalid transforms and report an error if it happens to prevent this issue from coming up again.

Ok, physics bug fixed. Shooting dummies doesn’t cause the game to crash or collisions to stop working.

The more difficult bug was random crashing when shooting stunned NPCs. (The above video just has “dummy” characters, they have no AI and can’t be stunned or knocked over). I couldn’t reproduce it reliably no matter what kind of crazy test maps I set up on various PCs. I could run these test maps on repeat for hours and some days nothing would show up. Other times it’d crash consistently for a few minutes, then disappear again.

I swear it became more elusive the harder I tried to look for it, but right when I started questioning its existence (and my own sanity) it’d come back again. Looking through error logs or debugging with visual studio yielded nothing meaningful – it was always a different random actor getting invalid transforms.

It seemed to have some relationship with CPU performance or heat? I found it happened more often on colder days than hot, and somehow by undervolting my CPU and cleaning out fans, I was able to repro it consistently.

Unfortunately visual studio and the error logs didn’t give anything useful still, so at this point there was a lot of trial and error: comment out some code, run tests, repeat. Blueprint debugging is fun.

Would run these tests for hours and wait for a crash every small change.

Fortunately the game is mostly in C++ and doesn’t use a lot of blueprints or this would’ve been an even worse nightmare. Turns out it was the bullet impact effects spawning with invalid transforms. This usually didn’t outright crashed the game, but like the above mentioned issues caused physics and collisions to get really weird, eventually causing other actors to teleport to invalid transforms which would (sometimes) finally lead to a crash.

One Reply to “Brute force bug fixing”

  1. Next time I’ll probably ask someone with more QA or engineering experience on how to approach this. I’m sure there’s a faster way to debug weird crashes!

Leave a Reply

Your email address will not be published. Required fields are marked *