DevLog #7 - Nerding out on testing

A fair bit has happened since the last development log. We've released a private demo for our Discord users and we've received all sorts of feedback and bug reports, ranging from nasty crashes to small UI issues. While writing this, we're in the process of releasing the 4th build already, fixing issues as fast as we can. The game is going to likely remain in this limited access state until everything is working as well as it should be.

This experience has only strengthened our belief that automated testing is the key to verifying that everything is working and that all corner cases are being handled correctly. In this devlog, I'll explain a bit about how the system works and why this particular genre can benefit immensely from it.

The anatomy of a bug

Bugs can range from the game crashing or freezing to UI getting stuck to the screen, or colonists getting stuck doing weird animations. All bugs exist because the code of a game does something that developers have not anticipated which is almost always incorrect. In rare cases, we tweak it a bit and call it a feature.

The code for a game usually flows through many states, changing things constantly at high speed, and doing many many computations to deliver every single frame. With every bit of new code that we add, the complexity keeps on increasing and it quickly becomes impossible to know for sure if it works correctly or not.

So we run the game, play it and verify that it does what it's supposed to do. While this works well and remains a very important aspect of development, it does have disadvantages. First of all, playing the game takes time. It's one thing to playtest for feel, and another thing to do the same thing over and over again. Why again and again? Because with every change in the code, everything needs to be validated again. Sure, assumptions can be made that changes in on part of the game won't have an impact on another part but there's plenty of cases where these assumptions are wrong. Second of all, with manual testing, it's hard to know exactly what is happening. Just because a colonist suddenly decides to walk through a wall in a straight line and never look back, it doesn't mean it's clear what is going on in the code. We only see the end result, but not the flow that got the game into this state.

When the player isn't really the player

With most games, user action drives the game directly. In the case of an FPS, the player presses the move keys and the character moves instantly. Picking up ammo, opening doors and climbing stairs are all manual actions which are easy to replicate. With NPCs the problem already starts to get a bit more fuzzy, as they're not controlled directly by the player, but rather indirectly through the players actions. It's not as easy to verify that an NPC can climb some stairs. First, you have to get it close to the stairs and then somehow give it a reason to do that.

In the case of a simulation game, the world is a sort of a living breathing organism, in a constant change. The player, through the UI, intervenes in the simulation but what happens inside isn't usually directly controlled. The colonists make decisions of their own and interact with the world. They operate crafting stations, sleep in beds (if available), move items around, mine the terrain and socialize in their spare time. In a sense, they're the players. The problem is that it's hard to validate what they do in various situations, not to mention that getting them to do specific things with precision is like trying to coordinate a couple of goats. For instance, what happens when a sick colonist on bedrest notices that the room is starting to lack oxygen? Well, to reproduce that, we'd need to first have a colonist get sick, build a medical bed, wait for it to get inside and then somehow drain the room of oxygen. Further, if there's a bug, then this needs to be replicated multiple times until it works correctly. Not exactly practical.

Using code to test code

Since early on in development, we've had parts of the game automatically tested. What this means is that the game starts, does actions by itself and checks that at every step everything is as it should be. I'm not going to post actual code here but an example goes like this:

Plant Plot basic cycle

1. Setup an empty map of size 8x8

2. Build a hull over all of it and fill it with oxygen

3. Place the constructed plant plot, a colonist and an empty storage container

4. Validate that saving works correctly

5. Check that both the colonist and the plant plot are idle

6. Set the plant plot to grow a tomatoes

7. Check that the colonist is still idle (there's no water)

8. Add water and fertilizer to the storage container

9. Verify that the colonist starts transporting things

10. Verify that the colonist starts working on planting after the transport is done

11. Wait until the plant has grown

12. Check that harvesting works correctly

13. Quit the map and return to the menu

This is obviously a very simplified example and even the most basic tests are more complex but it does show what's going on. In reality, there are a lot more checks performed and saving is validated at almost every step. We have hundreds of these tests and the list keeps on getting bigger. This is really useful because now, after fixing something, we can just fire up all or a relevant subset of these tests and we already have some confidence that nothing else broke.

So far a significant amount of these tests have been either checking for specific edge cases that were known to have had issues at some point, or what is called the happy path, like the example above which covers the most typical flow. The results here are, not great, not terrible. There's a large number of flows verified but there's also a lot missing. The parts that have had full coverage from the start ar the ones that we've generally never had any issues with.

As I said before, when something is visibly broken on screen, it's hard to tell what's going on or what even happened to get to this state. It's hard to keep an exact eye on the colonists and what they're doing, plus, not everything going wrong is visible. The graphic from before in reality is more like this.

There have been bugs where I spent days figuring out what on Mars was going on there. Most often, fixing something is trivial after having identified the root cause. One advantage with these automated tests is that when something fails, instead of having that blurry view of what happened and how the code got into this mess state, the picture is crispy clear. Typically when something covered by the tests is broken, it takes less than 5 minutes to have a fix, instead of hours, potentially days.

So why wasn't everything validated from the start? Well, writing up all these cases and checks also takes time which is a valuable and limited resource for us but going forward, with every build this coverage should be more and more complete. Past few days were spent checking every single aspect of the botany station and the plant plot. There were issues discovered doing this including one that would be possible but particularly rare for anyone to stumble upon but one that would break the game nonetheless. Another issue discovered was actually in the animation system but it could only be triggered in a very specific circumstance. Tests can cut power to a building with surgical precision, or make items appear out of thin air at specific times. This is something that manual testing cannot do.

Thank you for your continued support and interest in our project. Stay tuned for more updates!