Dev Summary - End-Of-Mar 2025

After last month, this month feels considerably more positive, despite many set-backs!

Restructuring the Game

By far and away the most difficult part of making Push a game that is capable of being used by ML-Agents ("AI Brains") for training, has been butting up against a lot of the core designs I made when creating the multi-player version. For example, the original intention when playing games is that you would start a game manually, enter a lobby with an opponent (where a lot of verification is done), then lock-step into the game world to play the game in sync.

For ML-Agent training I needed to revisit ALL of the areas that prevented me from live-resetting the game inside the game world, which obviously resulted in a LOT of rewriting of some pretty fundamental code. What was once the "Game Controller", "Player Controller" and "Board Controller" for the whole environment, has now become it's own logical unit that can be created and destroyed anywhere, be reset or started at any time. This means the ML-Agents can take control of these units, and play on them, and we can spin up as many as feasible to accelerate the training (though I haven't quite gotten there yet).

There were a lot of other issues, where previously we would expect a solid order-of-operations to occur (as we needed because we had to sync over the network), but now things can somewhat happen out-of-sync, and independently in any of these units. Things as simple as Undo-Redo were causing issues where resetting the game would result in actions from the last game being played out in the initial phases of the new games! It's like turning over an ancient rock, and all these bugs come scurrying out.

The silver-lining here, is that these changes might actually cover a request that some have made, which is to stay in-game and replay the person you are already connected with. With this re-design, that may be possible now, and I will definitely explore it.

Understanding Unity ML-Agents

While I may understand a lot of how my own game is implemented, whenever you introduce a "Black Box" like ML-Agents to your system, and try to integrate it, you open yourself up to all sorts of issues where you may simply not understand what it's doing, and even worse, understand how to fix a problem. A lot of the initial work restructuring the game was spent working on, reworking on, and re-doing over and again the ML-Agent controllers, as my assumptions were challenged, my knowledge increased, and much documentation was read.

It also didn't help in the slightest that a LOT of the examples out there are already woefully out of date, and not applicable. This is especially bad when you are learning a new system, as you can't tell what is correct, and what you should truly be doing.

A side-track for Upgrades

The good news is, I realized I was circling the drain a tad, so I bit the bullet and decided to upgrade the whole project to the latest Unity Build (6+), which then allowed me to also utilize the latest ML-Agents package (3.0), which was thankfully well documented enough, with enough examples, for me to intuit the remainder of what I was missing.

What kind of things were missing, you ask? Well, one example is how to make a turn-based game where you take TWO actions per-turn (out of a possible three), which was not clear how to do from a training standpoint. 99% of the tutorials were for 3D games, teaching Agents how to move around an environment, walk, play 3D puzzles, and only one was even close to a "board game".

It took a while to figure out the desired pattern, but the documentation was good enough to lead me there.

Training in Progress!

Enough words! How about a demonstration? I'm VERY pleased to say that I have training underway, which will no doubt be an iterative process, as I iron out the appropriate rewards, restrict illegal moves, and cover any edge cases, but already, the system appears to be working for the game!

Check out a snippet of the latest training session, where two Agents are learning to play against each other in an adversarial manner (i.e. they are both trying to learn to win, despite the other trying to thwart their attempts, very like human players!)

Exciting!

We shall continue into April, and while I was hoping to get a training set in March, I at least managed to suffer through most of the pain, and it seems like mostly clear skies ahead. April it is!

Goals for April

The carry over from last-month is to complete a training set, and test it out, human vs computer! If I get something even remotely interesting, I'll consider packaging it up for release.

Naturally, I changed a ton of code, so I suspect most of the remaining time is likely to be spent regression testing everything, making sure the multiplayer game still works, and that I haven't prevented normal play in any way.

My robot buddies and I are looking forward to challenging you soon!