Dev Diary #171 - Post-Release Support

My role within the studio is to strengthen communication between us and you, the players, to ensure that we understand what you want from the game and that you understand what our intentions are for the future. While I’m just one part of the broader Community Team for Crusader Kings, I’m ultimately responsible for nearly every piece of public-facing communication we publish as a studio: dev diaries, feature breakdowns, chapter premiere videos, social media posts, etc. I’m also responsible for the reverse; every piece of feedback that ends up on a designer’s desk goes through me at some point in the process.

Today, I’m going to talk about the release of Khans of the Steppe and the feedback we’ve received from players, as well as how we’re addressing it. After that I’ll give a brief overview of how our development cycles work, what the hell Post-Release Support even is, and then cap it off with a quick look at what our next steps are as a studio.

I am a map gamer, so fair warning: There will graphs and charts in this dev diary.

State of Launch​

As you may or may not be aware, Khans of the Steppe and the 1.16 “Chamfron” Update were released on April 28th, and the initial response was fairly positive both from a technical perspective and a player sentiment one. However, we quickly noticed a spike in crash reports and commentary from players confirming this. Setting our lovely QA team to work, we quickly identified two major contributors to instability in 1.16 and pushed hotfixes to tackle both of them.

These fixes have led to a significant reduction in crash rates, but we’re still seeing elevated levels, so we’re still working to identify and resolve the causes of these crashes.

[Crash rate analytics since the release of Khans of the Steppe. The 1.16.0.2 hotfix (circled in red) made a big difference, but there’s still work to be done.]

While there was an immediate spike in negative reviews due to stability issues, the response at large to Khans of the Steppe was quite positive right out of the gate. When you spend months working on a specific project, it’s always an immense relief to see that it went well and players were having fun with the new content, so everyone at the studio was elated at the response!

Then the review score started dropping.

[Steam reviews for Khans of the Steppe. You can see the ratio of positive to negative reviews shrinking over time; In the “biz”, this is considered a Bad Thing. While the amount of people who leave reviews are a sliver of a fraction of the greater playerbase, this is still a valuable source of information for us.]

With all of our releases, we do a series of internal reports on the state of things at predefined intervals. There’s a Day 0 report, Day 1, Day 7, etc. While the Day 0 and Day 1 reports were initially positive, by the end of the week it became clear that there were outstanding problems that took some time to reach a breaking point for players.

So, what were those problems? In order to figure that out, we have to do some basic analysis of the reviews themselves. To begin, I took every negative review on Steam and put them into a spreadsheet where they’re arranged, translated (we try to assess feedback from as many languages as we possibly can), and categorized based on what their main complaint is. (This isn’t the only way we analyze feedback, but reviews are fairly easy to explain so they make ideal content for demonstrating the point in this dev diary.)

Once everything has been neatly categorized (a task I find immensely soothing, for the record), I can generate a quick chart showing which complaints are dominating the conversation. The main cause of stability complaints in the reviews were already addressed or being investigated, so we can skip