Utility AI in an RPG - Devlog #15

There are many types of AIs used in games. My two favorites are G.O.A.P. and Utility AI. In All Hail Temos, I use Utility AI as the primary decision maker. In this post I’ll explain why, as well as give some technical details on how I am implementing this. I will also include an improvement I made for scaling up to very large behavior sets.

A simple Behavior Tree in Unreal Engine (NOTE: I use Unity)

Popular AI Systems for Games

Behavior Trees

Behavior Trees were popularized by Halo, and are essentially visual scripting tools. They use a type of flow chart to decide which actions are taken at any given time. They took over in popularity from the previous favorite of Finite State Machines, which is a core programming pattern, but difficult to build large game AI with as they have too much structure and rigidity.

Behavior Trees are great for making custom situations, and especially good for teams that have a lot of level designers who also create AI scripts. They can work with higher level tools than programmers, can visually see and test how the AI will make decisions, and change them in many unique situations. The bigger the team of people working on the AI behaviors, the more helpful this sort of tool is. It has scaling problems as well, but allowing people to make custom logic easily is a big win.

In my opinion, this is better for “lots of level designers” and with the emphasis that they will also be making custom AI for different situations.

With a solo developer, I think this is not a big win, as its another type of programming language, and basically breaks down into if/else statements, so unless thinking visually is better for you, it might be low improvement and more work to maintain separate than your existing code base.

Goal-Oriented Action Planning has a simple premise, but works well

GOAP

G.O.A.P. is Goal-Oriented Action Planning, and is a “planning” system that looks at all the available options, and creates plans for them, and tests how quickly the action can be accomplished to determine whether it should be performed over other actions.

GOAP has amazing results when tuned properly, and the game that popularized it was F.E.A.R. which remains a classic because of how good the AI was in responding to the player and unique scenarios. GOAP has some issues with performance due to the planning work, and re-planning as things changes, but is an amazing system. My initial plan was to mix GOAP underneath a Utility system, so we have higher level goals (Utility) and lower goals (GOAP). I may still do that later if I can see the benefit, but it may not be a better solution, and has a big performance cost as agents (NPCs) increase.

GOAP is very game-specific, in that it understands there is a very short loop going on for all NPCs, which is:

• Go somewhere (move)

• Play an animation (shoot, duck, throw grenade)

• Use a smart object (door, button, window, etc.)

Because of the performance and to get good plans, it’s important to “bucket” the different possible plans together (ex: “combat” different than “lifestyle” different than “travel” type AI actions). It’s not good to try to stuff all the potential AI actions together, purely from performance, but also in having good structure for your plans and actions.

In my opinion, this is less good for “lots of level designers”, but better for “a few AI programmers”.

I think this can be good for a solo developer as well, especially when there are scenarios you want to get very precise with, such as reacting to FPS combat.

Utility AI

Utility AI gets used a lot less in AAA games, but was used in The Sims, and Alien Isolation, both great showcases for how good Utility AI can be, in very different environments (huge sandbox vs survival FPS).

I decided on Utility AI because it allows all potential actions to be tested together in one giant pool. It has good performance, and scales to a lot of actions well, and does not need as much structure.

In my opinion, Utility AI is better when the game is supposed to be very AI heavy, and have many AI options, and so there is a dedicated team to working on the AI, which is not scripted for specific high points (level scenarios or being especially good at FPS). Alien Isolation does show it can work well for these kinds specialties, and with “bucketing” around the Utility AI behaviors the same thing could be accomplished, so then it’s just a preference.

For a solo developer I think Utility AI is a good choice, especially if there are a lot of actions, and you want to tune them to potentially all be available at any time.

Could an NPC decide to go do something besides remain in combat? If so, having all the options always available is a good choice and Utility AI provides that.

If you want to learn more about Utility AI, you can check out Dave Mark and Kevin Dill who were instrumental in bringing Utility theory to games. Here is an easy tutorial for Unity explaining how to implement it.

Implementing Utility AI

Utility AI has 3 basic components:

• Behavior Sets: All the behaviors we will score, so we can select a high scoring behavior.

• Behaviors: The things we will do with the AI. “Actions” are what we are scoring to get.

• Behavior Considerations: Information we use to make our decision. What we consider in making a decision.

We ultimate want to get a behavior, such as Attack, Eat or Sleep. This is what the NPC agent will want to do at this time.

We have all the possible behaviors in our behavior set, so after we score the behaviors we can pick the top scoring behavior, or a weight random of the top 3 behaviors, and do that, on a scale of 0 to 1. This behavior is said to have a “High utility value” the closer it gets to 1 and the further it gets from 0.

Here is an example of a Behavior Set (“Standard_Set”) with a single Behavior (“AttackRanged“) that has 2 Considerations (“EnemyDistanceRanged“ and “EnemyPowerDifferenceRanged“):

name: Standard_Set behavior: - name: AttackRanged consideration: EnemyDistanceRanged: info: Further away is better for ranged input_source: EnemyTargetDistance range: 3, 30 curve: IncSmooth score_weight: 1 EnemyPowerDifferenceRanged: info: If they are more powerful, worse to get close input_source: EnemyPowerDifference range: -5, 5 curve: IncMiddle score_weight: 1

These are not complete considerations, just examples to show how it works. Making them complete takes more considerations than I want to paste into a post.

I prefer to store my data in a YAML format whenever possible, because I like working with text, I can search it and back it up easily (and find differences over time), and it is fairly easy for modders to access.

Remember the “score_weight” value here, as I will be coming back to it later.

How it works

Utility AI requires 3 types of data to score:

• Input Value (number)

• Range: Minimum, Maximum

• Curve: An X-Y axis of values. X-axis is the position in the range. Y is the final score.

Next, there are 3 steps to creating a Utility AI score:

• Get an input value

• Create a 0-1 value, based on the input value inside a range of values. ex: 6 in 0-10

• Map the 0-1 value into a Curve to determine the final score: 0.6 on the X-axis of a curve.

When I want to see whether I should perform the behavior for “AttackRanged” I will first go get the input data, which comes from the source “EnemyTargetDistance”. I find out how far away the enemy is, such as 5 meters.

My input value is “5”. Then I compare this to a range, such as 3 to 30 meters.

3 is the start of the range and 30 is the finish of the range, and this produces a 0 to 1 value.

30 - 3 = 27. The total distance I will check is 27. 5 - 3 = 2. I will test 2 out of 27 total.

2 / 27 = 0.074. 0.074 is the value I will then pass into a curve to transform it.

For AttackRanged, I used the “IncSmooth” value, which stands for “Incrementing Smooth”, which means it gets higher values (Y=1) as it from X=0 to X=1, like so:

My “Increasing Smooth” curve at 0.074

The value 0.074 is basically 0 on this curve, and will probably not be chosen, because shooting at 5m away is not a good utility compared to doing melee attacks or running away to a further distance where ranged is more effective.

Here are some other curves I use. You can see with some of them, having a 0.074 would be 1 or closer to 1. So I can transfer the linear number of the range (3m to 30m) into some other value based on the curve.

Some of the curves I use to score

Ultimately this consideration score is combined with all other consideration scores, giving the behavior it’s overall score. If the score is high enough, this behavior will be selected for the NPC to act on.

Scaling to Many Behaviors With Utility AI, I can have many behaviors all competing to have the highest utility score at the same time, but defining all of these takes a lot of time. So I implemented a way to make Shared Considerations, so that I can instead define “AttackRanged” like this:

name: AttackRanged shared_consideration: - EnemyDistanceRanged, 1 - EnemyPowerDifferenceRanged, 1

This is much smaller to write out, and allows me to re-use considerations easily. I have another YAML file that has all the shared considerations inside it, and any behavior can just add a shared consideration.

But, what is the “, 1” doing after the name? That brings me to my improvement of Utility AI in how to scale to even more behaviors in a manner than makes development, testing and iteration easier to do, by allowing even more sharing and customization.

Score Weighting

Getting a score is the entire purpose of the Utility function. And the input value, range and curve allow us to get that.

But what if I want to make something similar, but a little different. Let’s look at 2 more behaviors “EatFood” and “EatFoodDrunk”, which will be invoked if the NPC is drunk:

- name: EatFood info: If they are hungry, and not in danger shared_consideration: - StatHunger, 1 - ThreatHigh, -1 - name: EatFoodDrunk info: If they are drunk, they will want to eat if they are less hungry shared_consideration: - StatHunger, 1.2 - StatDeductionLow, 1 - StatDrunkHigh, 1 - ThreatHigh, -0.7

So all of these are Shared Considerations I am importing into these behaviors, so I dont have to redefine them. The initial shared considerations will have a Score Weight = 1. This means they just pass through the score without changing it, because I already have the formula working to score between input value, range and curve.

But when I include a shared consideration, maybe I want to modify that value somehow.

First, I allow totally inverting the value. Why make a “ThreatHigh” and “ThreatLow” consideration for whether the NPC is in danger (enemies around that see them), when I can just invert the value. Using the “Score Weight = -1” means that if it was 0, now it’s 1. If it was 0.3, now it’s 0.7. It doesn’t multiply by negative one, it inverts the 0-1 range.

But, I also allow multiplication. For example, when they are drunk, I increase their hunger amount by 20% (1.2) and I decrease their threat by 30% (-0.7), so I can control both the magnitude and the direction of the value based on adding 1 more number after the shared consideration name.

This allows me to create a new behaviors that build on standard considerations, but I can adjust them very quickly.

Then if I decide the AI is not acting correctly, I can quickly change either the base shared consideration to affect all the behaviors that use it, or just make a small modification changing a Score Weight = 1 to Score Weight = 0.9 or 1.1.

This greatly improves my ability to add a lot of behaviors, but continue to balance them against each other to get a good result.

I don’t know if other people have done this before, but it is a big improvement for me to be able to implement Utility AI into a big Scrolls-like game, like All Hail Temos.

Score Weight as Priority

There is another important way to think about Score Weights, which is that they can be used for priority between a number of different considerations. If an element is less important, I can set the “Score Weight = 0.8” and now it’s only 80% as strong an indicator where previously it was 100%.

This means when I am setting up considerations I can weight them so they don’t all have equal value. Then I don’t need to balance all of the curves and ranges perfectly, because I also have another way to change the values at the end of the pipeline.

To really understand the formula:

1. The Range is the beginning of the processing, giving us the initial 0-1 value from our input.

2. The Curve is the middle of the processing, converting the initial 0-1 into a new contextualized 0-1. The context we learn from the curve is:

• “Is it getting more or less important to do this?”

• “How much more or less important is it to do this?”

1. The Score Weight is the end of the processing, making a final adjustment of priority.

Finally, each consideration is merged with all the others to form a final score, that is 0-1. Because all our scores are between 0-1 they can all be compared together equally, so the highest score is the option that best fits all the information we currently have.

Without the final score weight processing, all the considerations have the same value in terms of being merged. As an example, 2 considerations for the behavior “CraftMilkCow“, for milking a cow:

• NeedsMilk, 1

• HasTrait:FearOfCows, 1

“Needs Milk” is clearly more important than the “Fear of Cows” trait, so if NeedsMilk was 0.1 and FearOfCows was 1.0, the final value would be about 0.5, but this is too high because NeedsMilk is more important than FearOfCows, and FearOfCows should not force this to be high utility. So, instead it can be written like this:

• NeedsMilk, 1

• FearOfCows, 0.2

This now makes FearOfCows only 20% as important, so if NeedsMilk is 0.1 and FearOfCows is 1.0, but now FearOfCows becomes 0.2, and then the result is 0.15 instead of the previous 0.5, which may be a more appropriate balance.

The final processing step of applying the score weight allows the different considerations to get different priority amongst each other.

Argument Passing

I have one more method of making it easy to add more elements, by having an optional “Target” value, so I can have a generic condition such as “HasItem” and then pass different items to it to test. I write it like this:

name: GetDrunkSitting info: With low Deduction (WIT-LOW + AWR-LOW), wants to be drunk shared_consideration: - StatDeductionLow, 1 - StatDrunkLow, 1 - HasItem:Intoxicant, 1

The line “HasItem:Intoxicant” allows me to check if this actor has any intoxicants on them before they try to sit down and get drunk. This is basically like passing an argument into a function, instead of having a 1-1 mapping of shared considerations to definitions, I can use arguments to make some handle many different types of cases, where it’s essentially the same check.

The shared consideration looks like this:

HasItem: info: Do we have this type of item? input_source: HasItem target: None range: 0, 1 curve: IncBoolean score_weight: 1

True or False?

This again increases my ability to scale the number of AI behaviors that are available, while maintaining a more reasonable amount of work to create, test and iterate on them.

How does this affect gameplay?

Having interesting and believable AI in any game is difficult to do, and in a large open-world RPG game, it can be especially difficult to manage all the different AI behaviors and keep things interesting.

One element of Utility AI is that interesting behaviors often emerge when you don’t necessarily expect them to, because all options are on the table, all the time.

If an NPC is in combat (ThreatHigh) but they are drunk (StatDrunkHigh), they actually reduce their threat level by 30% and increase their hunger by 20%. If they were not in the thick of battle, maybe they would walk off to a corner and eat until they weren’t as hungry.

So that it doesn’t look like a bug, it will also need an accompany bark line such as, “I’m getting hungry”, so the player knows they are purposely leaving the battle to eat.

I think this can keep the game interesting and fun, once it’s all working properly.

Conclusion

It’s been a long run getting all the components an open world RPG needs (8 years so far), and good fundamental AI behavior is a critical component.