Dev Diary - Maps and Ancient Artifacts

Ancient Artifacts

One of the first things I did on this project was to make the map data files. Without the map, you can't test or build any of the core functionality of HAE-Eye that defines the majority of HAE Stack's gameplay. Map files are kind of weird. There's a lot of different formats for maps that are used for a bunch of different things. Often you can convert from one format to another (but not always), and different libraries and tools are only compatible with certain formats.

For my situation I ended up using a [c].pmtiles[/c] file for the map's visuals, and a [c].geojson[/c] file for place information and search. I ended up with these files by downloading an extract of Open Street Maps data, and then running it through a bunch of tools to generate the files necessary to render them in game. There were some issues, but for a prototype they worked fine:

Steam post image * Note how the water just abruptly cuts off at a weird angle

So there we go! I had my files, they rendered in game, and I could display detailed place information in search: time to start building the game! And for over a year I just used these same files, with the knowledge that eventually I would need to update them. But eventually comes eventually, and as the game quality increased, the janky-ness of the map became an obvious issue.

How the $%#& Did I Make Those Files??

More than a year removed from the initial file creation, I realized that I had almost no memory of how I'd made the [c].pmtiles[/c] file I was using in the game. I knew I'd installed some tools on my Macbook (I usually develop on my Windows PC, but some of the tools only ran on Mac/Linux). Did I use [c]tippecanoe[/c] or [c]tilemaker[/c] or something else? Did I export from Geofabrik, BBBike, Planet.osm? Was the original file a [c]pbf[/c], [c]osm[/c], [c]xml[/c], [c]geojson[/c]? I'm using [c]pmtiles[/c], but did I convert to an [c]mbtiles[/c] file first or go directly to [c]pmtiles[/c]? During the original file generation I'd experimented with pretty much all of those tools and technologies, but the exact sequence of steps I used to end up with workable files is something I'd completely forgotten.

Building the World

If you're thinking: "well it can't be that confusing", Here's a rough step-by-step breakdown of the steps I used to create the current version of the map.

1. Download a map region as a [c].osm.pbf[/c] from a source like Geofabrik or BBBike.

2. Generate a [c].pmtiles[/c] map from that [c].osm.pbf[/c] using tilemaker.

3. Check that the [c].pmtiles[/c] file works using pmtiles.io or in-game by replacing the old [c]map.pmtiles[/c] file.

4. Determine the bounding box longitude & latitude to limit the final map to in-game. (This will significantly reduce the final file size).

5. Crop the [c]osm.pbf[/c] file using Osmium.

6. Regenerate the [c].pmtiles[/c] and test it to make sure nothing is wrong.

7. Realize something is wrong: In my case, a certain large body of water was completely missing from the cropped file (presumably because it extended out past the bounds. Steam post image

8. Go to Open Street Map and find the ID of the missing "feature" by exploring the map.

9. Download the feature directly using the Open Street Map API with [c]wget[/c].

10. Merge that new data into the cropped [c].osm.pbf[/c], but first use Osmium to create a new ID for the feature that's higher than the highest feature ID that exists in the cropped [c].osm.pbf[/c] to avoid conflicts, then use Osmium to merge them.

11. Generate a [c].pmtiles[/c] file again and check that the feature appears now. Repeat Steps 7-11 until satisfied.

12. Rename things that you want renamed. For example if you want to change the name of the city, or any other features, you can do that as a one-off by using Osmium to convert the cropped [c].osm.pbf[/c] to a [c].osm[/c], and then running [c]sed[/c] with a replacement string, and then converting back to [c].osm.pbf.[/c]

13. Generate the "final" [c].pmtiles[/c] file using Tilemaker. Replace the old [c]map.pmtiles[/c] file in game.

14. Generate the .[c]geojson[/c] file using Osmium.

15. But we're not done! The generated [c].geojson[/c] file will include a bunch of junk, including duplicate entries for various IDs, so open up the [c]GeoJSONFixer.html[/c] file I wrote, and drop in the [c].geojson[/c] file to get a [c]fixed.geojson[/c] file. Replace the [c]map.geojson[/c] file in game with that new file.

16. Now you're done... (except not really).

So, just a simple... 16 step process (if you ignore all the installations, set up, and various configuration files, and most of the console commands) to go from a download to working game files! It works, it's reproducible, and I actually took notes about how to do it this time!

One More Wrinkle

Something that bugged me a little is that even with the name of the city changed, all the other map data is still identical to the real city this data comes from. I'm not a fan of that, since the implications are that you're viewing personal, potentially embarrassing information about people who live or work at specific locations. I'd been saying for a while that I should shuffle or rename all the streets in the game to more thoroughly distinguish between the real city and this fake HAE Stack version, so for the last couple of days I've been working on that.

Now after step 12 in the above list of steps, you need to add:

1. Open the [c]ShuffleOsmStreetNames.html[/c] file.

2. Convert the [c].osm.pbf[/c] to a [c].osm[/c] and drop it into that webpage.

3. If you haven't already, generate the [c]shuffle-names-list.json[/c] file. This will take all the street names listed in building addresses and export them into a big list.

4. Go through the list of street names and delete anything that shouldn't be shuffled.

5. Load that street names file into the shuffler and generate the [c]shuffled-names-mapping.json[/c] file.

6. Click the shuffle button and wait while several thousand find/replace operations run on your 100MB [c].osm[/c] file.

7. Run the steps to generate the [c].pmtiles[/c] file and test it in the game. Amend the mappings and street names files and repeat steps 3-7 as necessary.

This is still a work-in-progress and I haven't done a full QA pass to test it in game. The street name shuffle has resulted in some pretty dumb looking stuff before I iterated on it, for example, a neighborhood being renamed "Chesapeake Harbor Harry S Trumans", because I was replacing the word "Hill" with "Harry S Trumans".

Once I get all the bugs worked out and update the map, I'll also have to change any text that references specific addresses. Fortunately in the game data files I use location IDs that aren't affected by this street name change.

All of this so that the fake people living at the fake versions of the real places in this fake city won't be confused with the real people living at the real places in the real city.

User Generated Content

One of the other reasons it was important for me to formalize and document the map and data creation process is that eventually other people may be able to make their own content for the game. I've implemented much of the game such that you could load in your own story and people, and fully author everything from the chat system, the tasks, and people's position histories. But that's not quite ready for the public to start working on, so I'll save the discussion about that for a later Dev Diary post.