[Col] Decoding Colonization SAV files (1994 / Old / Classic)

C

carthanc

Chieftain
Joined
Dec 25, 2021
Messages
12
Classic Colonization Fanatics,

I am a programmer undertaking my own "voyage of discovery" to completely decode this game's SAV files. I have hacked together some utilities to get up to speed, but I've only cracked a few pieces of the SAV. For all I know, something may already exist which can do this!

I know of the following:
I do not regularly work in any of the languages above, but I'll be looking for answers there next. None of those repos have detailed READMEs, so I may start by submitting pull requests to each repos with the steps I take to make them work for me (if I can get them to work).

Again, I'm just starting out. If you can point me in the right direction, or if you share this dream, I'm eager for the help!

(Also, while I have broad curiosity in this area and appreciation for remakes such as Civ4:Col/2008, FreeCol, and Treasure Planet, I am looking for answers within this very narrow scope: 1994 Colonization, Patch 3.0, Decoding the SAV files.)
 
AyCe, thanks for replying. I can’t say how much I’ve appreciated your engagement and expertise with the Col94 topics I’ve researched. Your “playing as the natives” post, in response to Blake’s request, was an impressive flex. 10.0

I also just used your MADSPack tool — fantastically easy, by the way! — because the “marfsana/colsavs” code needed the graphical assets. I had to rename/organize the assets to match hard-coded expectations, but the actual MADSPack part was thoroughly painless.

Is that one file all the source code for your multiplayer tool?

Also, would you be interested in any pull requests from me? I can test, hunt bugs, document, whatever. I’m hoping to show tangible support for anyone else slinging code in this area.
 
Last edited:
Agreed, what you did for playing as the Indians was a stroke of brilliance. My son got some big laughs out of the video and thought it was awesome, and I have to say, seeing things like "Aztec Colonist" was exciting. It's a shame the game won't accommodate you really playing that way though.

I'm not a programmer myself, so can't be of tremendous help, and I might need some detailed instructions for using your tool(s). I'll be watching this whole thing with anticipation though. An assistant/companion/whatever for Colonization DOS would really be the cat's meow, and even something more modest that would give you, say, information on the native settlements you've visited would be a boon.
 
Last edited:
Between colsaves and viceroy, we have an outline for 50-75% of the SAV format (initial estimate). Those two efforts have automatic methods for locating colonies, units, villages, and map elements. They can also extract the sections for the overall game and player attributes. After that it get's tricky. Chunks within these outlined elements remain unknown -- some bytes in each section appear unused while others are changing without a clear purpose. Also, there are substantial areas of the SAV which are not outlined at all.

Unfortunately, village training and trade opportunities are not mapped yet. Most players keep a notepad for this data, so it is the most common-sense data to decode first. However, it may also be one of the harder ones to achieve. There is a recognizable flag for whether a village has "spent" its training (or not), but my testing shows that the "trading bytes" are changing within the "undiscovered" areas of the SAV. Moreover, it appears that at least ten data points change after a single trade, but I have not deciphered the pattern yet. The "debug supply/demand" info in cheat mode is pretty detailed, so I'd estimate at least a hundred data points feed each village's trade preferences, or at least the preferences for the tribe.

One immediate technical challenge is that colsaves and viceroy use different methods to outline the SAV data. Moreover, each one closely couples their data outlines to the code itself (python and C/C++, respectively). They are not using standard config files as inputs to the program's interpretation or in the program's output. (This type of standardized I/O is hard, so this isn't a surprise.) I've attempted two different mechanisms to directly bridge the two, but am still unsuccessful. The shortest path to unifying these efforts may be to enhance each with an "output to json/yaml" feature.

At this point, I am loathe to arbitrarily dictate/commit to one approach over the other, so I will keep pursuing a way to maximize the benefit of both.

Once I can unify the two outlines, it would be straightforward to measure our "SAV coverage" precisely. In addition, explorations into the "undiscovered" areas will be more rewarding, since they will push this metric higher. I will definitely target village training and trade when I start these explorations.

As an aside, ColonizationAssistant has yet another method -- in F# -- for outlining the SAV, but it only extracts some basic game attributes and the locations of the player's colonies. Where it shines is that it runs in the background and works with the SAV files as you play. It focuses on keeping revisions of each of the eight SAV slots. This means you could confidently use a single SAV slot for an entire playthrough -- you'd keep every save you made to that slot while playing. Definitely cool and something I want to integrate later... but off-track (for now) when it comes to a full decoding the SAV format.
 
Last edited:
carthanc said:
I also just used your MADSPack tool — fantastically easy, by the way! — because the “marfsana/colsavs” code needed the graphical assets. I had to rename/organize the assets to match hard-coded expectations, but the actual MADSPack part was thoroughly painless.
Click to expand...
Great to hear that. Most of the work was done by Zharramadar, I just made it "generally usable" and put in on Github. No animated water and so on, because this is done via palette shifting ingame.

carthanc said:
Is that one file all the source code for your multiplayer tool?
Click to expand...
This is all the code relevant to the savefile format. The rest of the code is only for the UI. But from what I can see, you already have the knowledge about european data in the other sources.

carthanc said:
Also, would you be interested in any pull requests from me? I can test, hunt bugs, document, whatever. I’m hoping to show tangible support for anyone else slinging code in this area.
Click to expand...
Sure, there is a lot to be done. Feel free to work on it if you want.

It might be that native supply/demand and/or education options are not saved, and are instead computed from the surrounding terrain. At least in the RNG-fixed exe, the offered jobs change randomly, so when you visit the same village with 3 colonists, you may be offered 3 different jobs. This might or might not happen in the normal ("RNG-broken") version, maybe because the RNG determines those, and is always seeded with the same value.

A full colo savegame explorer/editor (scenario editor?) would be awesome! I always love looking at normally hidden values and flags in games.
 
The more I dig into this the more I realize that some kind of GUI for guessing and testing out values will be necessary. There is a lot of decoding to do. We'll need some kind of GUI to make that process sane... and at that point we'll basically have an editor.

I'll look at hacking on ColonizationAssistant to see if I can use that GUI framework. This would also "bring it into the fold" so that we can increase the collaboration.
 
That one hadn't shown up on my searches. I'll be sure to digest it as well. Thanks!
 
I've dug through all four projects on Github and am trying to contribute to each as appropriate. Unfortunately, combining insights from all of them does not raise our "known" percentage significantly. I'm brute forcing my way through various tests (do an action, save, compare the files) to attempt to decipher each undiscovered byte. It's slow going.

AyCe said:
It might be that native supply/demand and/or education options are not saved, and are instead computed
Click to expand...

I am now certain you are right about this. I am pursuing a long-shot that I can determine the inputs (terrain, etc.) and maybe replicate what the EXE is computing. If we can manipulate the input and reliably predict the direct impact on the outputs, we might still be able to get to an "assistant" that can reliably predict what goods will be in demand and what will be provided for education.

Supply and Demand: I have deciphered that the whole tribe has "sold/bought stocks" that keeps track similar to the European/Economic advisor. This is stored in the "Indian Nation" section. There is a slot for each good and it goes positive or negative when you trade with any village in that tribe.

There is also a "decay" mechanic. Every turn the stocks move towards zero by some amount. I traded 50 muskets two turns previous, so the decay here is 2/turn. I observed the same decay with Tools:

-- indian --
Iroquois:capitol(42,25) tech(Agrarian)
00 00 00 00 02 00 00 00 00 06 00
Food(0) Sugar(0) Tobacco(0) Cotton(0) Furs(0) Lumber(0) Ore(0) Silver(0)
Horses(0) Rum(0) Cigars(0) Cloth(0) Coats(0) Trade goods(0) Tools(0) Muskets(48)
00 00 00 00 00 00 00 00 03 00 00 00
Eng_met( 96) Fra_met( 0) Spa_met( 0) Net_met( 96)
00 00 00 00 00 00 00 00
Eng_alarm( 9) Fra_alarm( 5) Spa_alarm( 6) Net_alarm( 1)​

Since the "sold/bought" stocks start at zero, there must be additional input somewhere that dictates the "initial demand" per good. My next step in this task is to manipulate the stock values for the tribe until I can trigger their shift to the next good. That will give me the initial demand and then I can test if that is global, randomized per game, or correlates with some other factor (like number of villages).

Last Trade: The last traded good is stored by village. This is a very simple mechanic that simply prevents that good from being offered for trade again at that village. I don't think it really affects the supply and demand other than that.

-- tribes --
[ 3] ( 42, 25):Iroquois: pop( 6) artillery(0) learned(0) capital(1) scouted(0) 0 0 0 0
mission(-1) 00 ff last_trade(Muskets) ff panic( 0) 00 00 00 00 00 04 lost_pop(00)​

Education Option: No new insights on this other than the previously-deciphered "learned" bit. Given my digging and testing, it is almost 100% certain that education is computed. I'm not sure the inputs yet. My next step on this task is to try to move villages on the map and change terrain to see if location and/or terrain is involved.

(Also, apologies for the slow progress. I had time to start this during the holidays but didn't get far before RL intruded.)
 
Supply/Demand and Education is also dependent if you use the RNG fix or not ( https://civilization.fandom.com/wiki/Bugs_(Col)#Random_Number_Generator ).
There are different goods and different education options depending on the EXE used. To a scout they might say they educate Master Sugar Planters, but then actually offer you something else. Try it enough times and you get different options.

At least with the RNG fixed EXE (possibly also without) the indians will sometimes bring you 2 tons of a good type you have already a full warehouse of, including muskets, horses, etc, even when they shouldn't have it. Possibly a bug in the RNG logic or in the "do NOT select this good" logic. No clue.

Initial demand and education options are most likely based on the civilization level of the indians too. Possibly even hardcoded to the tribe index. Apache always offer scouts, Tupi always offer planters, Arawak and Inca always offer Farmers/Fishers, and so on.
 
AyCe said:
Supply/Demand and Education is also dependent if you use the RNG fix or not ( https://civilization.fandom.com/wiki/Bugs_(Col)#Random_Number_Generator ).
There are different goods and different education options depending on the EXE used. To a scout they might say they educate Master Sugar Planters, but then actually offer you something else. Try it enough times and you get different options.
Click to expand...

I've seen that too! So it was the result of using the RNG fix - I had wondered. I'd better update my Col fan site to mention this.
 
With so many related efforts prior to mine, my strategy is to help push the other efforts forward as much as possible. Otherwise, I'd feel like I'm just adding noise.

For now, I've been working primarily on viceroy, so you could watch that for the time being. I appreciate the interest! I have a lot of un-pushed work that I will clean up and publish shortly.

That repo is forked from eb4x/viceroy which perfectly replicated my current mission of decoding the SAV file. (I went "off the rails" and put a lot of time into trying to implement JSON support; but I was not too happy with the results. It helped me learn C/C++, though.) I will move my experimental JSON changes into its own feature so that I can create a clean merge request. This project is in C++, so compiling it so that you can use it is a pain if you're not familiar with that (I was not).

The other branch that has been helpful has been colsaves. It doesn't decode as much as viceroy, but it decoded more of the map. The map is important for more than rendering the graphics. Data stored in the map is critical to game functions as well. My attempts to move a village have been fruitless so far because changing the village section does not move the tile where all the village functions occur. That means that village behavior is not only in the EXE, but shared with data stored in the map as well. This renews my hope a little that I can find the triggers for trade and education... but decoding the rest of the map has been very difficult.

(I also submitted a Readme for ColonizationAssistant and it was merged. Not a lot there related to decoding SAV files, though.)

My biggest hope, long term, is to advance institution/cc94 with the ability to read SAV files from the original game -- and render them. It's C++, too, so it should not be too hard to integrate with eb4x/viceroy.

If you haven't seen http://institution.github.io/cc94/cc94.html, you should definitely take a look (assuming you already have the image files from purchasing the game). It renders the map details quite well.
 
Last edited:
Wow, I wasn't aware of this ColonizationAssistant! Do you know more about it what it is, beyond "A tool to assist with the gameplay"? It sounds like it might have the potential to be for Col what the Gold Box Companion (https://gbc.zorbus.net/) was for the old D&D games.

Sadly, I don't think I can run it since I'm a Linux user.

C++? Say, if you're ever looking for another project down the road, you might consider joining the Race Into Space group (https://github.com/raceintospace/raceintospace). Unlike with a lot of DOS games (inc. Col), we have access to the source code and can make significant changes to the game itself.
 
Bladrov said:
Wow, I wasn't aware of this ColonizationAssistant! Do you know more about it what it is, beyond "A tool to assist with the gameplay"? It sounds like it might have the potential to be for Col what the Gold Box Companion (https://gbc.zorbus.net/) was for the old D&D games.
Click to expand...

This is what I put in the README:

Features​
Keep ColonizationAssistant running in the background while you play. Each time you save normally within the game, ColonizationAssistant will create a revision in a directory you specify. Every SAV file gets it's own set of revisions, which means you can confidently keep re-using the same save game slot. You also now have no real limit on the number of saves, instead of only having the eight saves allowed by the game.

In addition, ColonizationAssistant shows provides a GUI for navigating the backups. It will visualize the data in each save by showing a map of your colonies when you select each file.

Bladrov said:
It sounds like it might have the potential to be for Col what the Gold Box Companion (https://gbc.zorbus.net/) was for the old D&D games.
Click to expand...

I don't think we have anything close to that yet. Just a few simple tools. The most advanced is institution/cc94, which you can see here. It can render the map graphics correctly, so if I can get it to read a SAV, that would be a key feature we'd need in a companion tool.

The other tool I'd want the companion tool to emulate is JCivED, which runs on windows (I use this extensively). Never seen "Race Into Space" before. Looks interesting, but I'm trying to stay focused on decoding these SAV files. :)
 
Well, leave it to me to miss the obvious! Sorry about that.

Huh, I didn't know about JCivED either. It looks fairly impressive.

Yes, RIS is a whole different ball of wax - a re-creation of the US-Soviet Space Race in the 60s. I only brought it up because you mentioned C++ and I'm always on the lookout for someone who might be interested in working on the game. We're long overdue for a release and our other developers don't seem in a hurry to complete one.
 
Last edited:
Yep. It would be awesome if we had a tool with the same features, but for Colonization.
 
You must log in or register to reply here.

Similar threads

P
[Col] Thoughts on possible Colonization (1994) mod utility
Replies
5
Views
538
Blake00
Blake00
P
[Col] Colonization (1994) SAV files decoder and encoder
2 3 4
Replies
61
Views
10K
Bladrov
Bladrov
Blake00
  • Sticky
[Col] ATTENTION: Sid Meier's Colonization 30th Anniversary Celebration - Communities, Fan Mods & Remakes, Similar Games & More!
Replies
9
Views
4K
Bladrov
Bladrov
Top Bottom