[Col] Colonization (1994) SAV files decoder and encoder

pavelbel

Chieftain
Joined
Jan 9, 2023
Messages
60
UPD 23.10.2023: Release v1.3: Editor: Adjust Royal Expeditionary Force size, add 4-th, 5-th etc specialist to manufactures. Enc/Decoder: AUTO UPDATE mode.

UPD 06.03.2023: Release v1.2: Assimilate Indian converts or promote them to scouts, pioneers or warriors.

UPD 20.02.2023: Release v1.1: upgrade warehouse level above 2 & clear and plow tiles under AI's colonies.

UPD 18.02.2023: Release v1.0 ready. It features full-fledged decoder/encoder enc_decode_sav and basic editor smcol_sav_editor (by now it just removes fortifications and plants forests). Windows x64 binaries included, but it's recommended to install Python and use utilities directly in .py format.

Hello fellow colonizators!

Some time ago I've returned to playing classic Colonization and once again became upsed by inability to abandon colony with any fortification built. But this time I decided to not put up with it and created a simple ulitily for removing fortifications from any colony. Then I realized that removing stokades wasn't the only feature I wanted to implement. Then I've found this forum where people performed great efforts to decode SAV files (here https://forums.civfanatics.com/threads/decoding-colonization-sav-files-1994-old-classic.674707/ and links in it's posts). I was greatly inspired by these amazing works and decided to try to create my own ulitily to study and edit SAV files, using the mapping, discovered by my predecessors.

So why create a new SAV ulitity, if we already have colsaves, viceroy, Colonization-SAV-files, Hot Seat Multiplayer Savegame Editor and maybe others (sorry if I forgot to mention). Here's why:
1) carthanc mentioned that each of the existing utilities...
closely couples their data outlines to the code itself (python and C/C++, respectively). They are not using standard config files as inputs to the program's interpretation or in the program's output.
And it is really a problem! If you are studying the SAV structure and want to alter it, you must dig into the code of the utility and the programming language it is written with. That's not user friendly.

2) OK, with colsaves you can decode SAV file according to it's structure outline, save it to txt format and exctract some information from it, but what next? You cannot edit it and receive a modified SAV file to load it and continue playing.

That's why I'm working on Sid Meier's Colonization (1994) SAV files utility pack. By now it includes:

1) smcol_sav_struct.json file. It outlines a SAV file structure in universal user friendly format. Note: I've extracted all (well, not all, about 95%) the structure data from viceroy project. Credits to eb4x, hegemogy and their predecessors!

Here is the example of how a UNIT section is outlined:
Spoiler UNIT section description :

JSON:
"UNIT":
{
    "count": "unit_count",
    "struct":
    {
        "x, y": {"size": 2, "type": "coords"},
        "type": {"size": 1, "type": "unit_type"},
        "nation_info":
        {
            "bit_struct":
            {
                "nation_id": {"size": 4, "type": "nation_4bit_type"},
                "unknown14": {"size": 4}
            }
        },
        "unknown15": {"size": 1},
        "moves": {"size": 1, "type": "int"},
        "unknown16": {"size": 2},
        "orders": {"size": 1},
        "goto_x": {"size": 1, "type": "int"},
        "goto_y": {"size": 1, "type": "int"},
        "unknown18": {"size": 1},
        "holds_occupied": {"size": 1, "type": "int"},
        "cargo_items":
        {
            "count": 3,
            "bit_struct":
            {
                "cargo_1": {"size": 4, "type": "cargo_type"},
                "cargo_2": {"size": 4, "type": "cargo_type"}
            }
        },
        "cargo_hold": {"size": 1, "cols": 6, "type": "int"},
        "turns_worked": {"size": 1, "type": "int"},
        "profession": {"size": 1, "type": "profession_type"},
        "transport_chain":
        {
            "struct":
            {
                "next_unit_idx": {"size": 2, "type": "int"},
                "prev_unit_idx": {"size": 2, "type": "int"}
            }
        }
    }
}
Here you can see a section name UNIT, number of it's entries ("count": "unit_count" - value taken from HEAD section) and it's structure description ("struct": {...}). It has members of several types: simple ("orders": {"size": 1} - one byte field in hex format), typed simple ("goto_x": {"size": 1, "type": "int"} - one byte value of type int), fields with lookup-types ("profession": {"size": 1, "type": "profession_type"} - one byte field with "profession" type, it is described in __metadata section of the JSON file), arrays ("cargo_hold": {"size": 1, "cols": 6, "type": "int"} - 6 items array of int values, each of 1 byte size) and bit struct fields ("cargo_items": {...} - 3 item array of pairs of 4 bits values of lookup-type "cargo_type") and others. You can explore the JSON file and understand it, it's quite simple. Note: "size" attribute value means bytes count in "struct" and bits count in "bit_struct" fields.

2) smcol_sav_converter.py - Python module for reading SAV data according to structure outline in smcol_sav_struct.json and converting it to human readable and editable SAV.JSON format. Do not use it directly.

3) enc_decode_sav.py - a utility for decoding SAV files to SAV.JSON and vice versa. That's your main tool.

To use it install Python (and then bitarray module) place these 3 files to COLONIZE folder and run enc_decode_sav.py.

Sample use. Imagine, you want to remove a fortification from a colony "Jamestown" in a SAV file COLONY07.SAV. Here are the steps:

1) Run enc_decode_sav.py and choose COLONY07.SAV file. The new COLONY07.SAV.json file will be created
2) Open COLONY07.SAV.json with any text editor (VS Code, Brackets or so recommended, but not nessesary) and search for "Jamestown" string. You will go to the Jamestown colony's section.
3) Find it's member "buildings" and submember "fortification".
4) Set "fortification" value to "0", save file and close it.
5) Run enc_decode_sav.py again and choose COLONY07.SAV.json file. The new COLONY07.SAV.enc file will be created.
6) Remove '.enc' from the end of this file's name, run Colonization and load it.
7) That's it! No more stockade in Jamestown. You can abandon it. It's God's will!

The utility is quite functional. Decode/encode your SAV files, study and edit them, modify structure outline in smcol_sav_struct.json. It's SAV exploration time!)

The drawbacks:
1) Some errors are not processed correctly
2) MAPS section representation is not so descriptive. Working on a special format for them.

Bug reports are appreciated.

P.S. Sorry for my english. Not native speaker I am.
 
Last edited:
Here's the example of a wagon train unit section in SAV.JSON file created according to the structure description above:
Spoiler Wagon train json data :

JSON:
{
    "x, y": "34, 55",
    "type": "Wagon train",
    "nation_info": {
        "nation_id": "France",
        "unknown14": "0010"
    },
    "unknown15": "00",
    "moves": 6,
    "unknown16": "01 34",
    "orders": "0B",
    "goto_x": 37,
    "goto_y": 52,
    "unknown18": "00",
    "holds_occupied": 1,
    "cargo_items": [
        {
            "cargo_1": "ore",
            "cargo_2": "food"
        },
        {
            "cargo_1": "food",
            "cargo_2": "food"
        },
        {
            "cargo_1": "food",
            "cargo_2": "food"
        }
    ],
    "cargo_hold": [
        100,
        0,
        -1,
        0,
        1,
        0
    ],
    "turns_worked": 0,
    "profession": "Expert farmer",
    "transport_chain": {
        "next_unit_idx": -1,
        "prev_unit_idx": -1
    }
}
 
Very cool. Are you going to do a full spec documentation of the map format that people can contribute to? Or are we going to have to dig through code again to understand it? :D

Then people (like me? :p) could make for example a C# library that uses this spec to read/write savegame files into usable data structures, allowing other developers to write tools without having to reverse-engineer or write I/O code again.
 
Very cool. Are you going to do a full spec documentation of the map format that people can contribute to? Or are we going to have to dig through code again to understand it? :D
Thanks for your reaction! Full spec docs is a good and useful thing. I think I'll do it sometime, but that's not for sure) The map format is not absolutely stable and optimal for now. It's my priority to fulfill and optimize it. So yes, for now you have to dig through JSON code. But the good news is: python code of my ulitilies is absolutely universal, you won't find any mention of colonies, units, nations or tribes there, so you don't have to study it. All the info about Colonization's SAV file structure is located inside the smcol_sav_struct.json file, described with simple JSON data types.
 
Excelent!
I was going to ask you to host it on github but I could get to your repo through the download links.
Is the json format standardized? Are there binary metadata formats that could be applied to the the saves?
 
Is the json format standardized? Are there binary metadata formats that could be applied to the the saves?
Sorry, I think I don't understand you correctly. Is JSON format standardized? The JSON itself surely is. Or my attempt to define the SAV files structure in terms of JSON? I'm working on it right now. I think it's about 90% stable. If only I won't receive an unexpected insight of how to make it way more optimal.

And I completely don't understand what you mean by "binary metadata formats". Explain plz
 
I thought that maybe smcol_sav_struct.json was written in a standard format, but I now see that smcol_sav_converter.py interprets the smcol_sav_struct.json by hand.
 
Helpful information for Linux and Mac users: these utilities can run in Wine, but you have to open them from a command prompt.

On my system I had to open Wine configuration, and for the Default Settings, go to the Drives tab. Then set D: to wherever folder your COLONIZE directory is in. On my system I have it under ~/dos (it's /home/leon/dos/COLONIZE). So I set D: in Wine to /home/leon/dos/.

Be sure to modify the smcol_sav_settings.json file to point to the location (in my case, "D:/COLONIZE"). Yes, you use a forward slash, not a backslash.

Then open a terminal window wherever you have the utility, and run wine smcol_sav_editor.exe or wine enc_decode_sav.exe.
 
P.S. Sorry for my english. Not native speaker I am.

Not to worry, your English is good. The only thing I'd suggest is its vs. it's. Its is a pronoun, like his/her, and never has an apostrophe. It's is a shortening of "it is" or "it has" and the apostrophe is there to stand in for the missing letters. And its' doesn't work at all in English.

You're in good company though - way too many native speakers get these wrong, somehow.
 
Not to worry, your English is good. The only thing I'd suggest is its vs. it's. Its is a pronoun, like his/her, and never has an apostrophe. It's is a shortening of "it is" or "it has" and the apostrophe is there to stand in for the missing letters. And its' doesn't work at all in English.

You're in good company though - way too many native speakers get these wrong, somehow.
Okey, thanks, I'll take note)
 
Some time ago I've returned to playing classic Colonization and once again became upsed by inability to abandon colony with any fortification built. But this time I decided to not put up with it and created a simple ulitily for removing fortifications from any colony. Then I realized that removing stokades wasn't the only feature I wanted to implement. Then I've found this forum where people performed great efforts to decode SAV files (here https://forums.civfanatics.com/threads/decoding-colonization-sav-files-1994-old-classic.674707/ and links in it's posts). I was greatly inspired by these amazing works and decided to try to create my own ulitily to study and edit SAV files, using the mapping, discovered by my predecessors.
This is awesome! Great work mate!
 
That's pretty strange Pbsite. I admire the efforts you're putting into this; it's already more than I had hoped to be able to do myself.

I had an out-of-box moment this morning and thought maybe I could track down Brian Reynolds himself to ask if he remembers. But all I found for him was his Linkedin account, and you have to upgrade to Premium to be able to message him.

Of course, the credits at the back of the manual also show Jeff Briggs and Douglas Caspian-Kaufman in the game design. Of course, the credits don't give any indication who was responsible for how the Indian villages work or anything like that. Anyhow I managed to find contacts for both of them and have sent them a message. Probably won't hear back, but seems worth a shot. Kaufman and I share a Facebook friend; maybe that'll help. ;)
 
Last edited:
Of course, the credits at the back of the manual also show Jeff Briggs and Douglas Caspian-Kaufman in the game design. Of course, the credits don't give any indication who was responsible for how the Indian villages work or anything like that. Anyhow I managed to find contacts for both of them and have sent them a message. Probably won't hear back, but seems worth a shot. Kaufman and I share a Facebook friend; maybe that'll help. ;)

Thank you for the attempts. We'll see what they can say. Though I do not put any hope in it. The guy Zharramadar on this forum tried to contact Reynolds in 2008 or even earlier to ask some questions about another issue related to Colonization ( https://forums.civfanatics.com/threads/colonization-picture-viewer.256665/post-6365789 ). And Reynolds replied that he didn't remember anything - and it was 15 years ago! I'm sure none of the original developers remember anything about the code they had written 30 years ago. I'm a professional programmer too and I know from my own experience how quickly I forget the ideas I put into code after switching to another project.
 
Last edited:
D'ohhh, that'd be the same thing, wouldn't it? Or--wait, would it? Would mounted braves function as scouts? Or would they simply attack Indian settlements?
Mounted braves act like all other military units: they will attack. However we can promote Indian converts to regular scouts. I think I'll add this option. I've also been thinking about equipping them with tools, that is turning them to pioneers. Maybe will lesser tools capacity, 60 instead of 100 for example.

I have been active in GitHub elsewhere (as peyre), but I forget just now how to set up a pull request. Here's the file, for now. I had to change the extension to .txt for civfanatics to accept it.
Thank you very much! I've updated the readme (and tagged you).

P.S. For now the script clears and plows tiles right under the AI's colonies, not around them. The AI sometimes plows tiles around its colonies, but never the colony's tile itself.
 
Last edited:
Thanks! I think Indian scouts are an especially good idea since historically, Indians were used as scouts quite a lot by the Army.

Version 1.2 released. New features:
- Plant forest requires a pioneer on the tile, and he spends tools on planting (adjustable in settings)
- Assimilate Indian converts
- Arm/equip Indian converts (promote them to scouts, warriors or pioneers)
- Repair damaged artillery
 
UPD 06.03.2023: Release v1.2: Assimilate Indian converts or promote them to scouts, pioneers or warriors.

UPD 20.02.2023: Release v1.1: upgrade warehouse level above 2 & clear and plow tiles under AI's colonies.

UPD 18.02.2023: Release v1.0 ready. It features full-fledged decoder/encoder enc_decode_sav and basic editor smcol_sav_editor (by now it just removes fortifications and plants forests). Windows x64 binaries included, but it's recommended to install Python and use utilities directly in .py format.

Hello fellow colonizators!

Some time ago I've returned to playing classic Colonization and once again became upsed by inability to abandon colony with any fortification built. But this time I decided to not put up with it and created a simple ulitily for removing fortifications from any colony. Then I realized that removing stokades wasn't the only feature I wanted to implement. Then I've found this forum where people performed great efforts to decode SAV files (here https://forums.civfanatics.com/threads/decoding-colonization-sav-files-1994-old-classic.674707/ and links in it's posts). I was greatly inspired by these amazing works and decided to try to create my own ulitily to study and edit SAV files, using the mapping, discovered by my predecessors.

So why create a new SAV ulitity, if we already have colsaves, viceroy, Colonization-SAV-files, Hot Seat Multiplayer Savegame Editor and maybe others (sorry if I forgot to mention). Here's why:
1) carthanc mentioned that each of the existing utilities...

And it is really a problem! If you are studying the SAV structure and want to alter it, you must dig into the code of the utility and the programming language it is written with. That's not user friendly.

2) OK, with colsaves you can decode SAV file according to it's structure outline, save it to txt format and exctract some information from it, but what next? You cannot edit it and receive a modified SAV file to load it and continue playing.

That's why I'm working on Sid Meier's Colonization (1994) SAV files utility pack. By now it includes:

1) smcol_sav_struct.json file. It outlines a SAV file structure in universal user friendly format. Note: I've extracted all (well, not all, about 95%) the structure data from viceroy project. Credits to eb4x, hegemogy and their predecessors!

Here is the example of how a UNIT section is outlined:
Spoiler UNIT section description :

JSON:
"UNIT":
{
    "count": "unit_count",
    "struct":
    {
        "x, y": {"size": 2, "type": "coords"},
        "type": {"size": 1, "type": "unit_type"},
        "nation_info":
        {
            "bit_struct":
            {
                "nation_id": {"size": 4, "type": "nation_4bit_type"},
                "unknown14": {"size": 4}
            }
        },
        "unknown15": {"size": 1},
        "moves": {"size": 1, "type": "int"},
        "unknown16": {"size": 2},
        "orders": {"size": 1},
        "goto_x": {"size": 1, "type": "int"},
        "goto_y": {"size": 1, "type": "int"},
        "unknown18": {"size": 1},
        "holds_occupied": {"size": 1, "type": "int"},
        "cargo_items":
        {
            "count": 3,
            "bit_struct":
            {
                "cargo_1": {"size": 4, "type": "cargo_type"},
                "cargo_2": {"size": 4, "type": "cargo_type"}
            }
        },
        "cargo_hold": {"size": 1, "cols": 6, "type": "int"},
        "turns_worked": {"size": 1, "type": "int"},
        "profession": {"size": 1, "type": "profession_type"},
        "transport_chain":
        {
            "struct":
            {
                "next_unit_idx": {"size": 2, "type": "int"},
                "prev_unit_idx": {"size": 2, "type": "int"}
            }
        }
    }
}
Here you can see a section name UNIT, number of it's entries ("count": "unit_count" - value taken from HEAD section) and it's structure description ("struct": {...}). It has members of several types: simple ("orders": {"size": 1} - one byte field in hex format), typed simple ("goto_x": {"size": 1, "type": "int"} - one byte value of type int), fields with lookup-types ("profession": {"size": 1, "type": "profession_type"} - one byte field with "profession" type, it is described in __metadata section of the JSON file), arrays ("cargo_hold": {"size": 1, "cols": 6, "type": "int"} - 6 items array of int values, each of 1 byte size) and bit struct fields ("cargo_items": {...} - 3 item array of pairs of 4 bits values of lookup-type "cargo_type") and others. You can explore the JSON file and understand it, it's quite simple. Note: "size" attribute value means bytes count in "struct" and bits count in "bit_struct" fields.

2) smcol_sav_converter.py - Python module for reading SAV data according to structure outline in smcol_sav_struct.json and converting it to human readable and editable SAV.JSON format. Do not use it directly.

3) enc_decode_sav.py - a utility for decoding SAV files to SAV.JSON and vice versa. That's your main tool.

To use it install Python (and then bitarray module) place these 3 files to COLONIZE folder and run enc_decode_sav.py.

Sample use. Imagine, you want to remove a fortification from a colony "Jamestown" in a SAV file COLONY07.SAV. Here are the steps:

1) Run enc_decode_sav.py and choose COLONY07.SAV file. The new COLONY07.SAV.json file will be created
2) Open COLONY07.SAV.json with any text editor (VS Code, Brackets or so recommended, but not nessesary) and search for "Jamestown" string. You will go to the Jamestown colony's section.
3) Find it's member "buildings" and submember "fortification".
4) Set "fortification" value to "0", save file and close it.
5) Run enc_decode_sav.py again and choose COLONY07.SAV.json file. The new COLONY07.SAV.enc file will be created.
6) Remove '.enc' from the end of this file's name, run Colonization and load it.
7) That's it! No more stockade in Jamestown. You can abandon it. It's God's will!

The utility is quite functional. Decode/encode your SAV files, study and edit them, modify structure outline in smcol_sav_struct.json. It's SAV exploration time!)

The drawbacks:
1) Some errors are not processed correctly
2) MAPS section representation is not so descriptive. Working on a special format for them.

Bug reports are appreciated.

P.S. Sorry for my english. Not native speaker I am.
Would you please provide clearer instructions for installing the bitarray module for us non-programmers. I have Python installed but cannot get past the bitarray installation. Thanks in adavance.
 
Top Bottom