[Col] Colonization (1994) SAV files decoder and encoder

[pavelbel]

Would you please provide clearer instructions for installing the bitarray module for us non-programmers. I have Python installed but cannot get past the bitarray installation. Thanks in adavance.

Hello jaeenox! Try console command:
pip install bitarray
It's a console command, not Python script.

 

[Bladrov]

That ran successfully in a terminal window on my Linux machine, but may not work in a command prompt on a Windows box.

 

[Bladrov]

I have a question. I'm now trying to reforest a square next to a former AI colony that I just conquered. I get that you need to have a pioneer with tools in the square to be reforested, but why does it demand a Hardy Pioneer with 100 tools? It's not uncommon that I don't have a hardy pioneer at all, including in the current game. And why does the pioneer need to have 100 tools for the job?

 

[pavelbel]

I have a question. I'm now trying to reforest a square next to a former AI colony that I just conquered. I get that you need to have a pioneer with tools in the square to be reforested, but why does it demand a Hardy Pioneer with 100 tools? It's not uncommon that I don't have a hardy pioneer at all, including in the current game. And why does the pioneer need to have 100 tools for the job?

Hi Leon! Glad to hear you use my tool) It's my decision to make reforesting rather expensive. AI cannot do it, so this feature is the player's advantage and it shouldn't be cheap. In real life it's not as expensive, as it is time consuming. It's not possible (for a SAV files editor) to make pioneers work on a tile for 10 turns or so and produce forest. That's why I've decided that only a skilful pioneer could perform this hard task and spend lots of tools on it.

Anyway you can manually adjust it. Edit plant_forest_tools_cost variable in smcol_sav_settings.json file to set a reasonable reforesting price in tools. If set to 0 a pionner is not needed at all.

To make any pioneer able to perform reforesting open smcol_sav_editor.py file and delete line 228 (and unit['profession_or_treasure_amount'] == FIELD_VALUES['profession_type']['hardy pioneer']\).

P.S. I've got an new idea about reforesting tiles near colonies! On the colony screebn you can assign an expert lumber jack to work (as lumber jack) on a tile without forest (the game allows it), and after several turns (~10 or so) it will be possible to "grow" forest there (and after that it will immediately start to produce lumber)! That's how it will become more realistic. And I'm sure you have expert lumber jacks in every game.

 

[Bladrov]

Hey thanks! However there isn't an smcol_sav_editor.py file. I redownloaded the utility to make sure, and there isn't one. The only files in there are:
enc_decode_sav.exe
readme.txt
smcol_sav_editor.exe
smcol_sav_settings.json
smcol_sav_struct.json
The cost of 100 tools doesn't really bother me; I was just surprised by it.

That's a great idea, using a lumberjack to replant forest that way. I get that Expert Lumberjacks are common, but maybe non-experts could replant forest as well, but take twice as long or something. That would be in keeping with how the game works in general.

 

[pavelbel]

Hey thanks! However there isn't an smcol_sav_editor.py file.

Ah, I see... I thought you're using the utility in its native (Python) form under Linux. You'll need to install Python and bitarray package (seems that you've done it already), download smcol_saves_utility sources and manually edit and run smcol_sav_editor.py file.

That's a great idea, using a lumberjack to replant forest that way. I get that Expert Lumberjacks are common, but maybe non-experts could replant forest as well, but take twice as long or something. That would be in keeping with how the game works in general.

Ok. I'll think of it.

 

[Bladrov]

Thanks. The .exe was just easier to run. I've tried running python3 smcol_sav_editor.py, but get the following:

Traceback (most recent call last):
File "/home/leon/dos/COLONIZE/smcol_saves_utility-py/smcol_sav_editor.py", line 5, in <module>
from smcol_sav_converter import handle_metadata, read_sav_structure, dump_sav_structure, REL_VER
File "/home/leon/dos/COLONIZE/smcol_saves_utility-py/smcol_sav_converter.py", line 7, in <module>
from bitarray import bitarray, util
ModuleNotFoundError: No module named 'bitarray'

 

[pavelbel]

Thanks. The .exe was just easier to run. I've tried running python3 smcol_sav_editor.py, but get the following:

Traceback (most recent call last):
File "/home/leon/dos/COLONIZE/smcol_saves_utility-py/smcol_sav_editor.py", line 5, in <module>
from smcol_sav_converter import handle_metadata, read_sav_structure, dump_sav_structure, REL_VER
File "/home/leon/dos/COLONIZE/smcol_saves_utility-py/smcol_sav_converter.py", line 7, in <module>
from bitarray import bitarray, util
ModuleNotFoundError: No module named 'bitarray'

Module bitarray is absent. Try shell command pip install bitarray

 

[Bladrov]

I've tried, but it returns:

Defaulting to user installation because normal site-packages is not writeable
Requirement already satisfied: bitarray in ./.local/lib/python3.10/site-packages (2.8.1)

 

[pavelbel]

I've tried, but it returns:

Defaulting to user installation because normal site-packages is not writeable
Requirement already satisfied: bitarray in ./.local/lib/python3.10/site-packages (2.8.1)

Try install it under root user

 

[Bladrov]

Should've thought to mention this, but sudo pip install bitarray also returns an error--I didn't check to see if it was the same one though.

Edit: Yes, it was different. That returns:

Requirement already satisfied: bitarray in /usr/local/lib/python3.10/dist-packages (2.8.1)
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv

 

[Bladrov]

Hmph! I tried installing python3-bitarray in Synaptic, but it still returns

Traceback (most recent call last):
File "/home/leon/dos/COLONIZE/smcol_saves_utility-py/smcol_sav_editor.py", line 5, in <module>
from smcol_sav_converter import handle_metadata, read_sav_structure, dump_sav_structure, REL_VER
File "/home/leon/dos/COLONIZE/smcol_saves_utility-py/smcol_sav_converter.py", line 7, in <module>
from bitarray import bitarray, util
ModuleNotFoundError: No module named 'bitarray'

I ran pip uninstall bitarray and pip install bitarray, but I'm still getting the message. There must be something wrong with the python installation on system, I guess.

 

[pavelbel]

Hmph! I tried installing python3-bitarray in Synaptic, but it still returns

I've tested it on Linux in VirtualBox and it was ok. Let's try with virtual environment.

1. Go to the folder /home/leon/dos/COLONIZE/smcol_saves_utility-py/ and run python3 -m venv ./venv. See if there were any errors here. If ok...
2. Run source venv/bin/activate
3. Run pip install bitarray
4. If everything above is ok run python3 -m smcol_sav_editor

If it works you'll need to run source venv/bin/activate every time you open console window to run smcol_sav_editor

 

[Bladrov]

Ok wow, this morning I did a complete removal of python3, and that seems to have messed up my system. I finally got back up and running, and am experiencing the same issues with smcol_sav_editor. I tried what you suggest above, and it returned:

WARNING: pip is configured with locations that require TLS/SSL, however the ssl module in Python is not available.
WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError("Can't connect to HTTPS URL because the SSL module is not available.")': /simple/bitarray/
WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError("Can't connect to HTTPS URL because the SSL module is not available.")': /simple/bitarray/
WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError("Can't connect to HTTPS URL because the SSL module is not available.")': /simple/bitarray/
WARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError("Can't connect to HTTPS URL because the SSL module is not available.")': /simple/bitarray/
WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError("Can't connect to HTTPS URL because the SSL module is not available.")': /simple/bitarray/
Could not fetch URL https://pypi.org/simple/bitarray/: There was a problem confirming the ssl certificate: HTTPSConnectionPool(host='pypi.org', port=443): Max retries exceeded with url: /simple/bitarray/ (Caused by SSLError("Can't connect to HTTPS URL because the SSL module is not available.")) - skipping
ERROR: Could not find a version that satisfies the requirement bitarray (from versions: none)
ERROR: No matching distribution found for bitarray
WARNING: pip is configured with locations that require TLS/SSL, however the ssl module in Python is not available.
Could not fetch URL https://pypi.org/simple/pip/: There was a problem confirming the ssl certificate: HTTPSConnectionPool(host='pypi.org', port=443): Max retries exceeded with url: /simple/pip/ (Caused by SSLError("Can't connect to HTTPS URL because the SSL module is not available.")) - skipping
WARNING: There was an error checking the latest version of pip.

Do I maybe need to install python3-socksipychain or python3-certifi?

 

[pavelbel]

Ok wow, this morning I did a complete removal of python3, and that seems to have messed up my system.

Sorry, I really don't know. I'm not an experienced linux user( There is some problem with python installation: pip (package manager) fails to reach pypi.org to download bitarray module.

 

[Bladrov]

I did a complete removal of python3, and that trashed my network settings. In the end there was nothing for it but to wipe the drive and reinstall from scratch. On the positive side, I was able to get the python script to run. For reference, the pointer in smcol_sav_settings.json for a Linux system should be the absolute path to where Col is located:
"colonize_path": "/home/leon/dos/COLONIZE"
not
"colonize_path": "~/dos/COLONIZE"

So I got it to work, inputted the coordinates for the square to be forested, and it said it was now forested. Yet when I reloaded the save, the square is still Prairie, and my (non-hardy) pioneer on that square still has all his tools. I ended turn, thinking the tile might change, but next turn it still says Prairie and I still have the tools. If I rerun the utility and tell it again to forest the square, it tells me it's already forested.

 

[pavelbel]

So I got it to work, inputted the coordinates for the square to be forested, and it said it was now forested. Yet when I reloaded the save, the square is still Prairie, and my (non-hardy) pioneer on that square still has all his tools. I ended turn, thinking the tile might change, but next turn it still says Prairie and I still have the tools. If I rerun the utility and tell it again to forest the square, it tells me it's already forested.

That's so strange... Could you please send me the sav file you are editing? Attach it to a new post. And what tile (coords) you want to reforest.

 

[Bladrov]

Got it up: https://forums.civfanatics.com/threads/reforesting-squares-without-hardy-pioneer.685734/

 

[pavelbel]

Got it up: https://forums.civfanatics.com/threads/reforesting-squares-without-hardy-pioneer.685734/

Thanks. I've removed line 228 from smcol_sav_editor.py and performed reforesting square (48,49). And everything went ok: broadleaf forest appeared and the pioneer turned to colonist indentured servant. See screenshot attached.

I think it may be some problem with the virtual machine you are running to play Col: it doesn't notice changes of files from "outside" (made by host OS - your Linux). So, you'll have to use my win x64 binaries our use another virtual machine...

P.S. I'll make "hardy pioneer" a toggleable option

 

[Bladrov]

Fantastic! I'll redownload and try again tonight.

I'm actually not using a VM at all--Colonization runs beautifully in DOSBox. I can even double the size of the screen, making everything bigger for the benefit of my tired eyes. But if it's blocking the change somehow, that could be a problem. A lot of people run Col in DOSBox.