[Col] re:Colonization - Sid Meier's Colonization but with QoL updates and bugfixes

[personaalunioon]

What is this?
It's a dis- and re-assembling project that aims to improve the quality-of-life of and hopefully fix some bugs along the way in the original DOS 3.0 version of Sid Meier's Colonization.

What state is it in?
It compiles. It should be playable. I've tested it on DosBox-Staging on Linux and it seems to work.
Obviously, I cannot make any promises or give any guarantees. Give it a try at your own risk.

How does it differ from the original?
It's based on the disassembly of VICEROY.EXE, with some changes:

• The binary has been renamed to colonize.exe
• Most original switches have been disabled (except for /p)
• Only DOS-style switches are accepted ("/"), whereas the original also accepted *nix style ("-")
• The opening and closing animations do not work
• The /p switch now also supports sound effects, whereas the original only supported music
• ... some other boring stuff (.RTLINK stripped)

Edit: Actually, it seems I forgot to commit that striked change, so currently *nix style switches are also accepted. Oh well...

How to play?
Download it, unpack it with tar -xzf recol.tar.gz.Then run it according to the instructions.

Prerequisites

• DosBox (-Staging, -X, ...?) with same (?) settings you use to run the original game (ems=true xms=true umb=true, sound set up for the specific driver etc.)
• Original game files (duh!) of the '94 DOS game (3.0)
• Patience (there's never enough of it)

The next steps depend on where you dropped the binary. Note that the game (by design) when the /p switch is in effect can only access game data files from the remote game dir, not user data files (saves, maps, config).

Run the game

If you plopped it into a dedicated directory (recommended!), then first copy-paste or create a symlink to CONFIG.COL. Without it, sound/music won't work. Then copy over any save files or map files you want to access. Then you can run colonize /p:<path-to-original-gamedir>. The /p switch by the way is also present in the original game but I doubt many used it.

If you put it into the original game directory (not recommended), then you can simply run colonize.

If it doesn't run or if you notice something is off, do let me know your setup/configuration and any error messages you might have received.

Future plans?
You tell me. I hope to revamp the saving mechanism (why only 8 slots?) and in general improve the QoL of this old classic.

 

[Blake00]

What is this?
It's a dis- and re-assembling project that aims to improve the quality-of-life of and hopefully fix some bugs along the way in the original DOS 3.0 version of Sid Meier's Colonization.

This is very cool news and brilliant work! Well done mate. What programming language are you reassembling it in?
I've shared the news to my Colonization Fans Discord, Facebook, and Reddit groups (feel free to drop by and talk about your work) as well as some of our CivFanatics news groups too so hopefully you'll get some players/testers soon.

Future plans?
You tell me. I hope to revamp the saving mechanism (why only 8 slots?) and in general improve the QoL of this old classic.

My wishlist item is probably a bit hardcore but I'd love support for the features in the Windows version of Colonization that uses higher resolution assets and allows you to play the game in HD resolutions (as demonstrated in the video here) as the original Windows version is a 16bit app that won't work on modern machines without a full VM or WineVDM.

 

[Hammerfrog]

Will it be possible to change/ optimize the graphics with this?

 

[Lord_Hill]

I know enough to know just how challenging DOS decompilations are, so best of luck. It would be nice to target some of the known bugs in the original game,

The biggest (in my opinion) is that sometimes the random number generator gets into a state where it gets stuck and your units will either always win or always lose in combat until you load a new save. There's a description of it here (also a few other bugs on the page too)

Bugs (Col)

Bugs are coding mistakes that cause errors during gameplay. Known bugs for Sid Meier's Colonization include: In Sid Meier's Civilization and Civilization II, units sentried on ships could be activated at sea to transfer them between ships sharing the same space. In Colonization, units transfer...

civilization.fandom.com

Units sometimes just vanishing instead of being demoted is another big one (also mentioned on that page)

Super duper stretch goals would be increasing the limits of the game
Max number of units, max number of colonies, map size etc

 

[Bladrov]

This is exciting! If you get around to taking suggestions from the floor, what I'd like most to see is for the Indian villages to show:

1. Whether you've visited them
2. What they'll teach, if anything
3. What product(s) they want, and what they sell

 

[personaalunioon]

This is very cool news and brilliant work! Well done mate. What programming language are you reassembling it in?

The game was written (mostly) in C and (some) in assembly. So it's mostly assembly at this point, with a select few routines ported to C and re-compiled/re-assembled with tools of the trade of that era -- MS C compiler 6 and MASM 6.

My wishlist item is probably a bit hardcore but I'd love support for the features in the Windows version of Colonization that uses higher resolution assets and allows you to play the game in HD resolutions (as demonstrated in the video here) as the original Windows version is a 16bit app that won't work on modern machines without a full VM or WineVDM.

Due to the nature of my project being a 16-bit DOS dis/re-assembly, HD resolutions are impossible. Even just doubling the resolution to 640x400 would be a very difficult and time-consuming undertaking, not to mention that many of the original graphical assets are 16x16 pixels which would look tiny on the bigger resolutions. Quite honestly, I did look into it at some point, but quickly realised that there are better, lower hanging fruit to address.

 

[personaalunioon]

Will it be possible to change/ optimize the graphics with this?

If by change/optimise you mean bigger resolutions, then sadly no.

 

[personaalunioon]

I know enough to know just how challenging DOS decompilations are, so best of luck. It would be nice to target some of the known bugs in the original game,

The biggest (in my opinion) is that sometimes the random number generator gets into a state where it gets stuck and your units will either always win or always lose in combat until you load a new save. There's a description of it here (also a few other bugs on the page too)

Bugs (Col)

Bugs are coding mistakes that cause errors during gameplay. Known bugs for Sid Meier's Colonization include: In Sid Meier's Civilization and Civilization II, units sentried on ships could be activated at sea to transfer them between ships sharing the same space. In Colonization, units transfer...

civilization.fandom.com

Units sometimes just vanishing instead of being demoted is another big one (also mentioned on that page)

Super duper stretch goals would be increasing the limits of the game
Max number of units, max number of colonies, map size etc

Well, I started this sometime in 2021, so...

That random number generator bug is definitely a pain and on my list of things to tackle.
Btw, those workarounds posted on that site are a complete joke. They completely or partially disable the srand() standard library function. No wonder nothing that depends on the RNG works after doing that.

About that vanishing units bug -- save file or didn't happen.

On increasing the limits... In theory possible (except for map size), but since memory is tight on DOS (640kb, remember) and even tighter for re:Colonization after the changes I had to do to make it work, I cannot make promises. Map size is a bit special in the sense that it has some game logic attached that expects the hardcoded 58x72 size. So that is less likely to happen.

 

[personaalunioon]

This is exciting! If you get around to taking suggestions from the floor, what I'd like most to see is for the Indian villages to show:

1. Whether you've visited them
2. What they'll teach, if anything
3. What product(s) they want, and what they sell

I hear you, but the issue with that is where to put that information. The game computes that on-the-fly and doesn't save it anywhere. So to make this work, the save file format would need to be changed and that would most likely break compatibility with the original game. I'm open to suggestions, though.

 

[Lord_Hill]

Re the disappearing unit bug, it's another annoying one that doesn't happen every time. It's been discussed in a few threads here over the years. I imagine it's going to be memory corruption again (from chatting to the guy who did the civ 1 decomp, most of the bugs he found were caused by that, unsafe arrays are an occupational hazard of programming in DOS it seems!) There's a save here where you can make it happen, but not 100% of the time.

[Col] Post in thread 'Colonization: Artillery Dying'

Oct 29, 2020

This savegame (loaded from autosave) would sometimes do that (I have the RNG-fixed EXE). Sometimes I would win. Sometimes the dutch soldier would die. It seemed to be random. Dutch soldier getting demoted only seems to happen without the RNG-fixed EXE, but dutch soldier dying happens there as well (as well as losing the battle).

• AyCe

 

[personaalunioon]

Re the disappearing unit bug, it's another annoying one that doesn't happen every time. It's been discussed in a few threads here over the years. I imagine it's going to be memory corruption again (from chatting to the guy who did the civ 1 decomp, most of the bugs he found were caused by that, unsafe arrays are an occupational hazard of programming in DOS it seems!) There's a save here where you can make it happen, but not 100% of the time.

I downloaded the save file, tried dozens of times with the original 3.0 VICEROY.EXE and in re:Colonization, but I cannot reproduce the bug. Either the dutch soldier wins and the spanish gets demoted or vice versa. Nothing disappears.

Regarding the RNG "fixed" exe, as I already said, nothing is fixed, it just disables srand() from the standard library breaking the game in many aspects. Presumably also causing this "vanishing unit bug" which actually then might not be a bug at all.

 

[Sikklebell]

I hear you, but the issue with that is where to put that information. The game computes that on-the-fly and doesn't save it anywhere. So to make this work, the save file format would need to be changed and that would most likely break compatibility with the original game. I'm open to suggestions, though.

Not sure if possible, but might it be an option to add it to the info window (on the right side of the screen)? And then compute it when it is selected?

 

[Bladrov]

Not sure if possible, but might it be an option to add it to the info window (on the right side of the screen)? And then compute it when it is selected?

That was my first thought, and I was thinking of catching a screenshot and mocking something up...but then I realized that persaonaalunioon seems to mean where to save that information on the back end. For that I don't have a good suggestion (esp. since I don't know how this project is saving data), so I didn't offer any ideas.

 

[personaalunioon]

A month has passed since the first post and I want to share with all interested parties what I've been up to.

Saving mechanism
I've ported and modified save game related routines so that the 8 save slot limitation is gone. In fact, there is no limit per se anymore. To do that, I had to change the save file name template from COLONY<XX>.SAV to <COUNTRYABBR><YEAR>.SAV (e.g. ENG1500.SAV). The latter is used by Colonization for Windows.
Any original saves will still work, but new saves will henceforth use the latter format.

There is a catch. You can only have one unique save per year/turn as any old save with the same name will be overwritten. Such is life.

RNG/Combat streak bug
I looked into it and it seems like the cause is exactly what was assumed it would be: excessive re-seeding of the random number generator.
As a general rule of thumb, one should (re-)seed the RNG only once at the start of the program. For some reason, the programmers ignored that. The result being that re-seeding happens in 19 different routines (well, technically in one routine, but that routine gets called by 19 other routines).

Now, some of them make sense, like for example in the colony screen where the intention was to get the same random building layout consistently. Other times not so much.

Most crucially, the music shuffle routine re-seeds the RNG. You can test it yourself -- turn off all music in the settings and one-sided combat streaks should be less noticable.
Interestingly, the combat resolution routine itself does not re-seed the RNG. And in my testing I could not reproduce the ridiculous streaks that we all remember, like 6-7-8 or more times in a row. Weird.

Still, the issue is clearly there and will be fixed when I complete the rewrite of the random module.

Indian village intel
I think I have an idea how to do it. The easiest way would be to compute the information on-the-fly just like the original game does. However, imo this would be tantamount to cheating as you as a player should only get this information when you visit the village with your scouts or wagon trains. So, this leaves us with the more cumbersome approach.

• First, the game needs to track if the user has visited the village, not just "anyone", like it currently does.
• Second, additional data about preferred trade items and the taught skill needs to be tracked.
• Third, to make this persistent, it needs to be added to the save game. Luckily, preliminary testing shows that the original game doesn't mind when additional bytes are appended to the file, as opposed to modifying existing structures.

So that's the plan more or less, but as it is more ambitious it will probably take quite some time before I get this implemented.

 

[Blake00]

Great work mate.

Indian village intel

Yeah I think we've been spoiled by FreeCol in which you just have to click on the Native villages to get their training info. Although I think (it's been a while) you still had to visit them once with scout once to activate it, then you'd know what that village trains and trades for the rest of the game. I remember it was kind of annoying going back to classic Col and having to revisit places all the time lol.

 

[Harre]

Cool project. Do you have it on GitHub or anywhere else where the development is in the open?

 

[BlakeIsHere]

I'm interested in this project. I've been playing FreeCol recently so this peaked my interest.

 

[personaalunioon]

Cool project. Do you have it on GitHub or anywhere else where the development is in the open?

Not at the moment, no.

I'm interested in this project. I've been playing FreeCol recently so this peaked my interest.

Give it a go. If you want to contribute then you are welcome to provide some feedback how well the game runs on your setup (OS, DOSBox variant/version) and if you noticed anything that seemed off/different from the original game. Then I can drop the beta moniker from the version string after I get a couple of confirmations that it runs without major hiccups.