JCivED - a toolbox for Civilization

Have you looked into doing anything similar for Sid Meier's Colonization (Classic/1994)? I've been looking around, but it seems very little work has been done on that game. I found several repos, but they only scratch the surface (especially compared to JCivEd). For instance, only about 60% of the SAV file has been decoded, and I'm only seeing one patch out there (for fixing the RNG... but that "fix" breaks other parts of the game by randomizing things that are supposed to stay stable).

Getting to a tool like JCivED for the original Colonization -- from where things are today -- seems extremely challenging, but I'm trying to start by decoding the SAV files the rest of the way.

How did you get to something as awesome as this? I'd appreciate any tips you can provide. Thanks!

Hi carthanc,

Your post is a major trigger for me, especially after I browsed the thread you linked ! Lots of stuff going on Colo-wise

Ever since I broke into Civ, I have been caressing the idea of doing the same for Colonization.

When I nearly stopped working on JCivEd, in 2015, I had major changes in my personal life which made it extremely difficult to dedicate time to that project any more (international relocation, job change, kids).

A few years later, things are settling down, and since a few weeks/months ago I have started to respawn a dev env for JCivEd... It takes time and I still do not have much of it.
However, globally, my ideal objectives at this point would be:
- be able to maintain/develop JCivEd again
- further reverse engineer CIV.EXE
- possibly create a full Civ clone/port
- include CivWin support

I would gladly add a Colonization project to that list :)

Regarding your last remark: you can read the beginning of the JCivEd story here: https://forums.civfanatics.com/threads/map-pic-and-pal-formats-figured-out-almost.478234/

Note that while developing JCivEd, I did extensive disassembly work on CIV.EXE, with IDA Free and a special debug version of DOSBox. This definitely helped pushing the SVE/MAP coverage nearer to 100%, as well as reveal secret logic, unearth remains of abandoned features and shed light on several famous bugs.

There is now a great disassembly tool, free, called Ghidra (from the NSA ), and I am wondering what it could do on those good old 16-bit EXEs...

I'll see you around !

Cheers
 
Awesome to hear you are thinking about getting back into things. Of course take care of the real life stuff first, though!

I'll keep reading through your linked post and the other threads I found just now. I did notice that you started early on with the decoding and re-rendering the map data. I was trying to do go after some interesting logic/features first, but I think starting with the map like you did would actually work better. You also helped me realize just how much has been done because the Colonization graphics have already been cracked. I am glad I don't have to figure that out!

While I'm envisioning awesome software here... would it be crazy for me to try to submit a patch for one of the bugs on SourceForge tickets list? I've used git (with and without github) for over a decade, but never worked with SVN or SourceForge. It looks like most of the code is java, which I have a little of experience using. I'm willing to tackle the learning curve, but is the Apr 17, 2015 commit (Release candidate 0.0.18a) the latest version? Or would I be working in the wrong place?

I love this tool and I'd really like to try to help. Of course, I'm not sure about timeline. I also experience RL intrusions! I had a grand vision for decoding the SAV file during my holiday break, but I underestimated how much I had to learn and felt like I barely got started before I had to go back to normal work. In fact, although I've worked in software for almost fifteen years, I've never really done a project like this outside of normal work. It has a different pace and it takes a special kind of effort to fit it in around all the normal stuff in life.

In any case, glad to connect and I'll keep watching this thread for further updates.
 
- possibly create a full Civ clone/port
- include CivWin support

Oh baby!!! *drools*

CivWin support for JCivEd would be awesome.

A Civ1 clone would be epic too. Have you seen the existing CivONE project? It seemed dead but its being worked on by new people. Maybe their work can give you a head start.
https://forums.civfanatics.com/threads/civone-an-open-source-remake-of-civilization-1.535036/
https://github.com/fire-eggs/CivOne

Have you looked into doing anything similar for Sid Meier's Colonization (Classic/1994)? I've been looking around, but it seems very little work has been done on that game. I found several repos, but they only scratch the surface (especially compared to JCivEd). For instance, only about 60% of the SAV file has been decoded, and I'm only seeing one patch out there (for fixing the RNG... but that "fix" breaks other parts of the game by randomizing things that are supposed to stay stable).

Getting to a tool like JCivED for the original Colonization -- from where things are today -- seems extremely challenging, but I'm trying to start by decoding the SAV files the rest of the way.

How did you get to something as awesome as this? I'd appreciate any tips you can provide. Thanks!
I would gladly add a Colonization project to that list :)

Awesome!!! You guys should chat with @AyCe as he's a bit of a Col editing guru and has created a few tools (check his sig). I'm sure he'd have some excellent insights into anything you wanna do.

.
 
I'll keep reading through your linked post and the other threads I found just now. I did notice that you started early on with the decoding and re-rendering the map data. I was trying to do go after some interesting logic/features first, but I think starting with the map like you did would actually work better. You also helped me realize just how much has been done because the Colonization graphics have already been cracked. I am glad I don't have to figure that out!

Honestly I think there is no best way to go - for CIV, MAP data was really what I was after, and busting graphics on this path was really added joy and fun :) But if you're after game logic, I think disassembly will be really helpful.

At some point I had started a tutorial to use IDA for reverse engineering, never went past "Part 2", but you can take a look if this is any help (althgouh I just realized all the pictures have been lost when civfanatics forums were migrated... might work on fixing this at some point...): https://forums.civfanatics.com/threads/tutorial-civ-ida-reverse-engineering.521637/

While I'm envisioning awesome software here... would it be crazy for me to try to submit a patch for one of the bugs on SourceForge tickets list? I've used git (with and without github) for over a decade, but never worked with SVN or SourceForge. It looks like most of the code is java, which I have a little of experience using. I'm willing to tackle the learning curve, but is the Apr 17, 2015 commit (Release candidate 0.0.18a) the latest version? Or would I be working in the wrong place?

Well, sadly yes, it would be the wrong place: after the last public release of JCivEd, I did continue to develop some stuff, also not related to JCivEd directly, but that impacted the JCivEd source code (and dependent "ddcommons" library), but that I never publicly released.

Short story, all that stuff was on a private SVN hosted at Cloudforge, which went out of work 1 or 2 years ago. Right before it closed, I hastily exported everything to private githubs, and didn't touch this code until recently.

So what I am busy doing on this front is exactly this: start from latest code on SourceForge, compare to latest private commits on github (there are 4 or 5), and try to come up with a version 0.0.19a that would be an up-to-date fresh start for future endeavours, including "pull requests" or patch proposals as you propose.

Biggest challenge is to not break stuff and lose functionality on the way (regression !!!)

I love this tool and I'd really like to try to help. Of course, I'm not sure about timeline. I also experience RL intrusions! I had a grand vision for decoding the SAV file during my holiday break, but I underestimated how much I had to learn and felt like I barely got started before I had to go back to normal work. In fact, although I've worked in software for almost fifteen years, I've never really done a project like this outside of normal work. It has a different pace and it takes a special kind of effort to fit it in around all the normal stuff in life.

I don't know Colonization data yet, but for Civ, most data in SVE was pretty straightforward, and what wasn't was slowly deciphered by group efforts through the forums, as you can see here: https://forums.civfanatics.com/threads/sve-file-format.493581/
It definitly took more time than one holiday break XD

Oh baby!!! *drools*

CivWin support for JCivEd would be awesome.

Didn't do anything CivWin-wise so far, but I'm pretty sure there isn't much difference between the savegames of DOS and Win versions, since the data they contain is the same (same game logic). So, as a first target, I am thinking a back-and-forth converter of savegames between CivDOS and CivWin, not necessarily a complete CivWin-flavored JCivEd version.

A Civ1 clone would be epic too. Have you seen the existing CivONE project? It seemed dead but its being worked on by new people. Maybe their work can give you a head start

Yes, I was there when SWY started CivOne, but didn't have time to participate much in his efforts, expecially since I am not well versed in C# / .NET, and also was toying around similar stuff in my Java cave... :p

I did look at the progress from time to time, but not much more... I also wanted to FIRST crack CIV.EXE completely open, since I wanted to totally clone the game logic, bugs included :)

Awesome!!! You guys should chat with @AyCe as he's a bit of a Col editing guru and has created a few tools (check his sig). I'm sure he'd have some excellent insights into anything you wanna do..

When I get to COLONIZE I guess we'll get in contact, thanks for the tip !

Cheers
 
I too have no idea why we cannot do this with enemy cities. So use "Data" tab instead. 1st way: change your civ to civ of the city which you want to delete (in first, "Game data" tab - "Player Civ ID" field). Then delete the city on the map and then change your civ back. Alternative way: go to "Cities" tab, find a city which you want to delete and set all its flags (city disorder, coastal city etc.) on. When all flags are on, it means that this city does not exist.

I think I wanted to limit the popup menu actions based on what was actually shown on the screen, but the code for that as well as the code for showing things is kind of messy, thus I forgot to allow removing cities when "Show all items" was selected...

More bugs are present since it is possible to remove enemy units when clicking on them even when they are not shown, for exemple in the hidden map areas... Messy messy code :)

I will fix the city deletion limit in next release.
 
EDIT: if darkpanda will not have time and desire for JCivED anytime soon, maybe I will fork it one day just for small fixes. Like, 1 total move point instead of 1/3 by default, fix Great Wall/Oracle order, fix some incorrect help for SS parts, add diplomacy statuses names, add terrain owning for new city by default etc.. But cannot promise anything. I can only make these small fixes, cannot read and understand Java properly.

I have been reviewing your comment and started to fix what you mentioned, namely the wrong SS help text (swapped 3 and 4).

EDIT: Also "Great Wall/Oracle" order ? Ok, got it

EDIT: What is the problem with "1 total move point instead of 1/3 by default" ? Got it too...
 
Last edited:
After researches I found the problem - when editing savegame, and accidentally open and click the line [0] Settlers in the tab 'Data'->UnitTypes, then the editor will write wrong "FFFF" instead of "7F00" (Cancelling Tech ID). You can avoid this by manually typing "127" in "Cancelling Tech ID" instead of "-1" and press Enter.

Took me a while to understand, but it was silly: I thought "cancelling tech" behaved just like "required tech" and had to be -1 (0xFFFF) to mean NONE... In fact, "cancelling tech" is set to 127, so that a Civ NEVER discovers that tech (hence unit in never cancelled), while "required tech" is set to special value "-1", which means a Civ can ALWAYS build the unit.

I will fix that in next release (coming soon).
 
That's it folks, JCivED 0.0.19a is released !

Main post updated, but if you just read this one, here are the release notes, mainly just bug fixes and merging back previous code that wasn't committed before:

EDIT: sorry, I also did not bother with Java compatibility when setting up my env, so it's Java 16 for now ! Let me know if this is a problem.

version 0.0.19a
  • CIV.EXE modding:
    • Added a patch to replace pillaging by terraforming (SHIFT+P changes land to Arctic)
    • Improved patch panel decoration
    • Added experimental patch to load game with F2
    • Minor patch updates
    • Minor refactoring of patch and EXE handling code
  • GameSave editor:
  • Background changes:
    • Major split in graphics code, between generic and Civ-specific
    • Initial split in game logic code, preparing for proper city screen rendering
 
Last edited:
Hi DarkPanda, if you were interesting in looking at CivWin, I just wanted to draw your attention to this post. The creator of the 'DOS titleset for windows' mod also included some source code for his tools (see 'CivWin file format demonstator source' attachment in the linked thread) which to my untrained eyes at least, also appears to contain source code handling the RLE that CivWin uses for compressing save files. Might save you a bit of time reverse engineering from scratch!

https://forums.civfanatics.com/threads/dos-tileset-for-civwin.268133/

I'm absolutely not trying to pressure you in this direction, it's just that the information could so easily be lost given that it's just an attachment in a forum post. JCivEd rocks by the way, I think I currently only have Java 8 installed (because there's a Civ III map editor which uses it), but at some point I'll upgrade and give the new version a go.
 
And of course it is no longer working on Windows with Oracle Java (8 update 321). I installed OpenJDK to fix this, easiest way is through .msi: https://docs.microsoft.com/en-us/java/openjdk/download, but after that DBGL shell for Dosbox (and probably some other programs) no longer works... I use Windows like 90% of the time, but I guess it's ok to boot Mint for JCivED. Or I should to check how to double-install both Oracle and OpenJDK Java...

EDIT: I just installed Oracle Java again and then launched OpenJDK and pressed "Repair" , and both DBGL and JCivED 0.0.19 works now. So everything's ok...
 
Last edited:
And of course it is no longer working on Windows with Oracle Java (8 update 321). I installed OpenJDK to fix this, easiest way is through .msi: https://docs.microsoft.com/en-us/java/openjdk/download, but after that DBGL shell for Dosbox (and probably some other programs) no longer works... I use Windows like 90% of the time, but I guess it's ok to boot Mint for JCivED. Or I should to check how to double-install both Oracle and OpenJDK Java...

EDIT: I just installed Oracle Java again and then launched OpenJDK and pressed "Repair" , and both DBGL and JCivED 0.0.19 works now. So everything's ok...

Hi tupi,

Ok, good that you could fix it, but if it's really too troublesome for others too, I will fix this, I aim to make your life easier, not break functionality...
 
I'm absolutely not trying to pressure you in this direction, it's just that the information could so easily be lost given that it's just an attachment in a forum post

Well...


Image 2.png
 
The creator of the 'DOS titleset for windows' mod also included some source code for his tools (see 'CivWin file format demonstator source' attachment in the linked thread) which to my untrained eyes at least, also appears to contain source code handling the RLE that CivWin uses for compressing save files. Might save you a bit of time reverse engineering from scratch!

https://forums.civfanatics.com/threads/dos-tileset-for-civwin.268133/

Thanks, this was a very good pointer !

I just finished porting his RLE code into Java. It seems to work, decoding a SAV, then recoding it, CivWin can load it without problem, however the initial file vs the recoded file do not have the same length, I'm not satisfied with that, since it could mean some hidden data is corrupted on-the-fly... I might have missed border cases when porting Monza's code.

By the way, SAV contents is very close to SVE contents, and indeed disassembled game logic is also extremely close between CivWIN and CivDOS.

From the few examples I found. Most differences seem to originate in OS code (windows vs DOS).

Anyway I think that SAV support will come first, not planning on supporting patches/modding for CivWin at the moment.

Will keep you posted,

Cheers
 
Wow, glad to have helped, even in a small way. Given that the userbase of CivWin in 2022 is all of about 5 people, I quite understand not wanting to add mod support! But a two way converter where you make your changes in JCivEd, convert the save, play a few turns being able to see much more of the map, then convert back and make further changes, this would have some value I think. Looking forward to whatever you come up with :)
 
Hi tupi,

Ok, good that you could fix it, but if it's really too troublesome for others too, I will fix this, I aim to make your life easier, not break functionality...
Nah, it's ok. I don't think that there's a need to support very obsolete versions of Java. Even Microsoft themselves want everybody to move to OpenJDK.
And what happens exactly (win 10): for Java 8 update 281 (64 bit) some window opens in a blink and then nothing happens. With 8 update 321 (64 bit), an error window ("Java Virtual Machine Launcher") with this text appears: "Error: A JNI error has occured, please check your installation and try again". After that another error window appears with this text: "A Java exception has occured". And then nothing.

Maybe it's just me who not lucky and it's ok for others. Anyway, as I already said, installation of last version of OpenJDK fixed this problem. Works ok when launching through .bat, but through click on .jar file itself it still launch Oracle Java with the same error.
 
Last edited:
Nah, it's ok. I don't think that there's a need to support very obsolete versions of Java. Even Microsoft themselves want everybody to move to OpenJDK.
And what happens exactly (win 10): for Java 8 update 281 (64 bit) some window opens in a blink and then nothing happens. With 8 update 321 (64 bit), an error window ("Java Virtual Machine Launcher") with this text appears: "Error: A JNI error has occured, please check your installation and try again". After that another error window appears with this text: "A Java exception has occured". And then nothing.

Maybe it's just me who not lucky and it's ok for others. Anyway, as I already said, installation of last version of OpenJDK fixed this problem. Works ok when launching through .bat, but through click on .jar file itself it still launch Oracle Java with the same error.

In fact it is just that my new env uses Java 16 by default, and when I build the ZIP release the classes are versioned "Java 16", so they will not execute on any JDK older than 16...

But since JCivED does not use any feature newer thatn Java 7, I probably could just reconfigure my env setup to keep backwards compatibility. I might try that later and stealthily modify the release ZIP (alos considering including the full source code in the ZIP release).
 
Nah, it's ok. I don't think that there's a need to support very obsolete versions of Java. Even Microsoft themselves want everybody to move to OpenJDK.
And what happens exactly (win 10): for Java 8 update 281 (64 bit) some window opens in a blink and then nothing happens. With 8 update 321 (64 bit), an error window ("Java Virtual Machine Launcher") with this text appears: "Error: A JNI error has occured, please check your installation and try again". After that another error window appears with this text: "A Java exception has occured". And then nothing.

Maybe it's just me who not lucky and it's ok for others. Anyway, as I already said, installation of last version of OpenJDK fixed this problem. Works ok when launching through .bat, but through click on .jar file itself it still launch Oracle Java with the same error.

Similar problem here. I have Java Version 8 Update 321 64 bit, dated 1-18-2022, and I only get the blink and terminate when I double-click the Jcived batch file. This is a fresh install of windows 10, having unwillingly moved on from 7. Everything worked fine in 7.

How do you install openJDK?
 
Thanks for that @darkpanda, works fine using Liberica Java 8. As a quick test I opened a barbarian save, chucked in a few barb nukes, saved and started Civ. The save loaded up fine, so confirmation that java 8 and the barb save bug are fixed
 
Top Bottom