OpenCiv1 Project - Open source rewrite of the original Civilization 1 Game designed by Sid Meier and Bruce Shelley in year 1991

[rahorvat]

It might be worth having a look at how the city layouts are randomised, I don't remember seeing all the huts in a straight line like that in the DOS version. Would need to check though!

EDIT - Yep, no matter how many improvements you add, every building is in that first row instead of (presumably randomly) being built in a different row.

Yes, you are right. There is definitely some bug there. I will check and fix it.
Thanks

 

[Lord_Hill]

I think for my next trick I'll start a simultaneous game in DOS and OpenCiv. Obviously there's randomness, but it'll be really obvious for stuff like the wrong techs or wrong unit behaviour.

 

[rahorvat]

I think for my next trick I'll start a simultaneous game in DOS and OpenCiv. Obviously there's randomness, but it'll be really obvious for stuff like the wrong techs or wrong unit behaviour.

Yes, I think that would be very helpful. Since I'm still working on code I simply don't have the time to check everything
That's why your help guys is invaluable.

 

[rahorvat]

I also discovered that startup logic primarily sets all civilizations on a single continent?
Another bug to chase and fix...

 

[Lord_Hill]

I did wonder because everyone seemed to find each other very quickly! Also no other AI built a second city so far

 

[rahorvat]

I did wonder because everyone seemed to find each other very quickly! Also no other AI built a second city so far

It must be in connection with Per Continent Attack/Defense/Strategy flags. Perhaps they have another continent set as primary target and since they are on a different continent it is possible that they simply survive instead of flourish...
Will look at it tomorrow, I very curious what the flags say, and what will happen when they end on a proper continent!?

 

[darkpanda]

Hello. I registered just because of this thread.

As it's proven to be difficult to re-create source code from the original DOS game,
what about (instead) mapping out the game logic functions and how they flow? Do such "functional blueprints" exist?

Rephrased: A visual blueprint of game functions, arguments and types.. which functions call what, and which data, would be a great resource for anyone looking to learn how the game worked under the hood... OR for anyone looking to re-impliment it all in a fresh codebase.


Backstory: I remember figuring out that saving your game was not always a great way to cheat, or steer the game. Unit battles were not simply decided like ((player1 vs player2) * chance)), there was like this overlay of "good luck or bad luck" that seemed to ebb and flow over turns. That felt like a leap when compared to older implementations of Risk, or DOS Empire.

Hi there,

Building up such a visual blueprint is exactly where most of the difficulty lies with a 16-bit DOS game, and especially CIV.

I have spent years reversing it in IDA, to a point where many of the main game functions where mapped out (within IDA though), and yet there remain several unknown global variables throughout the code...

My main guess for this difficulty is that the code is a mess

Like rahorvat stated in a previous post, my take on this mess is that MPS had implemented a solid core, but then went on implementing new features and debugging through quick and dirty patches, then coping with side effects with even dirtier patches like by adding new functions to do the same things as others, but with different arguments, global variables that duplicate values keep game states between screens or actions, but without synchronizing them, etc.
In a word : dev as usual

When reversing CIV, this mess results in having pieces of code and data that are counter-intuitive (see bytes 27 et 28 in city data, to store info about fortified units in a city, also inconsistencies that keep info about Civs in different places), many bugs (path find, land value, arctic anomaly, pollution...), and also dead code never called in game, as well as unused game art (red space module, unknown castle thumbnails, unit designs, mouse pointers...).

Also, there are 2 or 3 gigantic functions containing a lot of game logic intertwined with other functions, and screen rendering, user interaction as well a game logic update get all mixed within sub-sub-sub-loops of such gigantic functions that update global variables to be used in other sub-sub-sub-sub-loops or other routines far down the code flow. For such functions, trying to map out a functional blueprint actually comes down to fully reversing the full code, in order to understand everything that it does.

In fact, some weird behaviours encountered during CIV games do come from this spaghetti approach - I'm thinking for example of destroyed CIVs that are not properly identified or recorded if they didn't have time to pass 1 turn before being destroyed, or also game freezes (actually endless loops) occuring when unit stacks (linked lists of units on a given square) get corrupt because a broken unit processing function forgot to update the linked list properly. Not to mention statistics screen that show abnormal values (negative number of settlers, etc.).

Spaghetti is delicious but it's still spaghetti

 

[darkpanda]

I think for my next trick I'll start a simultaneous game in DOS and OpenCiv. Obviously there's randomness, but it'll be really obvious for stuff like the wrong techs or wrong unit behaviour.

After startup, the "randomness" is actually very deterministic, the difficulty here could be to synchronize initial randomness between CivDOS game and OpenCiv game.
Now, displaying a City view does also reset the randomness in CivDOS, so if it does the same in OpenCiv, that could be a way to synchronize randomness.
However, every click on the main map makes a random call, so this must be taken into account when playing side-by-side.
Also, I saw somewhat of a bug in City view screen, so that randomness sync may not work until the City view is fixed.
I may try this side-by-side test, to see if it's anything worth... (and if time allows)

 

[rahorvat]

Hi there,

Building up such a visual blueprint is exactly where most of the difficulty lies with a 16-bit DOS game, and especially CIV.

I have spent years reversing it in IDA, to a point where many of the main game functions where mapped out (within IDA though), and yet there remain several unknown global variables throughout the code...

My main guess for this difficulty is that the code is a mess

Like rahorvat stated in a previous post, my take on this mess is that MPS had implemented a solid core, but then went on implementing new features and debugging through quick and dirty patches, then coping with side effects with even dirtier patches like by adding new functions to do the same things as others, but with different arguments, global variables that duplicate values keep game states between screens or actions, but without synchronizing them, etc.
In a word : dev as usual

When reversing CIV, this mess results in having pieces of code and data that are counter-intuitive (see bytes 27 et 28 in city data, to store info about fortified units in a city, also inconsistencies that keep info about Civs in different places), many bugs (path find, land value, arctic anomaly, pollution...), and also dead code never called in game, as well as unused game art (red space module, unknown castle thumbnails, unit designs, mouse pointers...).

Also, there are 2 or 3 gigantic functions containing a lot of game logic intertwined with other functions, and screen rendering, user interaction as well a game logic update get all mixed within sub-sub-sub-loops of such gigantic functions that update global variables to be used in other sub-sub-sub-sub-loops or other routines far down the code flow. For such functions, trying to map out a functional blueprint actually comes down to fully reversing the full code, in order to understand everything that it does.

In fact, some weird behaviours encountered during CIV games do come from this spaghetti approach - I'm thinking for example of destroyed CIVs that are not properly identified or recorded if they didn't have time to pass 1 turn before being destroyed, or also game freezes (actually endless loops) occuring when unit stacks (linked lists of units on a given square) get corrupt because a broken unit processing function forgot to update the linked list properly. Not to mention statistics screen that show abnormal values (negative number of settlers, etc.).

Spaghetti is delicious but it's still spaghetti

Oh, yes, lots of spaghetti infested redundant code packed in a delightful 16-bit segment : offset combination. I also noticed that they patched the code (possibly made by Sid Meier) with lots of dirty fixes, but as I progress with a code (mapping its data, structures, variables and translating a code to a high level code) the things will become much clearer...

Indeed, there are some huge parts of code, primarily contained into few segments (read modules) that will require significant effort to clear, but I'm not a stranger to this as well. The real benefit is where you replace references to objects in a memory with their real names, that should make things much clearer in the future when the time will come to translate this functions.

I'm cautiously optimistic that this process will last at least until the end of 2023 (to clear all code and map its data, structure and variables).

 

[rahorvat]

After startup, the "randomness" is actually very deterministic, the difficulty here could be to synchronize initial randomness between CivDOS game and OpenCiv game.
Now, displaying a City view does also reset the randomness in CivDOS, so if it does the same in OpenCiv, that could be a way to synchronize randomness.
However, every click on the main map makes a random call, so this must be taken into account when playing side-by-side.
Also, I saw somewhat of a bug in City view screen, so that randomness sync may not work until the City view is fixed.
I may try this side-by-side test, to see if it's anything worth... (and if time allows)

Don't bother until I map variable and structure references enough so I can fix the existing bug that puts all players to a single continent. I made significant progress, but there are a few things to finish. I also found the function that decides where to put each player to a Overlay 5, F5_0000_07c7(short playerID).

Sorry guys, but such bugs are to be expected at this stage

Cheers

 

[Lord_Hill]

You've already got further than everyone else that has attempted this, it's an amazing effort. I'll continue to test as little or as much as you want (as time allows!)

 

[darkpanda]

Don't bother until I map variable and structure references enough so I can fix the existing bug that puts all players to a single continent. I made significant progress, but there are a few things to finish. I also found the function that decides where to put each player to a Overlay 5, F5_0000_07c7(short playerID).

Sorry guys, but such bugs are to be expected at this stage

Cheers

If I can find time later today, I can share the function mapping I did while reversing Civ in IDA, it may boost you a little bit (I worked mostly on 474.01 though, hopefully this is not a problem).

 

[rahorvat]

If I can find time later today, I can share the function mapping I did while reversing Civ in IDA, it may boost you a little bit (I worked mostly on 474.01 though, hopefully this is not a problem).

Well, every little piece of a puzzle helps. I hope there are not too many significant differences between .01 and .05 ...
As my code is based on .05 version

 

[rahorvat]

@darkpanda I also found that you primarily use UInt8 (unsigned char) and UInt16 (unsigned int) in the SVE reference. But, the code suggests that by default Int8 (char) and Int16 (int) was used, based on disassembly code every reference to CBW, CWD, IMUL, IDIV, etc. suggests signed values!

So, expect a fixed .SVE reference table

I also did a table with all memory variables I so far translated and what they mean (the table is a work in progress) but I will share this unfinished table if anyone is interested.

Cheers

 

[darkpanda]

Well, every little piece of a puzzle helps. I hope there are not too many significant differences between .01 and .05 ...
As my code is based on .05 version

Well for starters I dropped raw listings of Segments and Functions from my IDA project, for Civ 474.01, you can see them here:
- Segments: https://sourceforge.net/p/jcived/wiki/CivDOS EN 474.01 RE segments/
- Functions: https://sourceforge.net/p/jcived/wiki/CivDOS EN 474.01 RE functions/

Several notes on those:
- As you will see, segment bases and surely functions offsets will not properly match those in version 474.05, especially the further you go down the list ; however, while spending time to design patches that work for all versions, I noticed that cdoe is still more or less spread out in the same way, so you will find the same functions more or less in the same areas, espacially in the same segments.
- beyond segment 31, all segments named "ovrXX" are the overlays contents ; it did require some hacking in IDA to successfully merge them into the same IDA project, including patching all "overlay interrupt calls" by replacing them with "far calls", but then IDA worked wonderfully to retrieve cross-references to variables and functions
- the "drvXX" segments, which I also added into the same IDA project, contain code disassembled from ASOUND.CVL and MGRAPHIC.CVL, as well as MISC.EXE ; not much work done on those, as they do not integrate as tightly into CIV code as Overlays
- In the function list, many of them have not been renamed yet, and many contain "unknown", "unk" or "??", which basically mean I don't know or didn't research waht the function does, or I just gave it a wild guess given the context when analyzing the flow

Tell me if this is any help, and/or if you would want anything else.

 

[darkpanda]

@darkpanda I also found that you primarily use UInt8 (unsigned char) and UInt16 (unsigned int) in the SVE reference. But, the code suggests that by default Int8 (char) and Int16 (int) was used, based on disassembly code every reference to CBW, CWD, IMUL, IDIV, etc. suggests signed values!

So, expect a fixed .SVE reference table

I also did a table with all memory variables I so far translated and what they mean (the table is a work in progress) but I will share this unfinished table if anyone is interested.

Cheers

The main entry for SVE reference is in the forums: https://forums.civfanatics.com/threads/sve-file-format.493581/page-2#post-12422448

If you are using text files from JCivEd, they do not necessarily match data types from the original CIV.

As a matter of fact, I didn't care too much about signed or unsigned, as I really tried to focus on game logic. When debugging, I usually ended up rewriting basic ASM instructions in Java to match exactly the behaviour of CIV.EXE, something probably similar to your x86 emulator in C# (see for instance the random routine implementation: https://sourceforge.net/p/jcived/code/HEAD/tree/branches/dev/src/dd/civ/logic/CivRandom.java)

I have mapped A LOT of stuff in IDA, but sadly I have no simple way to export "just" the memory variables... For the moment I made a raw dump of all things that have a "name" in my IDA project, so this includes functions (already listed separately) and all static variables, including game strings (name starting with "a"), and the listing only contains absolute address in project, so you'd have to do some math to find the segment and offset (arguably all variables are in segment 30 anyway).
Here it is:
- All names: https://sourceforge.net/p/jcived/wiki/CivDOS EN 474.01 RE names/

This is really raw, not sure how this will help.

 

[rahorvat]

Well for starters I dropped raw listings of Segments and Functions from my IDA project, for Civ 474.01, you can see them here:
- Segments: https://sourceforge.net/p/jcived/wiki/CivDOS EN 474.01 RE segments/
- Functions: https://sourceforge.net/p/jcived/wiki/CivDOS EN 474.01 RE functions/

Several notes on those:
- As you will see, segment bases and surely functions offsets will not properly match those in version 474.05, especially the further you go down the list ; however, while spending time to design patches that work for all versions, I noticed that cdoe is still more or less spread out in the same way, so you will find the same functions more or less in the same areas, espacially in the same segments.
- beyond segment 31, all segments named "ovrXX" are the overlays contents ; it did require some hacking in IDA to successfully merge them into the same IDA project, including patching all "overlay interrupt calls" by replacing them with "far calls", but then IDA worked wonderfully to retrieve cross-references to variables and functions
- the "drvXX" segments, which I also added into the same IDA project, contain code disassembled from ASOUND.CVL and MGRAPHIC.CVL, as well as MISC.EXE ; not much work done on those, as they do not integrate as tightly into CIV code as Overlays
- In the function list, many of them have not been renamed yet, and many contain "unknown", "unk" or "??", which basically mean I don't know or didn't research waht the function does, or I just gave it a wild guess given the context when analyzing the flow

Tell me if this is any help, and/or if you would want anything else.

The main entry for SVE reference is in the forums: https://forums.civfanatics.com/threads/sve-file-format.493581/page-2#post-12422448

If you are using text files from JCivEd, they do not necessarily match data types from the original CIV.

As a matter of fact, I didn't care too much about signed or unsigned, as I really tried to focus on game logic. When debugging, I usually ended up rewriting basic ASM instructions in Java to match exactly the behaviour of CIV.EXE, something probably similar to your x86 emulator in C# (see for instance the random routine implementation: https://sourceforge.net/p/jcived/code/HEAD/tree/branches/dev/src/dd/civ/logic/CivRandom.java)

I have mapped A LOT of stuff in IDA, but sadly I have no simple way to export "just" the memory variables... For the moment I made a raw dump of all things that have a "name" in my IDA project, so this includes functions (already listed separately) and all static variables, including game strings (name starting with "a"), and the listing only contains absolute address in project, so you'd have to do some math to find the segment and offset (arguably all variables are in segment 30 anyway).
Here it is:
- All names: https://sourceforge.net/p/jcived/wiki/CivDOS EN 474.01 RE names/

This is really raw, not sure how this will help.

Thanks for everything, will see if anything helps.
Meanwhile I almost mapped all variables from SVE file in code to objects. It's nice to see when things get their names, it instantly gets visual meaning instead of raw disassembly...

 

[darkpanda]

Thanks for everything, will see if anything helps.
Meanwhile I almost mapped all variables from SVE file in code to objects. It's nice to see when things get their names, it instantly gets visual meaning instead of raw disassembly...

I'm playing around with the Wiki at Sourceforge, which seems to accept HTML pretty well !

I just "ported" the SVE reference data tables from the CivFanatics forums to the JCivEd wiki, for what it's worth: https://sourceforge.net/p/jcived/wiki/SVE data reference table/

I'll probably be adding more stuff over there too, to serve as reference.

 

[rahorvat]

After startup, the "randomness" is actually very deterministic, the difficulty here could be to synchronize initial randomness between CivDOS game and OpenCiv game.
Now, displaying a City view does also reset the randomness in CivDOS, so if it does the same in OpenCiv, that could be a way to synchronize randomness.
However, every click on the main map makes a random call, so this must be taken into account when playing side-by-side.
Also, I saw somewhat of a bug in City view screen, so that randomness sync may not work until the City view is fixed.
I may try this side-by-side test, to see if it's anything worth... (and if time allows)

I think for my next trick I'll start a simultaneous game in DOS and OpenCiv. Obviously there's randomness, but it'll be really obvious for stuff like the wrong techs or wrong unit behaviour.

It's also worth noting that I used completely different random number generator. This new random generator has much better entropy (randomness) than old MSC compiler.
So any comparison that depends on randomness is pointless. Perhaps I will need to temporarily restore old random number generator for you guys to be able to compare two games simultaneously!?

That will perhaps be the best approach to testing?

What do you think?

 

[Lord_Hill]

Not necessarily, I was just gonna create a first turn save and load it in both and see how i do. Fully accepting that outcomes won't always match up. It'll still throw up things that are wrong.