Open source rewrite of the original Civilization game (1991) - OpenCiv1 Project

[darkpanda]

It's also worth noting that I used completely different random number generator. This new random generator has much better entropy (randomness) than old MSC compiler.
So any comparison that depends on randomness is pointless. Perhaps I will need to temporarily restore old random number generator for you guys to be able to compare two games simultaneously!?

That will perhaps be the best approach to testing?

What do you think?

The original random routine is part of MSC libs ?

Did you identify other functions in CIV.EXE which are statically libs from MSC ?

This could save me some time

No rush swapping the old RNG back, I'm not in a position to do parallel testing so far.

 

[rahorvat]

The original random routine is part of MSC libs ?

Did you identify other functions in CIV.EXE which are statically libs from MSC ?

This could save me some time

No rush swapping the old RNG back, I'm not in a position to do parallel testing so far.

Yes, it's simple int rand() and srand(int n) C functions.

I have done MSC libraries matching as a part of a Disassembler project, I created to give me the parsed assembly code. In the output the routine names are simply replaced by names from C library.
The LIB is composed of OBJ modules. The disassembler outputs location of matched OBJ modules (from LIB files) and locations of library functions (that are used by the code) inside of EXE.

As stated at the beginning of this thread the C compiler used to create EXE is Microsoft C 5.1 Optimizing Compiler.

The same effect can be achieved in IDA by loading FLIRT signature mq16rdos (MS Quick C v1.0/v2.01 & MSC v5.1 DOS run-time & graphic). Note that some functions IDA didn't recognize, but my decompiler recognized them and vice versa.

I will be glad to list the functions for version .01 if you want (so you don't have to run Disassembler as there are some paths to configure, and you have to install Visual Studio also)...

The output looks like this:

Matching MLIBC7
Matched library module dos\crt0dat.asm in segment 0 [0x2052a - 0x2068b]
Matched library module dos\crt0msg.asm in segment 0 [0x2068c - 0x206af]
Matched library module chkstk.asm in segment 0 [0x206b6 - 0x206d9]
Matched library module chksum.asm in segment 0 [0x206da - 0x206ff]
Matched library module dos\stdargv.asm in segment 0 [0x20700 - 0x20891]
Matched library module dos\stdenvp.asm in segment 0 [0x20892 - 0x208ff]
Matched library module dos\nmsghdr.asm in segment 0 [0x20900 - 0x20955]
Matched library module dos\stdalloc.asm in segment 0 [0x20956 - 0x20997]
Matched library module dos\dosret.asm in segment 0 [0x20998 - 0x209eb]
Matched library module fclose.c in segment 0 [0x209ec - 0x20ab3]
Matched library module flushall.c in segment 0 [0x20ab4 - 0x20ae7]
Matched library module fopen.c in segment 0 [0x20ae8 - 0x20b13]
Matched library module fread.c in segment 0 [0x20b14 - 0x20d05]
Matched library module fscanf.c in segment 0 [0x20d06 - 0x20d25]
Matched library module fwrite.c in segment 0 [0x20d26 - 0x20e6b]
Matched library module _filbuf.c in segment 0 [0x20e6c - 0x20f2d]
Matched library module _flsbuf.c in segment 0 [0x20f2e - 0x2108b]
Matched library module _freebuf.c in segment 0 [0x2108c - 0x210bb]
Matched library module _getbuf.c in segment 0 [0x210bc - 0x21129]
Matched library module _open.c in segment 0 [0x2112a - 0x21223]
Matched library module fflush.c in segment 0 [0x21224 - 0x21293]
Matched library module input.c in segment 0 [0x21294 - 0x215ed]
Matched library module input.c in segment 0 [0x215ee - 0x2199d]
Matched library module input.c in segment 0 [0x2199e - 0x21af1]
Matched library module stream.c in segment 0 [0x21af2 - 0x21b2b]
Matched library module ungetc.c in segment 0 [0x21b2c - 0x21b97]
Matched library module dos\close.asm in segment 0 [0x21b98 - 0x21bb7]
Matched library module dos\lseek.asm in segment 0 [0x21bb8 - 0x21c31]
Matched library module dos\open.asm in segment 0 [0x21c32 - 0x21dd5]
Matched library module dos\read.asm in segment 0 [0x21dd6 - 0x21eb2]
Matched library module write.asm in segment 0 [0x21eb4 - 0x21fdd]
Matched library module stackava.asm in segment 0 [0x21fde - 0x21ff1]
Matched library module nmalloc.asm in segment 0 [0x21ff2 - 0x22049]
Matched library module amalloc.asm in segment 0 [0x2204a - 0x221ac]
Matched library module dos\brkctl.asm in segment 0 [0x221ae - 0x22271]
Matched library module strcat.asm in segment 0 [0x22272 - 0x222b0]
Matched library module strcpy.asm in segment 0 [0x222b2 - 0x222e3]
Matched library module strlen.asm in segment 0 [0x222e4 - 0x222fe]
Matched library module strncpy.asm in segment 0 [0x22300 - 0x22327]
Matched library module strncmp.asm in segment 0 [0x22328 - 0x22361]
Matched library module atox.asm in segment 0 [0x22366 - 0x223b9]
Matched library module itoa.asm in segment 0 [0x223ba - 0x223d4]
Matched library module getenv.c in segment 0 [0x223d6 - 0x22439]
Matched library module isatty.asm in segment 0 [0x2243a - 0x2245c]
Matched library module perror.c in segment 0 [0x2245e - 0x224e7]
Matched library module dos\kbhit.asm in segment 0 [0x224e8 - 0x224f7]
Matched library module dos\getch.asm in segment 0 [0x224f8 - 0x22512]
Matched library module dos\int86.asm in segment 0 [0x22514 - 0x22593]
Matched library module closeall.c in segment 0 [0x20ab4 - 0x20ae7]
Matched library module fseek.c in segment 0 [0x22594 - 0x2284b]
Matched library module ftell.c in segment 0 [0x2284c - 0x229cd]
Matched library module dos\intdos.asm in segment 0 [0x229ce - 0x22a16]
Matched library module movedata.asm in segment 0 [0x22a18 - 0x22a35]
Matched library module dos\time.asm in segment 0 [0x22a36 - 0x22a88]
Matched library module tzset.c in segment 0 [0x22a8a - 0x22c1d]
Matched library module dtoxtime.c in segment 0 [0x22c1e - 0x22d3f]
Matched library module stricmp.asm in segment 0 [0x22d40 - 0x22d81]
Matched library module strnicmp.asm in segment 0 [0x22d82 - 0x22dd8]
Matched library module strupr.asm in segment 0 [0x22dda - 0x22dfb]
Matched library module strstr.asm in segment 0 [0x22dfc - 0x22e56]
Matched library module memcpy.asm in segment 0 [0x22e58 - 0x22e83]
Matched library module memset.asm in segment 0 [0x22e84 - 0x22eb0]
Matched library module xtoa.asm in segment 0 [0x22eb2 - 0x22f10]
Matched library module abs.c in segment 0 [0x22f12 - 0x22f27]
Matched library module rand.c in segment 0 [0x22f28 - 0x22f61]
Matched library module dos\ovlm6l.asm in segment 0 [0x2335e - 0x234a3]
Matched library module dos\unlink.asm in segment 0 [0x234a4 - 0x234b0]
Matched library module dos\b_disk.asm in segment 0 [0x234b2 - 0x234dd]
Matched library module dos\d_close.asm in segment 0 [0x234de - 0x234ea]
Matched library module dos\d_creat.asm in segment 0 [0x234ec - 0x2350b]
Matched library module dos\d_getvec.asm in segment 0 [0x2350c - 0x2351d]
Matched library module dos\d_open.asm in segment 0 [0x2351e - 0x23534]
Matched library module dos\d_rdwr.asm in segment 0 [0x23536 - 0x23558]
Matched library module dos\d_setvec.asm in segment 0 [0x2355a - 0x2356e]
Matched library module dos\freemem.asm in segment 0 [0x23570 - 0x2357c]
Matched library module dos\getdrive.asm in segment 0 [0x2357e - 0x23590]
Matched library module lshl.asm in segment 0 [0x23704 - 0x2370e]
Matched library module lshr.asm in segment 0 [0x23710 - 0x2371a]
Decompiling
Adding undefined API function _FF_MSGBANNER at 0x0002068c
Adding undefined API function _NMSG_WRITE at 0x0002092b
Inconsistent function call type in _NMSG_WRITE
Adding undefined API function _NMSG_TEXT at 0x00020900
Inconsistent function call type in _NMSG_TEXT
Adding undefined API function _cinit at 0x0002052a
Adding undefined API function _setargv at 0x00020700
Adding undefined API function int86 at 0x00022514
Adding undefined API function _maperror at 0x000209b8
Adding undefined API function fopen at 0x00020ae8
Adding undefined API function _getstream at 0x00021af2
Adding undefined API function _openfile at 0x0002112a
Adding undefined API function open at 0x00021c32
Adding undefined API function _cXENIXtoDOSmode at 0x00021dc5
Adding undefined API function fscanf at 0x00020d06
Adding undefined API function _input at 0x00021294
Adding undefined API function _aaltstkovr at 0x000206b6
Adding undefined API function _filbuf at 0x00020e6c
Adding undefined API function _getbuf at 0x000210bc
Adding undefined API function malloc at 0x00022004
Adding undefined API function _amallocbrk at 0x0002218c
Adding undefined API function brkctl at 0x000221ae
Adding undefined API function _amalloc at 0x0002204d
Adding undefined API function _amlink at 0x0002216a
Adding undefined API function _amexpand at 0x00022130
Adding undefined API function read at 0x00021dd6
Adding undefined API function ungetc at 0x00021b2c
Adding undefined API function memset at 0x00022e84
Adding undefined API function _aFlshl at 0x00023704
Adding undefined API function strcpy at 0x000222b2
Adding undefined API function strcat at 0x00022272
Adding undefined API function itoa at 0x000223ba
Adding undefined API function fclose at 0x000209ec
Adding undefined API function fflush at 0x00021224
Adding undefined API function write at 0x00021eb4
Adding undefined API function stackavail at 0x00021fde
Inconsistent function return type in F0_3045_1b0c
Adding undefined API function _freebuf at 0x0002108c
Adding undefined API function _nfree at 0x00021ff2
Adding undefined API function close at 0x00021b98
Adding undefined API function remove at 0x000234a4
Adding undefined API function _dos_getvect at 0x0002350c
Adding undefined API function _dos_setvect at 0x0002355a
Adding undefined API function kbhit at 0x000224e8
Adding undefined API function strlen at 0x000222e4
Adding undefined API function abs at 0x00022f12
Adding undefined API function rand at 0x00022f3a
Adding undefined API function _aFlshr at 0x00023710
Adding undefined API function getch at 0x000224fc
Adding undefined API function time at 0x00022a36
Adding undefined API function _dtoxtime at 0x00022c1e
Adding undefined API function __tzset at 0x00022a8a
Adding undefined API function tzset at 0x00022a9a
Adding undefined API function getenv at 0x000223d6
Adding undefined API function strncmp at 0x00022328
Adding undefined API function strncpy at 0x00022300
Adding undefined API function _isindst at 0x00022b54
Adding undefined API function srand at 0x00022f28
Adding undefined API function intdos at 0x000229ce
Adding undefined API function perror at 0x0002245e
Adding undefined API function exit at 0x000205ee
Adding undefined API function _nullcheck at 0x000206da
Adding undefined API function _ctermsub at 0x0002064c
Adding undefined API function _dos_read at 0x00023536
Adding undefined API function strnicmp at 0x00022d82
Adding undefined API function _dos_open at 0x0002351e
Jump to relative address BP in function F0_1000_1318 - instruction at 0x1000:0x1320
Adding undefined API function _dos_close at 0x000234de
Adding undefined API function _dos_freemem at 0x00023570
Adding undefined API function fseek at 0x00022594
Adding undefined API function ftell at 0x0002284c
Adding undefined API function lseek at 0x00021bb8
Adding undefined API function fread at 0x00020b14
Adding undefined API function memcpy at 0x00022e58
Adding undefined API function stricmp at 0x00022d40
Adding undefined API function fwrite at 0x00020d26
Adding undefined API function _flsbuf at 0x00020f2e
Adding undefined API function isatty at 0x0002243a
Adding undefined API function strstr at 0x00022dfc
Adding undefined API function strupr at 0x00022dda
Adding undefined API function _dos_getdrive at 0x0002357e
Adding undefined API function _bios_disk at 0x000234b2
Adding undefined API function movedata at 0x00022a18

 

[rahorvat]

As I'm adding variable names, object and arrays to a code I realized that they didn't properly check if the city exists, instead they used value -1 which translates to 0xffff hex. No wonder I got bunch of errors as the code was trying to access non existing city array elements... No surprise that game crashed often!
I wonder what other skeletons I will find in code

No wonder they called memory pointers from these days: Pointers from hell

 

[rahorvat]

- the "drvXX" segments, which I also added into the same IDA project, contain code disassembled from ASOUND.CVL and MGRAPHIC.CVL, as well as MISC.EXE ; not much work done on those, as they do not integrate as tightly into CIV code as Overlays

Hm, yes they do. They are mapped with far calls as stated in the documentation (for version .05) I compiled: CVL and EXE overlay driver documentation

 

[darkpanda]

Hm, yes they do. They are mapped with far calls as stated in the documentation (for version .05) I compiled: CVL and EXE overlay driver documentation

Yes I noticed something similar in .01, although some are acutally double-mapped (first calling another nearby function before the far call).

By the way I saw your overlay listing, and can complete your missing descriptions if not already done:
- Overlay 3: Hall of fame (FAME.DTA)
- Overlay 4: In-game help
- Overlay 15: Schism (when a large Civ splits into 2 civs when their capital is captured)
- Overlay 17: Palace
- Overlay 18: Sapceship

I am considering of restarting my reverse engineering efforts with .05 so our efforts could be sync'd, and it would also be a good opportunity to try out Ghidra, which apparently can decompile 16-bit DOS code to some extent.

 

[rahorvat]

Yes I noticed something similar in .01, although some are acutally double-mapped (first calling another nearby function before the far call).

Yes, they used wrapper functions for some graphic driver far calls... I replaced much of those functions with direct graphic driver function calls, and the rest of the wrapping functions will be removed later in OpenCiv1.

By the way I saw your overlay listing, and can complete your missing descriptions if not already done:
- Overlay 3: Hall of fame (FAME.DTA)
- Overlay 4: In-game help
- Overlay 15: Schism (when a large Civ splits into 2 civs when their capital is captured)
- Overlay 17: Palace
- Overlay 18: Sapceship

Thanks, much appreciated

I am considering of restarting my reverse engineering efforts with .05 so our efforts could be sync'd, and it would also be a good opportunity to try out Ghidra, which apparently can decompile 16-bit DOS code to some extent.

I'm not sure about Ghidra (since I came this far on my own). But, perhaps, I should first try it before I make any comment Apparently it looks much like IDA and can import IDA database.
As I said before, there is much by hand assembly used in source, so I'm not sure how successful Ghidra decompiler would be in those situations...

Much work is already done in OpenCiv1, so I'm not sure if Ghidra (at this point) would be of any help, perhaps with some functions it would be easier to translate, but I looked at examples, they are nothing spectacular.

I started with version .05 because it's most up to date, perhaps, with less bugs. As far as I know only unit definitions, difficulty multipliers and some texts are different than in .01 (somebody please correct me if I'm wrong).

Also, I'm almost not using IDA anymore since code is stable (for now). I'm only using IDA for reference, for example: if generated code doesn't work as expected...
My work in IDA is nothing spectacular, some redefined data and code sections and no renames... otherwise, I would share my IDA database with you to help you get started.

 

[darkpanda]

The same effect can be achieved in IDA by loading FLIRT signature mq16rdos (MS Quick C v1.0/v2.01 & MSC v5.1 DOS run-time & graphic). Note that some functions IDA didn't recognize, but my decompiler recognized them and vice versa.

Wow, you just taught me something, now I have a bunch of static lib functions auto-recognized as well, thanks a lot !

By the way, if you want to achieve the exact same "City View" look as in the game, you will need to use the original random routine, since it relies heavily on it. In particular, the City View is the same whenever you open it, because the random generator is re-seeded with the same seed every time before generating the City View (which incidentally can be exploited to win every battle and more, see here: https://forums.civfanatics.com/threads/civ1-exploit-design-flaw-in-randomness.509679/ )

 

[rahorvat]

Wow, you just taught me something, now I have a bunch of static lib functions auto-recognized as well, thanks a lot !

Glad to be of help

By the way, if you want to achieve the exact same "City View" look as in the game, you will need to use the original random routine, since it relies heavily on it. In particular, the City View is the same whenever you open it, because the random generator is re-seeded with the same seed every time before generating the City View (which incidentally can be exploited to win every battle and more, see here: https://forums.civfanatics.com/threads/civ1-exploit-design-flaw-in-randomness.509679/ )

No, I don't think that's necessary. It's only important that City view works properly (random layout placement)

It's expected that random number generator behaves identically every time, because it's seed is stored in SAV file. If it's reseeded (reset) every time you open City view then you can take advantage with that for sure.

 

[Blake00]

Hi mate, have you got any demonstration videos of your project? Can't remember if there was one earlier in this thread I've forgotten about. Hopefully we can get to that news post about your work in the next week or 2 and figured I'd check if there's a demonstration video I can put in it.

 

[rahorvat]

Hi mate, have you got any demonstration videos of your project? Can't remember if there was one earlier in this thread I've forgotten about. Hopefully we can get to that news post about your work in the next week or 2 and figured I'd check if there's a demonstration video I can put in it.

No, I never did a demonstration video. I don't think it's time for that as the code/graphics is still somewhat sluggish and needs more work.
In a month or two I think the game will be ready for a video.

 

[Barking Iguana]

I love that this is being done. I hope it can be coded in a way that leaves the possibility of later adding options that would eliminate some player exploits and AI cheats.

Off the top of my head
P!) It should not be possible to pay cash for hammers and then switch production to something where those hammers would have been more expensive. Not without paying the difference, anyway.
P2) Arguably, it should not be possible to sell you palace.
P3) Arguably, it should not be possible for diplomats on railroads to escort military units through zones of control without limit.
P4) Should there be some limit on ICS?
C1) The AI should have to actually build and move caravans in order to establish trade routes.

I think I'm forgetting one or two more.

I know implementing this like that now would impede progress, and I'm not suggesting that. But if hooks can be included to allow development of those options in the future, I think that would enhance the game.

 

[rahorvat]

I love that this is being done. I hope it can be coded in a way that leaves the possibility of later adding options that would eliminate some player exploits and AI cheats.

There should be a cheat options definitely, but regular gameplay should be without exploits (Settler activate, build and so on... comes to mind).

Off the top of my head
P!) It should not be possible to pay cash for hammers and then switch production to something where those hammers would have been more expensive. Not without paying the difference, anyway.
P2) Arguably, it should not be possible to sell you palace.
P3) Arguably, it should not be possible for diplomats on railroads to escort military units through zones of control without limit.
P4) Should there be some limit on ICS?

Could you, please, explain what ICS means, of the top of my head I can't remember...

C1) The AI should have to actually build and move caravans in order to establish trade routes.

I think I'm forgetting one or two more.

I know implementing this like that now would impede progress, and I'm not suggesting that. But if hooks can be included to allow development of those options in the future, I think that would enhance the game.

Yes, I totally agree
There are many ideas already about what should be fixed.

I also would argue that player should not be able to conquer your newly founded city with a Settler (attack strength of 0!) unit, but the city should be attacked with a unit of attack strength 1 (at least).

I will create a table of gameplay fixes as a remainder and a reference for other people.

After I complete work on a main code (somewhere at the beginning of 2024) I will start a debate about what should be fixed and start to fix things that we agree upon.

Cheers

 

[Barking Iguana]

ICS stands for infinite City Sprawl, where the cites are founded right next to each other or one space apart. The idea is that each city works its central square for free, so if you keep cities at Size 1 and build a Settler as soon as it gets to Size 2, you can rapidly increase the number of squares worked at the beginning of the game.

 

[rahorvat]

As I'm moving all structures and arrays to a managed code, there are so many array indexing mistakes that pop up!
It's a wonder that the old game worked somewhat reliably!

I'm almost near completion, and hope that the next Alpha release will be this month.

Cheers

 

[Hans Lemurson]

It's easy to romanticize old games, but they were often hacked together by people under time pressure who didn't always remember what they were doing.

 

[rahorvat]

It's easy to romanticize old games, but they were often hacked together by people under time pressure who didn't always remember what they were doing.

Yes, I agree. It happens to everyone under time pressure

That's why open source is so important, the others can unhack it with time

 

[rahorvat]

Dear all,
The new release is ready! This is a bug and speed fix release.

Please test this release for any bugs as soon as you are able.

I wont be available until after the new year.

Happy holidays!

 

[axx]

I've been playing it and sofar the only real bug is with the palace view - there are no transitions when building palace pieces and a part of the panel doesn't disappear.
Other than that there's a noticeable increase in game speed, it's more normal now. So god job overall.

Edit: Also, for me the #1 'optional' feature would be introduction of hitpoints. Reading Digital antiquarian's Civ1 entry, Sid based his game on a game called Empire (from 1977. One look at a screenshot from that game and it's pretty evident he even copied the looks). However he removed the damage rating system from Empire so you have those battleship loses to militia situations. It's what makes Civ1 what it is, but it would be nice to have an option to make it a bit more complex than that.

 

[Akka]

By the way, I never received any feedback from any of you on an Alpha prerelease version, what's with that?

Because I just discovered this marvelous thread today !
Count me in among those who would LOVE to see a Civ1 that actually looks, feels and plays exactly the original DOS one (I can still play it on DosBox, but I'd really appreciate a more configurable and reliable version for modern OS).

I just tried the Alpha6 by launching it through VS2022.
Feels already pretty playable (if very sluggish, at least in debug), though I didn't get any sound nor music (even after checking the sound option). Do I need something specific or is it a bug ?

 

[rahorvat]

Because I just discovered this marvelous thread today !
Count me in among those who would LOVE to see a Civ1 that actually looks, feels and plays exactly the original DOS one (I can still play it on DosBox, but I'd really appreciate a more configurable and reliable version for modern OS).

I just tried the Alpha6 by launching it through VS2022.
Feels already pretty playable (if very sluggish, at least in debug), though I didn't get any sound nor music (even after checking the sound option). Do I need something specific or is it a bug ?

Hi to everyone from USA, Florida

Glad that you love it The sound is currently not a priority, as there is already so much work to be done on main code that I simply didn't find the time for the sound engine...
These things are one man job at the beginning and I just started the work on OpenCiv1 on may 2023!

The speed will increase as I translate and optimize more code, please be patient