Quick Modding Questions Thread

[hankinsohl]

Quick question - is the Civ4 SDK up to date with the game as it is currently released, for example through steam or GOG?

The reason I'm asking is that I've found a mismatch between CvGame.cpp and the binary save files created by the game. In particular, this code doesn't match my binary saves (line number 7769):

pStream->Write(m_eWinner);
pStream->Write(m_eVictory);

Instead, there's an undocumented 32-bit integer written in between m_eWinner and m_eVictory. The code that's actually producing my saved games is something like:

int iUndocumentedValue = 57;
pStream->Write(m_eWinner);
pStream->Write(&iUndocumentedValue);
pStream->Write(m_eVictory);

Note: I'm running an unmodified version of Civ4 I got off Good Old Games.

 

[f1rpo]

[...] Instead, there's an undocumented 32-bit integer written in between m_eWinner and m_eVictory.

How do you establish this? Apart from a short prolog, savegames are zlib-compressed. Did you uncompress a savegame and then look for easily identifiable values like m_iStartYear? If I were to compile a GameCore DLL from the source code that comes with BtS (and, to my knowledge, there is only one version of that [edit: well, once patch 3.19 has been applied], which indeed writes m_eVictory directly after m_eWinner) and replace the original DLL with that (or place it under CustomAssets, I guess; putting it under Mods won't work for this test because of the mod name check performed by the EXE), then I'd expect to still be able to load savegames created with the original DLL. This would demonstrate that the original DLL does not insert additional data. Is that what you've tried, and was your experience that the recompiled DLL is not save-compatible with the original game? I've never heard that players of the various digital and nondigital releases of BtS had trouble loading each other's savegames, so it shouldn't matter which edition is used for this test. If anything, I could imagine Steam altering the save format, not GoG. I'm a little too lazy to set up an environment for compiling a DLL straight from the SDK source right now, but some simple DLL mod from the CFC download database might be good enough, e.g. No Non-State Religion Commerce. This mod's DLL seems to be able to load a turn-0 autosave that I've created with the original DLL, i.e. with my complete edition from disc.

 

[hankinsohl]

How do you establish this? Apart from a short prolog, savegames are zlib-compressed. Did you uncompress a savegame and then look for easily identifiable values like m_iStartYear? If I were to compile a GameCore DLL from the source code that comes with BtS (and, to my knowledge, there is only one version of that, which indeed writes m_eVictory directly after m_eWinner) and replace the original DLL with that (or place it under CustomAssets, I guess; putting it under Mods won't work for this test because of the mod name check performed by the EXE), then I'd expect to still be able to load savegames created with the original DLL. This would demonstrate that the original DLL does not insert additional data. Is that what you've tried, and was your experience that the recompiled DLL is not save-compatible with the original game? I've never heard that players of the various digital and nondigital releases of BtS had trouble loading each other's savegames, so it shouldn't matter which edition is used for this test. If anything, I could imagine Steam altering the save format, not GoG. I'm a little too lazy to set up an environment for compiling a DLL straight from the SDK source right now, but some simple DLL mod from the CFC download database might be good enough, e.g. No Non-State Religion Commerce. This mod's DLL seems to be able to load a turn-0 autosave that I've created with the DLL that comes with my complete edition from disc.

Hi f1rpo:
Thanks for the quick reply.

In answer to your response, I'm writing a saved game parser. The parser implementation correctly decompresses the portion of a saved game file that is zlib compressed as part of its processing. In case it helps, attached is some text output generated by the parser which corresponds to a saved game file it's reading. Line 523 of the output file corresponds to the undocumented 32-bit integer mentioned in my previous post.

Edit:
Another reason I think that this undocumented field is present, is because I examined the uncompressed saved game file in a hex editor to confirm its presence. So unless the decompressor somehow inserted an extraneous integer or there's a bug in my code, it seems like this undocumented integer is actually present in the saved game. If, however, and as you've pointed out, people are able to compile the DLL from the SDK sources, and the SDK-derrived DLL is able to read/write saved games that are compatible with extant Civ4 saved game files then this points to some sort of bug in my code.

 

[f1rpo]

Oh, this is pretty far along. Impressive, and potentially useful for debugging serialization issues in mods.

Seems that something is going wrong earlier than that unexpected integer. All of these ...

Spoiler :

		CvGameExpansionFlag=0
		ElapsedGameTurns=17376
		StartTurn=1
		StartYear=0
		EstimateEndTurn=0
		TurnSlice=-4000
		CutoffSlice=500

... are clearly out of step, probably by two integers, i.e. the StartTurn data is actually CvGameExpansionFlag etc. Perhaps the EXE inserts the 0 and 17376 in between CvInitCore::write and CvGameAI::write.

 

[hankinsohl]

Oh, this is pretty far along. Impressive, and potentially useful for debugging serialization issues in mods.

Seems that something is going wrong earlier than that unexpected integer. All of these ...

Spoiler :

        CvGameExpansionFlag=0
        ElapsedGameTurns=17376
        StartTurn=1
        StartYear=0
        EstimateEndTurn=0
        TurnSlice=-4000
        CutoffSlice=500

... are clearly out of step, probably by two integers, i.e. the StartTurn data is actually CvGameExpansionFlag etc. Perhaps the EXE inserts the 0 and 17376 in between CvInitCore::write and CvGameAI::write.

Yeah, the values seem wonky and warrant more investigation. But mission accomplished in terms of my post - it seems like I can tentatively rule out the possibility that the SDK doesn't reflect the current game.

Edit: You're right about being off by two integers. Strangely, in CvGameAI.cpp we have (two ints from CvGameAI written after CvGame):

void CvGameAI::write(FDataStreamBase* pStream)
{
    CvGame::write(pStream);

    uint uiFlag=0;
    pStream->Write(uiFlag);        // flag for expansion

    pStream->Write(m_iPad);
}

whereas the binary seems to reflect (two ints from CvGameAI written before CvGame)

void CvGameAI::write(FDataStreamBase* pStream)
{
    uint uiFlag=0;
    pStream->Write(uiFlag);        // flag for expansion

    pStream->Write(m_iPad);

    CvGame::write(pStream);
}

 

[f1rpo]

Edit: You're right about being off by two integers. Strangely, in CvGameAI.cpp we have (two ints from CvGameAI written after CvGame): [...] whereas the binary seems to reflect (two ints from CvGameAI written before CvGame) [...]

I only see m_iPad getting set to 0 in the DLL. Mysterious what it's even supposed to be for. Unless we conjecture that the semantics (uh, none? in the DLL source) were changed before compiling the available DLL, the CvGameAI data should be all zeros. I'd expect to see those in between CvGame and CvMap. Your CvInitCore data looks legit, so I would still assume that the 0, 17376 is the EXE serializing 8 byte that the DLL doesn't get to see.

On a side note, speaking of padding, I've noticed that CLinkList (LinkedList.h) writes raw bytes into savegames, which will include uninitialized padding bytes when the template argument is a struct whose byte size isn't divisible by 4, e.g. TradeData. [Edit: And OrderData, MissionData; I think those are the only relevant cases.] Just as a heads up, if you come across random data in a linked list.

 

[hankinsohl]

I only see m_iPad getting set to 0 in the DLL. Mysterious what it's even supposed to be for. Unless we conjecture that the semantics (uh, none? in the DLL source) were changed before compiling the available DLL, the CvGameAI data should be all zeros. I'd expect to see those in between CvGame and CvMap. Your CvInitCore data looks legit, so I would still assume that the 0, 17376 is the EXE serializing 8 byte that the DLL doesn't get to see.

On a side note, speaking of padding, I've noticed that CLinkList (LinkedList.h) writes raw bytes into savegames, which will include uninitialized padding bytes when the template argument is a struct whose byte size isn't divisible by 4, e.g. TradeData. [Edit: And OrderData, MissionData; I think those are the only relevant cases.] Just as a heads up, if you come across random data in a linked list.

The pad turns out to be the zlib header offset. I've renamed Pad in the generated text accordingly.

I also added c++-style line comments to optionally show the field offset - looks terrible that way. Going to move the offsets to the start of the field output. I may also scrap this output format entirely in favor of XML output which would facilitate parsing by others... Lot's of work remains until this project is finished .

BeginCvGameAI
    CvGameAiExpansionFlag=0 // 0x00000f47
    ZlibMagic=0x000043e1 // 0x00000f4b
    BeginCvGame

Thanks for the heads up about CLinkList!

 

[f1rpo]

The 17376 being zlib-related sounds sensible, but that's not something the DLL (which is to say CvGameAI::write) would handle. Well, once you get to CvMap, it'll become apparent whether the CvGameAI data appears where it should.

 

[hankinsohl]

The 17376 being zlib-related sounds sensible, but that's not something the DLL (which is to say CvGameAI::write) would handle. Well, once you get to CvMap, it'll become apparent whether the CvGameAI data appears where it should.

void CvGameAI::read(FDataStreamBase* pStream)
{
    CvGame::read(pStream);

    uint uiFlag=0;
    pStream->Read(&uiFlag);    // flags for expansion

    pStream->Read(&m_iPad);
}

void CvGameAI::write(FDataStreamBase* pStream)
{
    CvGame::write(pStream);

    uint uiFlag=0;
    pStream->Write(uiFlag);        // flag for expansion

    pStream->Write(m_iPad);
}

CvGameAI reads/writes m_iPad just to get it out of the way. It's never referenced after that. The Game Engine handles compression/decompression later on. BTW, all of CvGame is compressed too. Output of a saved game is roughly:

1. Game Engine
2. CvInitCore.cpp
3. CvGameAI.cpp. Compression starts after m_iPad is written. CvGameAI writes uiFlag, then m_iPad, and then a bunch of CvGame stuff, all of which is later compressed.
4. CvMap.cpp
5. CvTeamAI.cpp and then CvPlayerAI.cpp for each player
6. Game Engine

I'm way past where compression begins in the parser, about midway through CvGame. Items 4-6 above are based on what I've read but have not verified myself.

 

[f1rpo]

My own crib sheet based on the DLL source:

Spoiler :

CvInitCore
CvGameAI --> CvGame --> CvDeal
                    --> CvRandom
                    --> CvReplayMessage
CvMap --> CvPlot
      --> CvArea
CvTeamAI --> CvTeam
CvPlayerAI --> CvPlayer --> CvCityAI --> CvCity
                        --> CvUnitAI --> CvUnit
                        --> CvSelectionGroupAI --> CvSelectionGroup
CvStatistics

 

[hankinsohl]

Another quick modding question...

Does anyone know the behavior of pStream->ReadString() if the stream buffer is currently positioned on a non-printable ascii character, such as 0x13?

 

[f1rpo]

I'll be surprised if it cares about special characters other than '\0'. But I can't say that I've experimented with those functions. Of course the wchar versions don't use ASCII at all (but UTF-16); or so I assume.

 

[hankinsohl]

Has anyone had success in using zlib to fully decompress and parse the compressed portion of Civ4 save game files?

What's happening is that decompression using zlib invariably runs into a zlib error because Civ4's compressed data isn't correctly terminated according to the zlib standard. A work around for this problem has been developed (https://stackoverflow.com/questions...uld-i-inflate-deflated-data-of-unknown-length) with help from one of the zlib authors. For short Civ4 savegame files, the workaround seems to result in well-formatted output which can be successfully parsed. For larger Civ4 save game files, decompression is initially successful but eventually the decompressed data stops making sense. This usually (always?) occurs in CvPlot.

Browsing the web, I found that this same zlib decompression was previously encountered (https://pypi.org/project/civ4save/#description - see the section discussing "Plots Bug").

The decompressed data remains low entropy well past the point where parsing fails - so there's good reason to believe that decompression was successful but that the Civ4 game engine does some extra processing on top of decompression prior to making game data available to the DLL.

I'd like to understand this extra processing in order to complete the saved game file parser project I'm working on. Any tips/pointers would be appreciated. Otherwise, it looks like I'll need to put Civ4 under a debugger to make sense of what's going on.

 

[varangian96]

One week later and one question more!

Could someone confirm whether this resource works: https://forums.civfanatics.com/resources/byzantine-greek-flamethrower.15680/

Specifically, when I try to use it, I get something similar to this: https://forums.civfanatics.com/threads/help-with-empty-civilopedia.383539/#post-9670976

The Civlopedia page only shows the unit's button. In-game, when I select the unit and attack with it, the text pop-ups ('99% of victory' and such) disappear, and entering and exiting the city interface doesn't actually close the city interface.

Were you able to get the unit to get the Byzantine flamethrower to work? If so, how did you do it, or is there some guide out there?

I got the other flamethrower unit to work with sounds and graphics, but the Byzantine one either appears looking just like the modern flamethrower, like the image below. I've been playing around with the xml files and nothing has worked.

 

[Mattygerst]

Hello all,
.
I have a mod that I created that has been available on the site for probably close 10 years - MMod. I am wanting to add some religions to my mod, and to start out as practice, I am seeking to move some of the religions that are already available in the AND mod. I'll doing one at a time.
.
Is adding these as simple as copying the XML from the assets of AND for: Religions, Buildings [temple, monastery, cathedral, shrine], Units [missionaries], and moving them over to the assets in MMod? Is there any step I'm missing? [I would change certain things such as prereq techs since MMod does not change the tech tree like AND does]. Does the artwork come over thru XML? I'm not sure?

 

[Walter Hawkwood]

Were you able to get the unit to get the Byzantine flamethrower to work? If so, how did you do it, or is there some guide out there?

I got the other flamethrower unit to work with sounds and graphics, but the Byzantine one either appears looking just like the modern flamethrower, like the image below. I've been playing around with the xml files and nothing has worked.

The post you quoted already has a link to the suggested very simple fix. Just rename the ByzantineFlameThrowerFinal.nif file to Paratrooper.nif

 

[Charky]

Hello. Disclaimer: I am not very good at modding... also a bit lazy. But most of what I learnt was back when it first came out when I was... (OMG) in my mid 20s.

I used to like having a slow game research wise, mega huge maps, with shorter building times for units (I didnt like the way marathon games just increased building times).

I could do all this by editing the main xml files. I assume you are supposed to create copies and stick them in the mod folder, but it seemed like too much effort to work out how to do this (Im lazy too) when I could just mod the core files directly.

So I have 2 hopefully quick questions:

1. So I am playing with by son now, in a lan multiplayer. I copied all of my XML folder to his game files, but it still registers a difference in our files when we start or load up a save game. I am wondering what this could be?
2. Can I edit any settings on the xml files once the game has started? My issue is that I want to increase movement speeds of ships and aircraft, as the bigger map does not translate so well when movement speeds for these are so slow. IIf I double them on the CIV4unitinfos.xlm, copy this across to my sons files will the current save game pick this up?

 

[<Nexus>]

2. Can I edit any settings on the xml files once the game has started? My issue is that I want to increase movement speeds of ships and aircraft, as the bigger map does not translate so well when movement speeds for these are so slow. IIf I double them on the CIV4unitinfos.xlm, copy this across to my sons files will the current save game pick this up?

Some mods allow that but IIRC the base game doesn't.

 

[Walter Hawkwood]

No, one can definitely edit values such as movement speeds / strength etc., and those will take effect upon reloading the game - just be careful not to change the checksum (by adding a new unit or removing one, for instance), or the save will crash. Simple value changes can definitely be picked up - but of course not in real time; a save / quit / load will be required.

 

[hankinsohl]

I've finally had time to run Civ4 under a debugger to figure out why straight forward decompression of save game data isn't working. For those who are interested, details follow:

When writing compressed data to a save game file, Civ4 writes the data in chunks. The size of the compressed data chunk
is written first, followed by the compressed data itself. Each chunk is usually 64K in size, except for the last chunk.

For example, let's say that compressed data for a game consists of 2 full 64K chunks and a last chunk of size F29D. The save
game file starting at where compressed data is written would look something like this (actual data values of course depend
on the game itself; also the initial offset into the save game will vary from game to game):

Offset            Size            Data            Remarks
0x00000B35        00 00 10 00     78 9C E4 BD     Note the 78 9C zlib magic at the start of the data.
0x00010B39        00 00 10 00     6D 72 3E CC     Offset is B35 + 64K + 4 bytes for the size field.
0x00020B3D        9D F2 00 00     F6 9B 63 8E     Offset is 10B39 + 64K + 4 bytes for the size field.

Change log:
9/4/2024 - Change to reflect that first compressed chunk isn't always 64K in length. For Lock Modified Assets games, the first chuck is (sometimes? or perhaps always?) shorter.