Quick Modding Questions Thread

[Mattygerst]

That's one of the awkward parts of this file: You do not create any pathway, the game will go with the order in the xml file. If you check the CIV4ReligionInfo file, you'll see that the entries are in the same order as the symbols. This would not be too bad, if it would not be the case that the corporations are directly put behind the religion icons in the BtS GameFont files. The file you modified is probably from the vanilla game and will not work correclty with your mod, you'll need to use the one in ...\Sid Meier's Civilization 4\Beyond the Sword\Assets\res\Fonts. In the end, it should look something like this:

Spoiler :

View attachment 700687

There are some other important bits, namely that the single turquoise pixel at the right side of each icon is important, since Civ4 will use it to identify entries. Also, all of the pink pixels need to be exactly the right pink, otherwise the file will not work properly. And you also need to make sure, that the alpha channel is setup up correctly as well.

As for the link for the religion adviser screen, I believe it should be the <Button> entry in the CIV4ReligionInfo file. If you are using that already and it does not work, try the entry of one of the existing religions instead. If that button then appears, there might be something wrong with your entry or the button itself.

Quick question before I go further into this.
.
If there is no pathway and the game recognizes those boxes in order ... could it be as simple as copy and pasting each existing icon and moving it over two slots [or as many slots as needed to accommodate as many religions as are added --- to be clear, I'd start at the end of the corporations and move each icon to the right two slots and work my way back until I had two empty slots which Id then input the new religion icons] and then saving the file when everything has been copy and pasted and moved down?

 

[SaibotLieh]

Quick question before I go further into this.
.
If there is no pathway and the game recognizes those boxes in order ... could it be as simple as copy and pasting each existing icon and moving it over two slots [or as many slots as needed to accommodate as many religions as are added --- to be clear, I'd start at the end of the corporations and move each icon to the right two slots and work my way back until I had two empty slots which Id then input the new religion icons] and then saving the file when everything has been copy and pasted and moved down?

Yes, that is what you will have to do, plus don't forget to move the 'icons' in the alpha channel as well (if the game font editor does not already take care of that).

 

[Mattygerst]

Yes, that is what you will have to do, plus don't forget to move the 'icons' in the alpha channel as well (if the game font editor does not already take care of that).

Hello SaibotLieh and also Nexus,

I have made some progress, but am also still hitting some stumbling blocks. I edited the GameFont and GameFont75 files. To this point, I have fixed one issue I was having. The religion icon now appears in my city correctly - so thats good news!

But, I do have two recurring issues that I cannot find a remedy for: #1: The religion adviser screen remains a pink square; #2: On the scoreboard, my religion icon is STILL the cereal mills icon even though I edited both GameFont and GameFont75 files the exact same way. What gives? Any ideas? See pics below

I also have a third issue that isn't game-breaking but its odd. I changed the prereq tech for my religion to Mysticism. But, oddly, on the tech select screen, the icon for mysticism is now the icon for my religion. Weird. Any ideas? See pics below along with full xml entry created

 

[<Nexus>]

Score board: I think (but not completely sure) that this one has a separate path still pointing to the original game font. I would check the python files, iirc there's a Scoreboard.py or so. I would start there. I can't check it though at the moment.

Tech button: The answer is on your last screen shot <TechButton> is the path for the founding technology. So if a tech can found a religion but it's not found yet, a different icon is displayed for the tech. After founding the religion all other give will see the usual tech button. Remember the vanilla religions. They all have a a little symbol in the corner of the tech's button, indicating that you can fouoa religion if you are the first onn to research that tech.

 

[Mattygerst]

Score board: I think (but not completely sure) that this one has a separate path still pointing to the original game font. I would check the python files, iirc there's a Scoreboard.py or so. I would start there. I can't check it though at the moment.

Tech button: The answer is on your last screen shot <TechButton> is the path for the founding technology. So if a tech can found a religion but it's not found yet, a different icon is displayed for the tech. After founding the religion all other give will see the usual tech button. Remember the vanilla religions. They all have a a little symbol in the corner of the tech's button, indicating that you can fouoa religion if you are the first onn to research that tech.

Hello Nexus,
.
I found scoreboard.py; but I have absolutely zero clue what I'm looking for. I just read thru each of the 585 lines of code [religion is only written a few times in the code here] and I just don't have any clue how I am supposed to ensure that my new religion uses the new symbol on the scoreboard... I triple checked and the GameFont and GameFont75 files are correct and the symbol appears in my cities. Any further advice anyone can give as this seems to be almost my final step to being able to complete this portion of the mod I'd like to re-post onto this site for others to use.

 

[hankinsohl]

In Civ4 save game files, following the zlib compressed data is uncompressed data. Does anyone know what this data means?

In the files I've looked at, these bytes immediate follow the compressed data:
02 00 00 00 4E 2E 00 00 00 00 00 00 00 00 00 00

After these bytes are some more values that might be integers of various length. For longer games, there then appears several KB of random looking data; possibly this is padding.

Each file then ends with a length-prefixed 32 byte string which presumably is a hash value (SHA256?) over some portion of the game file.

If anyone knows anything about the data above please reply. I'd like to correctly parse any meaningful data as part of the save game editor I'm working on.

 

[f1rpo]

I've taken a look at some old sample savegames and, in one case, noticed two noncompressed strings that apparently correspond to player-placed (Alt-S) signs; screenshots attached. Those are handled by CyEngine::addLandmark, I believe. So maybe the other data at the end is also written by the graphical engine. Though I've no clue what exactly. Strategy layer? I see nothing there when I load the savegame ... The units have apparently been given custom names, but I don't see those strings in the mystery data.

Edit: I found my source for the savegame (it's not my own) - the first one attached to the opening post of this thread.

 

[Mattygerst]

Yes, that is what you will have to do, plus don't forget to move the 'icons' in the alpha channel as well (if the game font editor does not already take care of that).

SaibotLieh: Two things; #1: the gamefonteditor has an Alpha! button but I'm unclear what it does or how it applies to your comment above. #2: Are you able to help with any additional advice regarding my final issue whereby I am able to get my new religion icon to show on my city but it does not show on the scoreboard [it still shows cereal mills on the scoreboard as if my new icons have been skipped over even though they are clearly working on the city screen and both GameFont and GameFont75 files were treated exactly the same...

Edit; added gamefonts files for reference

 

[hankinsohl]

Thanks @f1rpo. I'll look into that.

Do you happen to know why save game files include a 32-byte digest at the end? Is this supposed to be for file integrity? Or is it part of an anti-cheat implementation?

Curiously, I've found that if you change any value in a save game file in a hex editor - other than the game version at the start - Civ4 simply exits or crashes when the modified file is subsequently loaded. I would have thought that the game would pop up a dialog about the save game file being corrupt instead of simply crashing.

​

 

[f1rpo]

I've looked into the Lock Modified Assets (LMA) game option at one point because I wanted my mod to be able to bypass that. When the LMA option is used, then four 32B checksums get written to the start of the savegame. Based on strings that I've seen the EXE push onto the stack before calculating those checksums, I've nicknamed them DLL, Shader, Python and XML checksum. (Specifically, the pushed strings were "\Assets\CvGameCoreDLL.dll", "*.fx", "\Shaders\FXO", "*.py", "\Assets\Python", "*.xml", ".\Assets\XML", "*XML\Art\*;*XML\Audio\*;*XML\Text\*".) Plus a fifth checksum when LMA is used with a mod loaded, presumably covering the mod's assets. Based on those observations, I've been assuming that the 32B at the end of a savegame act as another checksum – though apparently one that doesn't deal with LMA, so I didn't worry about it. Edit: I understand now that they're truly only 16B checksums.

I was not aware that even changes to the noncompressed portion of a savegame will crash the game, but it appears to be true, unfortunately. Seems like a good guess that this is indeed due to that final checksum. I noticed that I never got a proper error message when the LMA checks failed (e.g. when I had modified an XML file); the game just crashed. I had been assuming that this is a bug in the EXE. It will show an error message when e.g. the SAVE_VERSION in GlobalDefines doesn't match, so this kind of error handling is implemented. Well, still, perhaps the developers didn't feel that an error message was merited when files had apparently been tampered with. Anyway, the lack of an error message does not strike me as inconsistent with the hypothesis of an overall savegame integrity check (I'll call that the Save checksum for the moment). I guess it does hint at an anti-cheat measure, but it's hard to say what exactly Firaxis were thinking with that.

Hopefully it's really just SHA-256 applied to the whole savegame in its (almost) final form. Even then, it sounds challenging to guess all the details correctly. The function that I've identified as being likely responsible for the LMA-related checksums is apparently not used for the Save checksum, but it might call a more generic hash function (SHA or a wrapper around that? but I've really seen no specific evidence that it's SHA) that could lead up the call stack to the code that handles the Save checksum. Hopefully, you'll have better ideas how to reverse-engineer this if need be. Well, I'll just post the virtual memory address (non-Steam copy of BtS) that I've seen called, for whatever it's worth: 0x0040C520, called e.g. from 0x0040DA68 when calculating the XML checksum.

 

[Walter Hawkwood]

SaibotLieh: Two things; #1: the gamefonteditor has an Alpha! button but I'm unclear what it does or how it applies to your comment above. #2: Are you able to help with any additional advice regarding my final issue whereby I am able to get my new religion icon to show on my city but it does not show on the scoreboard [it still shows cereal mills on the scoreboard as if my new icons have been skipped over even though they are clearly working on the city screen and both GameFont and GameFont75 files were treated exactly the same...

Edit; added gamefonts files for reference

One of the many, many cursed things about the gamefont files is that gamefont and gamefont75 are, indeed, treated differently when it comes to looking up items. IIRC, one of those is read per row, while the other is always read in blocks of... 14, if my memory serves me?

 

[Mattygerst]

One of the many, many cursed things about the gamefont files is that gamefont and gamefont75 are, indeed, treated differently when it comes to looking up items. IIRC, one of those is read per row, while the other is always read in blocks of... 14, if my memory serves me?

Can you expand on what that might mean for me in this case and anything I need to change [I uploaded both files in my previous post]

 

[Walter Hawkwood]

I am not touching those myself! Even though I edited the gamefonts files countless times by now, I still have a very vague idea how they really work. What you could try and see what happens is append another 12 empty boxes after your new religion and/or insert another row (again, I am not 100% sure about it, but IIRC row number matters for one file but not for the other). TBF an easier solution would be to take someone else's longer working gamefont and then simply replaces stuff there with things you want (maybe even something like this: https://forums.civfanatics.com/resources/large-gamefont-without-modular-loading.19615/).

 

[hankinsohl]

In case anyone's interested, the following describes how the Civ4 savegame checksum algorithm works.
============
The checksum located at the end of a Civ4 savegame file is a 16 byte, MD5 digest in ASCII string format. Since each byte takes 2 characters to encode, the string is 32 characters long.

The Civ4 savegame file layout starts as follows:
GameVersion - DWORD - little endian value of game version
RequiredMod - CvString - name of mod required for the game to load. Does not impact the checksum algorithm
ModMd5 - CvString - MD5 hash for the required mod. Does not impact the checksum algorithm.
ChecksumDWord - DWORD. This value is zero every time I've watched the game load and its purpose is unknown. It does impact the checksum algorithm (see below).
LmaString - CvString -string of length 0 unless Lock Modified Assets (LMA) is enabled. If LMA is enabled this string is length 16. I don't know what this string represents.
LmaMd5_1 - CvString - string of length 0 unless Lock Modified Assets (LMA) is enabled. If LMA is enabled this string is length 32. This is an MD5 hash over asset content.
LmaMd5_2 - CvString - string of length 0 unless Lock Modified Assets (LMA) is enabled. If LMA is enabled this string is length 32. This is an MD5 hash over asset content.
LmaMd5_3 - CvString - string of length 0 unless Lock Modified Assets (LMA) is enabled. If LMA is enabled this string is length 32. This is an MD5 hash over asset content.
LmaMd5_4 - CvString - string of length 0 unless Lock Modified Assets (LMA) is enabled. If LMA is enabled this string is length 32. This is an MD5 hash over asset content.
HeaderSize - DWORD - number of bytes in Header

The algorithm to generate the checksum is as follows:
1) Generate an MD5 digest for the game header. Data for the game header digest starts after the HeaderMd5Size field in the savegame and continues until the start of the compressed game data minus 4 bytes (the CvGameAI expansion flag is not included in the digest).
2) Generate an MD5 digest for the compressed game data. This digest excludes the compressed data chunk length fields.
3) Create a memory stream to hold the data to MD5. The length of the memory stream varies depending on what's to be hashed. Into the memory stream write the following values:
ChecksumDWord (see Civ4 savegame layout in paragraph above)
The Civ4 game version as a little-endian, DWORD value.
ChecksumByte - this byte is from the savegame file and is located immediately prior to the checksum string at the end of the savegame (note that strings are saved with a length prefix; ChecksumByte imediately precedes the length)
LmaString
LmaMd5_1
LmaMd5_2
LmaMd5_3
LmaMd5_4
Admin password hash. This hash is 0x20 wide string characters long if a password is set, or 0x00 characters long if no password is set. The string length is written first, followed by the wide characters (if the password is set).
All password hashes are in this format.
* Game password hash
* Password hash for each possible player. 19 total player password hashes are written.
The compressed data MD5, prefixed by its length (0x20 00 00 00) as a little-endian DWORD.
The header MD5, prefixed by its length (0x20 00 00 00) as a little-endian DWORD.
The "magic" value 0x4D, 0xE6, 0x40, 0xBB. This value is hard-coded in the exe.

Entries marked with a single asterisk are assumed. These are hash fields for passwords that I haven't tested yet, but the code path is the same as for the Admin password hash and so what's written to the memory stream is likely the same as for the Admin password hash.

To double-check the algorithm above, I wrote a utility program to calculate the MD5 digest using the algorithm. The utility calculates the checksum and writes it to a copy of the targetted savegame file. Using the utility I've been able to successfully load and run Civ4 savegames modified in a hex editor. I've also been able to successfully generate the checksum for a LMA game.

Some notes and observations...
Not all of the Civ4 savegame data is covered by the checksum. Initial savegame fields RequiredMod and ModMd5 are not covered, nor is any of the uncompressed data following the zlib-compressed content (other than ChecksumByte and the checksum itself).

If you attempt to load a modified savegame file (e.g., after using a hex editor to change some data), but don't adjust the checksum, 1 of 4 things happens:
1 - the game loads and seems to run. This indicates that the value you modified wasn't covered by the checksum and doesn't cause the game to immediately crash. Of course, the game might be unstable at this point.
2 - the game seg-faults or throws an exception immediately when loading. In this case, you'll see a dialog pop up stating that the game exited in an unexpected way.
3 - modification of GameVersion, RequiredMod or ModMd5 fields results in various load failure dialogs assuming that the change made is consistent with the field type (e.g., you must enter a CvString in the RequiredMod field; entering a DWORD will misalign the rest of the savegame with the parsing code likely resulting in failure type 2).
4 - the game simply exits after a while with no dialog. This likely means that you've modified a value covered by the checksum.

Interestingly, I found that the Civ4 exe prepares a warning message when the savegame checksum fails to pass. However, it does not display the error message. This indicates a bug of some sort that's never been fixed. Also of interest, I believe that the same text associated with checksum mismatch actually is displayed in a dialog if the ModMd5 field is modified with a valid CvString.

Change log:
9/4/2024 - updated to reflect understanding of several previously unknown fields. Lock Modified Assets savegame layout documented.
9/5/2024 - updated to document ChecksumDWordValue and ChecksumByteValue.
9/6/2024 - updated to document RequiredMod and ValidationText fields.
9/6/2020(2) - updated to rename the ValidationText field to ModMd5.

 

[SaibotLieh]

SaibotLieh: Two things; #1: the gamefonteditor has an Alpha! button but I'm unclear what it does or how it applies to your comment above. #2: Are you able to help with any additional advice regarding my final issue whereby I am able to get my new religion icon to show on my city but it does not show on the scoreboard [it still shows cereal mills on the scoreboard as if my new icons have been skipped over even though they are clearly working on the city screen and both GameFont and GameFont75 files were treated exactly the same...

Edit; added gamefonts files for reference

Sorry, was on travels for a week, hence my late reply.

From what I can see those files should work for displaying the new religion icons, but I always modded within the Better BAT AI, which could have changed how those icons are read out. But in any case I am confused that you still get the cereal mills icon although you replaced it in the GameFont75 file. Maybe your modified file does not get used by your mod for some reason?

As for the icons you moved, seems mainly okay, only the turquoise pixel is missing for each of the diamond company icons.

For the religion adviser screen problem, did you try to test with a button that is already working, for example one of the other religions? The path seems to be alright, especially since it seems to be working in some way for the tech screen.

 

[Mattygerst]

Sorry, was on travels for a week, hence my late reply.

From what I can see those files should work for displaying the new religion icons, but I always modded within the Better BAT AI, which could have changed how those icons are read out. But in any case I am confused that you still get the cereal mills icon although you replaced it in the GameFont75 file. Maybe your modified file does not get used by your mod for some reason?

As for the icons you moved, seems mainly okay, only the turquoise pixel is missing for each of the diamond company icons.

For the religion adviser screen problem, did you try to test with a button that is already working, for example one of the other religions? The path seems to be alright, especially since it seems to be working in some way for the tech screen.

Without being home to test it, what would your guess be would happen if I moved everything down 2 slots in the 75 file and had 2 blank spaces after the last BTS religion? [Imagine, skip 2 past taoism and leave blank, then go to the new religion and keep going]

 

[SaibotLieh]

What I would rather do is just change the cereal mills icon a bit, like put some red lines over it, just to see if those changes translate into your mod. If that is not the case, than for some reason your 75 file is not recognised.

 

[hankinsohl]

@f1rpo

Hi f1rpo. I'm trying to make sure that the checksum algorithm I've documented is complete. I'm attempting to create a Lock Modified Assets game with the 5th checksum you mentioned in your previous post on this topic (a 5th checksum is added if LMA is enabled and a mod is loaded). I've created a LMA savegame which had Bug and Bull mods loaded but I'm only getting 4 checksums (see screenshot). If it matters, I configured Bug and Bull to load automatically.

Could you upload or point me to a savegame file that has the 5th checksum and let me know what mod I should load to get the checksum?

Thanks very much in advance.

 

[f1rpo]

If it matters, I configured Bug and Bull to load automatically.

If that means Custom Assets, then that's probably the issue. I don't think the game treats Custom Assets as being a mod at all (which keeps BUG/BULL in Custom Assets savegame-compatible with the unmodified game). If a mod is loaded, then its name should also appear as an ASCII string, I believe.

Thanks for sharing your findings – and creating that savegame editor. This final checksum was quite an obstacle, I see. I don't think guessing the details could've worked. Interesting to see (pretty much) confirmed that the crash upon checksum failures is a bug.

 

[hankinsohl]

If that means Custom Assets, then that's probably the issue. I don't think the game treats Custom Assets as being a mod at all (which keeps BUG/BULL in Custom Assets savegame-compatible with the unmodified game). If a mod is loaded, then its name should also appear as an ASCII string, I believe.

Thanks for sharing your findings – and creating that savegame editor. This final checksum was quite an obstacle, I see. I don't think guessing the details could've worked. Interesting to see (pretty much) confirmed that the crash upon checksum failures is a bug.

Thanks f1rpo. I worked out the meaning of the two fields I wasn't sure of before I saw your post The 5th checksum seems to be the 3rd field in the savegame file which I named "ValidationText"... oh, and if you want to actually see the checksum failure dialog box, you can enter a well-formated bogus CvString into the ValidationText field.

Working out the details of the checksum algorithm was actually a pain in the butt, haha, and I'm glad that's over.

I'm still woking on the savegame editor itself. I need to incorporate what's been learned while studying the checksum algorithm and a bunch of other things. Probably a month or 2 away.

The hash signing utility though is nearly done - I'll probably finish that tomorrow.