The great mystery of GameFont symbols: adding Religions to BUG/K-Mod

[tantanmen]

Hello hello! So I was attempting to merge JARM (Just Another Religions Mod) with K-Mod and ran into some problems...

Background Info (a not so brief history):

Spoiler :

The main problem encountered with this merge is that as religion symbols are added to the GameFont files, these symbols can become offset (depending on how they are read) and consequently mismatched in-game (especially in the city billboards).

Xyth posted on this topic in the K-Mod thread:

Some symbols in the gamefont files are read sequentially (offset from start of line), while others are referenced by fixed indexes. This means that adding new symbols (and especially new rows) can cause later symbols to get offset. Another annoyance is that the same symbol can be correct in one place (e.g the pedia), but incorrect in another (e.g. above the city nameplate) - because they're referenced differently in each case. This is not a problem unique to K-Mod, though its inclusion of BUG complicates things further, as BUG adds its own symbols and way of referencing them.

The only advice I can give is to experiment with adding extra unused symbols (with corresponding xml) in order to correct these offsets. For example, my mod has 18 religions and 16 corporations, but I found I had to add symbols for a 17th corporation (and some dummy xml) or the symbols above the city nameplate would get offset. And the last time I added some new resources I found I had to add exactly 4 or everything would break. It gets rather bewildering at times.

In summary, JARM + K-Mod is absolutely possible, but you'll have to experiment a lot to get everything lined up.

As can be inferred, 35 pairs of symbols (18 religions + 17 corporations) seemed to be the magic number for Xyth.

I wanted to add more religions to the JARM module, so I decided to create more space for symbol entries in the gamefont files by 'unplugging' corporations from the game. I followed the steps to remove corporations (by deleting xml entries) as suggested in a thread similar to this one. I also deleted the CIV4CorporationInfo.xml file (there was a thread for this, but I can't seem to find it).

Then I added religions to the GameFont files (with corresponding xml entries) until I arrived at the 'magic number' of 35:



Here is the result:


Almost there, but not quite (the main problems, as indicated by the titles of the screenshot attachments, are missing symbols and text overlay). To clarify, this is only a problem in the 'pedia. Otherwise everything else looks good and all 35 religions match up in-game. Religions appear correctly in city billboards.

So it is simply a matter of resolving the issue concerning the pedia – does anyone have a suggestion on how to accomplish this?

I should also mention that recently I read this thread started by Leoreth: Extending GameFont files, some of which I understood most of which I did not understand.

Nightinggale generously provided a GameFont Display utility, but unfortunately I have no experience with python…

Finally, keldath also posted on this topic, which has made me consider switching to WoC Lite:

so, the tga, is a mod created long time ago, world of civ used it, i think by johnny smith, so its not mine, i just carried it along the years.
anyway - the code is a c++ code that lyes in the dll sdk files, if you would compare my source files with that of advanced civ, you will be be able to spot the relevant parts - mostly marked with //tga.
you can also browse the mod parts thread for woc light mod, it is there as well.
the way it works is simple, assigned the number of the icon location on the gamefonts, and that's it,
looking at the tga files, you can find a reference location with my mod for example - lets say Islam is numbered 54 in the tga tag in the xml,
when you'll look on the tga file, find the Islam symbol, and bang, you got a reference point.

look at the tga tags in the religion xml and corporations and also in the bonus xml file. all have the tga xml tag.
its quite easy to get it going.

the pedia icons are not controlled via tga files, but individual dds buttons files, usually assigned by a reference in the XML file of the data it self or the corresponding art XML file.

If I could, I would like to continue adding, maybe just a couple/few more religions

Does anyone have experience implementing WoC Lite with K-Mod? Would I be able to merge the JARM module with WoC Lite?

If it would work, I'd like to make the jump to WoC Lite, but I confess I don't understand how to implement (merge?) dll sdk files

Could someone please advise me how to proceed?

Many thanks in advance

 

[tantanmen]

UPDATE:

@keldath, I was finally able to strip down this mod to 122mb. I’m hoping, like you suggested, that I just need to merge cpp files, or run a (re)compile. Right now I can only do xml, so I still need to learn how to add components. I looked at the Fractured States mod, but the only thing I understand is the <iTGAIndex> in the xml - this part I can do no problem. If you have some time to take a look, I could use the help

And @f1rpo, as our resident genius for all things post-KMod, I would also like to request your advice. I would be really grateful if you can provide any input. I *think* my basic question/conundrum is: within K-Mod, can WoC-Lite be combined with JARM (Just Another Religion Mod)?

My abilities are quite limited, and over the past two-three years I’ve taken this project as far as I can on my own. There’s been a fair amount of stumbling around, but at least I’ve learned some things

Again, thank you both very much. Any guidance would be greatly appreciated

 

[f1rpo]

I've never once edited the GameFont files, so I'm afraid I know less about this subject than you and anybody else whom you've mentioned. I also don't know how the TGA indexing from WoC (if that's where it's from) works. To make some contribution, here are my Git commits from merging the GameFont Display (GFD) into AdvCiv. But I've no idea if GFD could help you with your actual problem.

Spoiler :

• Commit 1: Copying CvScreenEnums.py from BtS. CvScreensInterface.py was already in AdvCiv.
• Commit 2: Applying the GFD changes to CvScreensInterface.py and CvScreenEnums.py, Since BUG replaces CvEventManager.py with BugEventManager.py, the keyboard shortcut (Ctrl+Shift+F1) needs to be implemented (only) in BugEventManager.py. I've got some non-KMod code in there. Apart from the import statement, this would hopefully be all you need (with tabs instead of spaces):

if eventType == self.EventKeyDown:
    if not InputUtil.isModifier(key):
        # GFD start
        if int(key) == int(InputTypes.KB_F1) and self.bShift and self.bCtrl and not self.bAlt:
            GameFontDisplay.GameFontDisplay().interfaceScreen()
            return 1
        # GFD end

(For the record, this isn't the proper way to add keyboard shortcuts to BUG.)
• Commit 3 (just the change to GameFontDisplay.py; the rest is unrelated): Adding non-BtS symbols to GFD. (In this case, the symbols from the BULL GameFont file.)

 

[tantanmen]

I've never once edited the GameFont files, so I'm afraid I know less about this subject than you and anybody else whom you've mentioned. I also don't know how the TGA indexing from WoC (if that's where it's from) works. To make some contribution, here are my Git commits from merging the GameFont Display (GFD) into AdvCiv. But I've no idea if GFD could help you with your actual problem.

Spoiler :

• Commit 1: Copying CvScreenEnums.py from BtS. CvScreensInterface.py was already in AdvCiv.
• Commit 2: Applying the GFD changes to CvScreensInterface.py and CvScreenEnums.py, Since BUG replaces CvEventManager.py with BugEventManager.py, the keyboard shortcut (Ctrl+Shift+F1) needs to be implemented (only) in BugEventManager.py. I've got some non-KMod code in there. Apart from the import statement, this would hopefully be all you need (with tabs instead of spaces):

if eventType == self.EventKeyDown:
    if not InputUtil.isModifier(key):
        # GFD start
        if int(key) == int(InputTypes.KB_F1) and self.bShift and self.bCtrl and not self.bAlt:
            GameFontDisplay.GameFontDisplay().interfaceScreen()
            return 1
        # GFD end

(For the record, this isn't the proper way to add keyboard shortcuts to BUG.)
• Commit 3 (just the change to GameFontDisplay.py; the rest is unrelated): Adding non-BtS symbols to GFD. (In this case, the symbols from the BULL GameFont file.)

Hello f1rpo,

Thanks for taking the time to read through this and I'm sorry if I roped you in a little. From my computer semi-illiterate perspective, I tend to think of programmers as all-knowing, all-out polymaths

It's helpful to see how you merged GFD with AdvCiv. Even if it doesn't help me with my problem, I'm sure this documentation will benefit others.

But I think there may be light at the end of the tunnel (did I dig this tunnel?): @keldath has been very generous in helping me dig myself out and develop this project.

If I can prattle on a bit more about it, this is part of a larger flavor mod that I have been merging with K-Mod; K-Mod mainly because I have the most familiarity with it, it seems to offer perfect stability, and (for karadoc anyways) is in an end-state.

I've played a good amount of AdvCiv and Doto and love the direction you and keldath are taking. In the future I would like to merge my flavor mod to yours, but I want to wait until I stabilize it on my end (I still get crashes to desktop )

So thank you again for your time and goodwill and please keep up the awesomeness of AdvCiv

 

[Leoreth]

I am not surprised you were confused by the thread I started on this topic, I did not exactly come away with better understanding of it either.

Xyth is definitely right that some "magic number" of boxes works in the file and you sometimes have to add more empty boxes than you need. You can also always extend the image to the right, this is usually safer than trying to introduce new rows.

I also recommend always using BUG FontUtils, in case you aren't already.

I can share "my" GameFonts files that I managed to successfully extend to some extent later (mostly through trial and error than real understanding of what works). Maybe that is already enough for your purposes.

 

[tantanmen]

Hello Leoreth, thank you for your advice

You can also always extend the image to the right, this is usually safer than trying to introduce new rows.

This is an astute observation. I'll feel pretty, pretty foolish if I simply need to expand the line, instead of *breaking* down to a new row

I also recommend always using BUG FontUtils, in case you aren't already.

I still have a lot to learn about BUG. I'm actually not familiar with FontUtils - I'll read up on it... aaand in a quick search for it, I just found another post by Xyth on adding extra symbols in BUG's Gamefont tgas (all the way back from 2009):

preferably without breaking everything horribly in the process as so often tends to happen with these notorious files.

Xyth was already on the job over a decade ago At least now we have tools like Asaf's editor.

I can share "my" GameFonts files that I managed to successfully extend to some extent later (mostly through trial and error than real understanding of what works). Maybe that is already enough for your purposes.

Thank you for offering to share it - I would certainly like to see what you did. Is this available in the most recent version of the uber-awesome DoC?

Thank you again for the help

 

[Leoreth]

Hello Leoreth, thank you for your advice


This is an astute observation. I'll feel pretty, pretty foolish if I simply need to expand the line, instead of *breaking* down to a new row

I still have a lot to learn about BUG. I'm actually not familiar with FontUtils - I'll read up on it... aaand in a quick search for it, I just found another post by Xyth on adding extra symbols in BUG's Gamefont tgas (all the way back from 2009):

Xyth was already on the job over a decade ago At least now we have tools like Asaf's editor.

Right. Unfortunately Asaf's editor does not support adding new boxes. Still, the safest possible approach of doing so is: 1) add empty boxes manually 2) open the game to see if anything is broken 3) use Asaf's editor to add the actual symbols.

Thank you for offering to share it - I would certainly like to see what you did. Is this available in the most recent version of the uber-awesome DoC?

Thank you again for the help

I think it's not in the current state of DoC, but maybe I already extended it over the original BtS font file? Not sure. Unfortunately I forgot to send you the file yesterday when I had access to it.

 

[Leoreth]

Does that help? Those font maps should be at least somewhat larger than the vanilla ones.

 

[tantanmen]

Thanks Leoreth, yes it helps to be able to see how you did this

I think I’ve been using an expanded font map from Xyth’s History Rewritten:


To get even more space, I’ve also begun to clear out all the boxes in the WoC Lite font map:

Does anyone know if there is a faster way to clear a font map? I’ve been deleting symbols one at a time

add empty boxes manually

I wanted to ask: Is there a thread for this, or if you could, could you please explain how to add boxes?

You can also always extend the image to the right, this is usually safer than trying to introduce new rows.

I think I might follow your advice and try extending the symbols right, instead of moving down to a new row. In my current version, I am using three rows for religions. The mod folder with all my xml for custom religions is an expanded version of JARM (Just Another Religion Mod). JARM normally uses two rows for religion symbols in the font map, so it may be worth a try to keep my ‘row count’ for religions to two instead of three. With the WoC Lite font map, I could probably even revert to a font map structure that is identical in its overall row count to K-Mod/Vanilla BtS. (EDIT: K-Mod only seems to have a different GameFont_75.tga file, so does K-Mod expect to reference the other GameFont.tga file from Vanilla BtS?? I think I thought of this at one time but promptly forgot it. More questions than answers...) I dunno, I guess I can experiment

Finally, although I’m trying to attack this from multiple angles, I’m arriving more and more at the conclusion that WoC Lite may be my best bet. With a proper merge, I should be able to assign numbers to the font map symbols and reference them in the xml via a tag: <iTGAIndex>

@keldath has a lot of experience with WoC Lite, and I think he’s making some headway on a merge within a K-Mod / DOTO platform. And then, my goodness, I don’t even know if I should allow myself to imagine it, but maybe, just maybe, all these GameFont woes and shenanigans will disappear

 

[Leoreth]

I wanted to ask: Is there a thread for this, or if you could, could you please explain how to add boxes?

I know of no smarter method than doing so manually. My editor of choice is GIMP. I just zoom in far enough to see the pixels, and copy/paste the white boxes that are already there. Unfortunately I cannot tell you what to do about those magic cyan pixels. I mostly ignored them.

(EDIT: K-Mod only seems to have a different GameFont_75.tga file, so does K-Mod expect to reference the other GameFont.tga file from Vanilla BtS?? I think I thought of this at one time but promptly forgot it. More questions than answers...)

As with all mod files, K-Mod will use the GameFont.tga in its own path first, then try BtS, Warlords, Vanilla in order. It does not expect to use the vanilla file specifically, if you place your own in the mod it will be used.

 

[Nightinggale]

Does anyone know if there is a faster way to clear a font map? I’ve been deleting symbols one at a time

Control+C/V is copy paste. You can likely copy an empty one and then paste the empty stuff into other boxes. Still not great, but it's the best I can think of.

I wanted to ask: Is there a thread for this, or if you could, could you please explain how to add boxes?

Use a graphics application, which supports alpha channel. Select a row and copy paste. You don't want to do this more often than absolutely needed because it's a hassle.

I once looked at the source code for GameFontEditor to allow it to add more boxes. I didn't get very far before giving up. Adding the ability to add/remove boxes would be the ideal solution, but I suppose it takes skill and time.

 

[tantanmen]

Hello Nightinggale, thank you for the advice I’ve been reading through some of your older posts - they have shed a lot of light on this topic.

Control+C/V is copy paste. You can likely copy an empty one and then paste the empty stuff into other boxes. Still not great, but it's the best I can think of.

That is definitely a smarter and smoother method. I estimate that with the gazillion boxes I still need to blank out, this pro-tip should save me 30+ minutes and reduce eyestrain by 200%. Thank you

I once looked at the source code for GameFontEditor to allow it to add more boxes. I didn't get very far before giving up. Adding the ability to add/remove boxes would be the ideal solution, but I suppose it takes skill and time.

Yes, I’m very grateful for the documentation you provided. This post of yours pretty much sums up what this journey has been for me (thank you as well for the wise words):

Nobody said that we are looking for an easy solution. We are looking for a working solution. Ease of use is a luxury, which may come later.

As for design and if it is good or bad.. well I would say that it is a good solution used incorrectly.

it does make modding harder. I agree that it wasn't the best design decision to put all icons in one file.

---

For the next batch of experimentation, I’m leading out with Leoreth’s observation that it’s safer to extend the row to the right, rather than dropping down.

The plan right now is to try the WoC Lite font map and aim for a single row for religions. On the next row, I’ll start with corporations (I have two versions of this mod, one with corporations, the other with corporations ‘unplugged’).

After I posted yesterday I realized I’ve really been talking up magic numbers and magic row counts. Maybe that’s what it will take. But I also get the feeling that pretty soon I’ll be superstitiously carving runic symbols into the pixel art

It made me wonder if I might need to take a step back for a moment and wait until I can access the smart part of my brain…

In the meantime, suggestions are always greatly appreciated. Thank you everyone for your help

 

[Leoreth]

Yes. Always respect the runes, follow the ancient rituals, and sacrifice to the pixel gods.

 

[Nightinggale]

Hello Nightinggale, thank you for the advice I’ve been reading through some of your older posts - they have shed a lot of light on this topic.

I hope you haven't overlooked my GameFont modcomp utility.

I have spent ages getting GameFont to do what I want it to do and fixing all the issues I have encountered in two mods (Medieval Conquest and We The People). I'm happy if my knowledge can be put to good use by other people.

One thing I haven't mentioned is that I have recently discovered another potential issue. Well recently means after writing a number of those old posts. Padding for GameFont in text and on city billboards don't have to be in sync. Text can round up the number of boxes in a row, like 15->16 and if that happens, the same might not happen with billboards. Also text might include an empty box while the billboard skips it. As a result, religion.getChar() might not print as the same in the advisor as it does on a billboard even though it's the same number. This is truly a mess.

I did however come up with a solution. In We The People, if a debug DLL is used, the domestic advisor will show an extra button (GameFont debug or something like that, the far right one). This will show a list of GameFont just like the modcomp. If you click on a line in that table, then the billboards will display a bunch of GameFont IDs and what the icon they see as belonging to those IDs. This can be used to locate such desyncs between the two systems and verify that they match again.

Porting this to BTS... well while it's doable, I don't feel like starting to mess with the BTS DLL to make it possible. Much easier if BTS modders drop in their own GameFont in WTP.

Now that I write about it here, I get the idea that instead of a specific DLL file, I should add this to UserSettings.txt. This way people can download the next release and use this feature without recompiling the DLL. All it takes is changing a 0 to a 1 in a txt file, which looks kind of like the ini file.

 

[tantanmen]

Medieval Conquest and We The People

These two mods (and RaR of course) are the reason I went and got Civ IV Complete

I hope you haven't overlooked my GameFont modcomp utility.

Yeah, I referred to it in the first post, but I mainly whinged about how I'm too much of a wimp to take the next step and venture outside xml. Xml is very comfortable to me.

Spoiler Confession :

For the purposes of this game at least, it seems to me that if you can acquire basic knowledge of xml (hard-earned in my case), you also more or less obtain a comprehensive knowledge. And if you can follow simple, very basic rules, everything usually works out the way it should.

Whatever other beasts are out there roaming the wilderness, C++, DLL, Python - these positively terrify me, so I've opted to stay in my comfortable den of xml and just kinda peek out my head every now and then I really think I might take a computer literacy course (quite possibly with grannies) just so I can have a better grasp on what I'm reading in these forums.

I have spent ages getting GameFont to do what I want it to do... I'm happy if my knowledge can be put to good use by other people.

Much easier if BTS modders drop in their own GameFont in WTP.

I'll continue to study up on your posts. Anything you can teach me I will be very grateful to learn. I'll drop by the WTP forum and say hi and maybe reminisce about how not too long ago I daydreamed that I would have loads and loads of free-time to play the best 15 mods for Civ4 Complete. Ah well, a few minutes snatched up here and there for modding can be fulfilling too

Speaking of which, instead of blanking out boxes, I made this post, so I'll put that off for another day I suppose?

 

[tantanmen]

Hello again Nightinggale,

I've had some more time to think about what you said:

Text can round up the number of boxes in a row, like 15->16 and if that happens, the same might not happen with billboards. Also text might include an empty box while the billboard skips it. As a result, religion.getChar() might not print as the same in the advisor as it does on a billboard even though it's the same number. This is truly a mess.

This 'rounding up' of the text - would this have anything to do with the fact that in GameFont each religion comes in a pair (one unstarred and one starred icon)? Corporations also come in pairs.

In Vanilla BtS, I don't recall any place where the starred religion icon (for the holy city) appears in the text. Corporations don't seem to have the starred HQ icon in the text either.

If this is the case, the original developers might have had a thought process along the lines of: "Since we only need the unstarred icons for the text part of this game, we can just round up (pair?) the box-count through religions and corporations and surely by then we'll be in the land of resource icons" (Resource icons, as we all know, come in singles, not pairs) The devs might have created or left some extra 'paired box' space for religions/corporations in the event of future expansion packs. Hence in K-Mod the 'magic number' of 35. Xyth did mention that even resource icons can be problematic and perhaps not incidentally, the magic number to keep everything from breaking was... "exactly 4":

And the last time I added some new resources I found I had to add exactly 4 or everything would break. It gets rather bewildering at times.

As you suggested, the rounding up might also explain why although everything appears perfectly fine in my city billboards, I'm still experiencing this in the 'pedia:

In the first two pics, the mouse scroll-over is okay, but in the actual text section (the last two pics), you can see the text overlay and missing symbols. The only icon that is reliably placed in the 'pedia is the coin for the financial trait What's so special about the coin icon? It just sits up there in the top GF row with the food and hammer icon, la-di-da, big whoop. And I've been reasonably quasi-analytic so far, so forgive me if I can't restrain myself and crack a lame joke: But is the coin the new lead? Should we follow the money? I dunno

Padding for GameFont in text and on city billboards don't have to be in sync.

I'm going back through your posts right now. I think I remember you stating that you had adjusted the padding for a mod. I know among everything else you're busy setting up for a WTP Modding Academy ( ), but if you ever have time to elaborate on 'padding', I would really appreciate it.

As a result, religion.getChar() might not print as the same in the advisor as it does on a billboard even though it's the same number.

I'm also researching: religion.getChar()

Thank you Nightinggale

 

[Nightinggale]

In Vanilla BtS, I don't recall any place where the starred religion icon (for the holy city) appears in the text. Corporations don't seem to have the starred HQ icon in the text either.

I can't remember offhand either, but the advisor screen where you can change state religion and help text for buildings spreading religion comes to mind as candidates.

"Since we only need the unstarred icons for the text part of this game, we can just round up (pair?) the box-count through religions and corporations and surely by then we'll be in the land of resource icons" (Resource icons, as we all know, come in singles, not pairs)

Unrelated. It's purely how the exe views the file. How the DLL decides to use the file is different.

if you ever have time to elaborate on 'padding', I would really appreciate it.

It would be better if I could remember it myself. I can remember that there is an issue, but I can't remember the details. I think it's a bit trial and error.

I'm also researching: religion.getChar()

Do keep in mind this is pseudo code.