[Dev] GameFont.tga editing

dbkblk · Jun 18, 2014

Hi !

I have some question about the editing of Gamefont. I would like to add new languages. I have studied how the russian is added to the game. Basically, they replace accents by cyrillic letters and call in-game accents to render letters. They created a modded sylfaen font to render letters.

I've noticed the modded gamefont.tga from AND use many more icons than the base game tga.

- How do you call a slot of gamefont ingame to show an icon in-game ?
- Do you think it's possible to call a letter from this file like you call an icon ?
- Is there a limit of slot in this file ?

My plan is to introduce new languages without replacing any letter.

Nightinggale · Jun 18, 2014

I could have written that post. I looked into ganefont and came up with the very same idea for a solution. You are one step ahead of me though as you have a Russian gamefont.tna. Where do I download it?

dbkblk said:
- How do you call a slot of gamefont ingame to show an icon in-game ?
- Do you think it's possible to call a letter from this file like you call an icon ?

It has to be and it might not be that complex.

InMedieval Conquest there is this python line.

Code:

screen.setTableColumnHeader( self.StatePages[iState][iPage] + "ListBackground", iYieldOnPage + 2 + offset, "<font=2> " + (u" %c" % [COLOR="Red"]gc.getYieldInfo(iYield).getChar())[/COLOR] + "</font>", iWidth )

The column header gets the yield icon with this line.
CvYieldInfo::getChar() returns an int, though it appears that it is the exe, which decides which int it is. Presumably each icon in gamefont.tna has a unique int and it will just be a matter of figuring out the right conversion to print that int.

The question is how much can be controlled by TEXT XML and how much require python/DLL coding. If we are really lucky, we should be able to write the output of (u" %c" % 244) in XML and it will print the icon at 244. However I fear that it isn't this simple. I guess I should try to debug a bit to see how ints and icons actually fit together.

EDIT: yeah it wasn't that simple. Row 5 slot 1 has the int 8483. The good news is that increasing the int by one also increases the slot by one meaning once we have one int to icon conversion, we can just add numbers to that and get the entire row.
8483 is 0x2123, which in ASCII is !#

dbkblk said:
- Is there a limit of slot in this file ?

The picture in the file is 1920x320. The first 70 pixels goes to ASCII characters. Using 16x16 icons we get room for 120x15=1800. That would be the theoretically max, but there might be a lower one. For starters there might be wasted pixels between lines and rows.

Also how do you add more icons? I get how to fill out blank spaces, but how do you add spaces?

dbkblk said:
My plan is to introduce new languages without replacing any letter.

That's my goal too. It seems like it is the absolutely easiest gamefont.tna to maintain and release.

dbkblk · Jun 18, 2014

Nightinggale said:
I could have written that post. I looked into ganefont and came up with the very same idea for a solution. You are one step ahead of me though as you have a Russian gamefont.tna. Where do I download it?

I think we can collaborate on this

Here are the polish and russian versions:
Russian
Polish

Nightinggale said:
The question is how much can be controlled by TEXT XML and how much require python/DLL coding. If we are really lucky, we should be able to write the output of (u" %c" % 244) in XML and it will print the icon at 244. However I fear that it isn't this simple. I guess I should try to debug a bit to see how ints and icons actually fit together.

I'm afraid i'm not as skilled as you in Civ modding because i've never looked in Civ python algo yet. Try to find out how the images are called.

Nightinggale said:
Also how do you add more icons? I get how to fill out blank spaces, but how do you add spaces?

You should take a look at the tga from our mod as it contains much more cases.
http://svn.afforess.com/

I've acquired a methodology to edit Gamefont with dxtbmp and gimp which is not that hard.

From what i've found, this is much simplier than recoded the characters with a new algo (even if that would be much smarter).

dbkblk · Jun 18, 2014

After hours of searching informations, i've finally managed to merge russian with the mod gamefont in 5min of work... -_-
https://dl.dropboxusercontent.com/u/369241/Russian_test.tga

EDIT: OK but it's buggy as hell:

Is that normal ? Is there a special character call that i miss ?

Here are the modified gamefonts.

Nightinggale · Jun 18, 2014

I have had great progress. As mentioned earlier my debugging revealed that 8483 is the food icon (at least in M:C). I decided to just go with a string called "℣". That one displays as the food icon ingame. This mean it is possible to map to specific icons directly from text XML files. The next question is how to figure out which number goes to which icon. If that goes well, Russian can be added as an extra row of icons instead of replacing existing characters.

dbkblk · Jun 18, 2014

Nightinggale said:
I decided to just go with a string called "℣". That one displays as the food icon ingame. This mean it is possible to map to specific icons directly from text XML files.

Great job ! Could you detail the process when you're done ?

Nightinggale · Jun 18, 2014

Oops the forum/browser broke the important part of my post :badcomp:

The line I put in was

Code:

<English>& # 8483 ;</English>

Note that I added spaces to avoid it to be valid html code for the browser to convert. The entire line should be written without spaces.

Next step is to figure out how to look at GameFont and predict which number goes to which icon.

raystuttgart · Jun 18, 2014

1) It is possible to map some icons in specific rows of the Gamefonts by references in XML for text.

So for example you can reference stuff like that:

[ICON_BULLET]
[ICON_FOOD]
[ICON_GOLD]
...

Their relative positions are mapped by namespaces.
Anything that is not mapped by the namespaces can not directly be referenced in XML for text like that.

2) Other rows in the gamefonts are simply referenced by order or a special tag in functional XML.
Those are not meant to be referenced directly from XML for text.

You will need to get them through logic.

e.g. in Civ4:Colonization you will often find stuff like that:

Code:

gc.getYieldInfo(iYield).getIcon()

Then your logic can use them to fill defined placeholders in texts.

3) Other parts of the Gamefonts are directly referenced by position inside the exe to my knowlege.
(e.g. the Fonts)

So to my knowledge you can only replace Font characters in Gamefonts but not add to them.

If you simply add additional lines there, you will shift everything else.

---------------

I had lots and lots of discussions about those Gamefonts with other modders some years ago.
(Because I had been working very intensively on those Gamefonts.)

But as I said, that is years ago.
I also forgot a lot again during that time and may be wrong in some aspects.

---------------

General Comment:

Working on Gamefonts is quite tricky since every row follows its own rules. :crazyeye:

Yield Icons (position) behave and are referenced differently than Special Building Icons (XML tag) which again behave and are referenced differently than General Purpose Icons (namespace) which again behave and are referenced differently than Fonts (exe).

Even though a modder figured out how Yield Icons work, he might still know nothing about the way Fonts work yet.
Many modders have already gone mad because of that.

By the way:
I am mainly talking about Civ4:Colonization where there are more than 50 Yields available in some mods.
Things might be slightly different for Civ4(Bts) even though they are built on the same technical base.

Advice:

Change these Gamefonts carefully.
Otherwise you will create yourself a total mess of shifted Icons that will get extremely difficult to fix.

dbkblk · Jun 18, 2014

Pffiou. What a mess inside this.

raystuttgart · Jun 18, 2014

dbkblk said:
What a mess inside this.

Exactly.

Still can't understand how professional programmers could come up with something like this.
It is a total design desaster. :shake:

Wish you good luck though. :thumbsup:

(Hope I could help to clarify a bit.)

Nightinggale · Jun 18, 2014

raystuttgart said:
Anything that is not mapped by the namespaces can not directly be referenced in XML for text like that.

According to my last post I figured out that you can access something if you know the address (int) the icon is mapped to, regardless of namespace.

raystuttgart said:
If you simply add additional lines there, you will shift everything else.

Which is why I proposed trying to add a new row below the existing ones.

Also I think ray makes GameFont more complex than it really is. From what I get from this is that you shouldn't view the icons as rows. Instead they are one long row with some hows in it. The exe then has some specific values. For instance yield is 8483 and buildings are 8533. getChar then returns those numbers + index of the yield/building.

I managed to write a bit of python code to look into what number fetches which icon. It would appear that there is a random amount of unused numbers whenever there is a new row. I assume the number isn't random, but I haven't figured out the pattern yet. Either way I can get python to display every single icon and the number that prints it.

This ability is presumably very valuable to any mod. Sadly I coded it in M:C's domestic advisor and it relies heavily on not only colo specific data, but also M:C specific additions. This mean it isn't just a copy paste to get it into a civ4 mod.

However the idea can be coded with little effort.

Code:

for iIndex in range(1000):
	print (u" %c" %  (iIndex+8483)) + unicode(iIndex+8483)

Then add this to a list or something where you can scroll through every single value.

raystuttgart · Jun 18, 2014

Nightinggale said:
According to my last post I figured out that you can access something if you know the address (int) the icon is mapped to, regardless of namespace.

Might be possible that anything in the Gamefonts can be accessed by number. :dunno:

(Never tried that but also wouldn't like to.)

But referencing with something that "speaks" like [ICON_GOLD] is only possible if it is mapped in the namespace.

I said:
"... can not directly be referenced in XML for text like that."

I did not say:
"... can not directly be referenced in XML for text at all."

But do you really want to reference something like all characters for a Font by number encoding ?

How would your XML read if you wanted to have a text in Cyrillic letters ?
Something like "1234524236534573452435237563456234256234523 ..." ?
(Well ok, there will be some "#" and "&" but those will not make it more readable.)

And do you really want to find out all those numbers for all possible characters and icons ?

Also, what happens if you modify your Gamefonts and the numbers get shifted ?
(Those references are relative to a huge part and not absolute.)

It is quite common for a mod that a new Yield or Specialbuilding or Bonus Ressource or ... is added.
Will you then throw away all of your texts that now reference by wrong numbers and do it all again ?

Nightinggale said:
Which is why I proposed trying to add a new row below the existing ones.

And this will cause problems that are really annoying to fix.
You cannot (easily) add new lines in the Gamefonts.

Even adding new Icons at the end of a line can already result in shifting of other icons.
(Adding complete new lines will cause heavy shifting of icons almost for sure.)

Nightinggale said:
Also I think ray makes GameFont more complex than it really is.

I was just trying to give some information how referencing different parts of the Gamefonts works.
Or at least, what I know about it.

I have been working with Gamefonts a lot and have been in contact other modders that also did.
None of us enjoyed working with them.

Well, good luck. :thumbsup:

(Hope my informations were at least a bit helpful.)

Afforess · Jun 18, 2014

raystuttgart said:
Still can't understand how professional programmers could come up with something like this.
It is a total design desaster.

In my experience there is no difference between "professional" programmers and most modders on these forums, except one type has a job, and the other usually doesn't. I suspect the GameByro Engine and its limitations dictated most of the shoddy art-related design decisions. I understand Civilization 5 no longer uses that engine, and I'm fairly certain the TGA font rendering is gone as a result in Civ5.

raystuttgart · Jun 18, 2014

Well, I know nothing about the way Civ5 handles something like this or how its engine might work.

But I do still remember quite well how annoying it was for me and some befriended modders to figure out the details of those Gamefonts
so we could really do all the things we needed for our mods and how time consuming it was to fix bugs.
(A lot of bugs were simply caused because of lack of knowledge.)

I am just happy that I don't need to do this again.

(Today I know everything I need to know about them and don't run into such problems anymore.)

Nightinggale · Jun 18, 2014

raystuttgart said:
And this will cause problems that are really annoying to fix.
You cannot (easily) add new lines in the Gamefonts.

I admit it wasn't easy, but I did manage to add another line. I have no idea why I can't make that new line longer than 25 icons though. Nobody said that we are looking for an easy solution. We are looking for a working solution. Ease of use is a luxury, which may come later.

The way I added a new line was to go into the alpha channel, make it all black. Copy the bottom row of icons and paste that row in with precisely 1 pixel between the lines. After that I went to the alpha channel again and selected undo to make the area black. I saved this and opened the file in GameFontEditor where graphics could be added normally to the new row. An ingame test reveal that this new row is accessible just like the other rows.

As for design and if it is good or bad.. well I would say that it is a good solution used incorrectly. If you have 20 icons, which are often displayed, you can use a single file and then use x,y of display and x.y in offset to make the icons you need, which is what happens here. I have seen this approach before, but not in a game. I have seen it on a web server where the same icons was used over and over. The browser would then download the image once and use it from cache each time, limiting the number of requests to the server and hence reduce server load.

I have no good reason why it is used for the civ4 engine. Possibly it is to reduce the number of files to make the game load more efficient or something, but it shouldn't matter much and it does make modding harder. I agree that it wasn't the best design decision to put all icons in one file.

Nightinggale · Jun 18, 2014

I have had great progress. I copy pasted the alpha channel from the Russian Colonization into the end of GameFont_75.tga. I then added a white box for each letter and I gained the entire Russian alphabet without removing anything. However once I was done I realized a potentially very serious problem in this approach. I need to do the same to GameFont.tga and the each character must have the same index in both files. That could be a problem, but I will worry about this later. Right now I'm just happy that I have one GameFont file, which adds Russian without removing anything.

A picture tells more than a thousind words.
Here is a great deal of the story of adding Russian support.

dbkblk · Jun 19, 2014

Wouh ! I just wake up and read what you've done here ! Great job ! How did you found the reference to the new letters ? Could you send me your modded files ?

Thank you.

raystuttgart · Jun 19, 2014

Nightinggale said:
Nobody said that we are looking for an easy solution. We are looking for a working solution.

Not everything that generally works is a real solution.

But you are on an interesting path.
Maybe this will get something that can really become useful.

Nightinggale said:
Ease of use is a luxury, which may come later.

"Ease of Use" is not a luxury.
For something like writing texts, which need to be done a lot, it is essential.
(Especially since many people that are writing texts are not hardcore modders.)

But as you said, it might come later. :thumbsup:

dbkblk · Jun 19, 2014

raystuttgart said:
For something like writing texts, which need to be done a lot, it is essential.
(Especially since many people that are writing texts are not hardcore modders.)

Don't worry about that as i wrote a parser which automatically convert any set character to another. The translators can write his sentence in a translation platform and the rest is automatically converted to civ 4 format

This is why nightinggale is collaborating on this !

Nightinggale · Jun 19, 2014

dbkblk said:
Wouh ! I just wake up and read what you've done here ! Great job ! How did you found the reference to the new letters ?

See the screenshot.
Robert Surcouf made some great changes to the domestic advisor in Religion and Revolution. The important part here is that he added multiple pages for each button, hence the left and right arrows in the top corners. I then changed it to generate more pages automatically based on column width, number of columns and screen resolution.

Having this column "engine" in place, all I did was adding a new button and then add 1000 columns.

Code:

imax = 1000
for iIndex in range(imax):
	iIndexOnPage = iIndex % self.MAX_BUILDINGS_IN_A_PAGE
	iPage = iIndex // self.MAX_BUILDINGS_IN_A_PAGE
	[ iWidth, iColumns ] = self.getColumnWidth(iIndex, imax, self.MAX_BUILDINGS_IN_A_PAGE, iPage)
	self.createSubpage(self.BUILDING_STATE__, iPage, iColumns)
	screen.setTableColumnHeader( self.StatePages[self.BUILDING_STATE__][iPage] + "ListBackground", iIndexOnPage + 2, "<font=2> " + (u" %c" %  (iIndex+8483)) + unicode(iIndex+8483) + "</font>", iWidth)

I intend to clean up this hack a bit, make an on/off switch and then commit it to git in the off state as it will be good to keep for future reference whenever somebody wants to change something.
I also wonder if I can get it to display the big font in the first row or something like that.

Because it relies on one of the python files, which is specific to Colonization and the file has been rewritten in one mod, then rewritten again in another mod, I don't think it will be easy to port it to Civ4. Maybe I will look into BtS and see if I can pinpoint a place where I can copy the concept without too much work.

dbkblk said:
Could you send me your modded files ?

I attached the small font image. I'm really wondering what to do about the big font one. They need to be kept in sync. I'm currently considering moving the Russian font up before all the icons. The exe sets the int in the info classes at startup, where some offset will be needed. Replacing the exe setup with a start value written in XML for each namespace will not only allow adding the fonts, but also make it a whole lot easier to add other icons and deal with offset issues later.

The way to disable the exe setup of icon index would be to rename setChar() to setCharDLL() or something. Then I add setChar(), which does nothing. The exe has a function to call and is happy and I will have a function to call to control the char for each info class.

raystuttgart said:
"Ease of Use" is not a luxury.
For something like writing texts, which need to be done a lot, it is essential.
(Especially since many people that are writing texts are not hardcore modders.)

When I started adding Russian letters, my goal wasn't to make them reachable for translators. My goal is to make them reachable for dbkblk's translation tool. The translation tool can then convert from regular UTF-8 to whatever the game needs. The translators will write readable text and a piece of programming will deal with the part where humans have problems.

Come to think of it, maybe the tool could be build into the DLL itself. At one place it loads the string. I modded that location to pick the English string if the selected language is missing or empty. It would then not be tricky to add a next step, which is to convert UTF-8 to the needed indexes. That will allow the DLL to read the translated XML files while they are encoded in UTF-8 (read friendly). The UTF-8 standard is written in a modder friendly approach for this task. Read the string byte by byte and if the first bit in the first byte is 1, then the number needs conversion.

The TEXT XML part of the code is unmodified between BtS and Colonization. This mean it is more or less a copy paste job to move this feature into any mod, regardless of what it mods.

Speaking of TEXT XML reading, the modifications I made to M:C are inspired by Caveman2Cosmos. However C2C uses apache and because I didn't want to include a new 3rd party lib, I rewrote everything from scratch in plain C++ and it works without any libraries at all.

[Dev] GameFont.tga editing

Emperor

Deity

Emperor

Emperor

Deity

Emperor

Deity

Civ4Col Modder

Emperor

Civ4Col Modder

Deity

Civ4Col Modder

The White Wizard

Civ4Col Modder

Deity

Deity

Emperor

Civ4Col Modder

Emperor

Deity

Attachments

Similar threads