Translation / text handling related questions

[Daw]

In-game texts are built from bits taken here and there, sometimes from different files, and many bits are actually variables. Some bits - although identical - can be taken from different files, like "per citizen" production bit in Pedia entries about buildings is taken from "", while the same "per citizen" text for city screen buildings pop-up descriptions is taken from elsewhere, I have not found yet where.
And vice versa, the content of what looks to be a single pop-up message or a single text block is in fact assembled from an unknown number of bits taken from a vast variety of locations.

This sometimes makes me wonder about "where this particular thing is supposed to go?" because it affects how it shall be translated. A clear example of that is that "general" may equally mean:
- "common" if it goes into "general information";
- "military officer" if it goes to "great general".

A live example:

	<TEXT>
		<Tag>TXT_KEY_CIRCUMNAVIGATE</Tag>
		<English>Circumnavigate</English>
	</TEXT>

"Circumnavigate" what? The globe or the island? It looks like an order, but is it actually? Whatever it is, is it a "standalone" something, or is it a part of a more complex expression?

Currently I practice 2 approaches:

- Find a text that looks familiar in XML; have it translated; look for it in the game to see how it fits the context; have it corrected if needed.

- See in game a bit I want to translate; try to find it in XML (usually it's a hopeless attempt); if found successfully, return to the game to see if it was the right one (sometimes it is not, then wonder where it went instead ).

THE POINT IS: Is it REALLY inevitable to have these text sources messed up untidy shuffled organized in a way that seemingly still may be improved?

Or may be there is a way for me to find out which goes where similar to how computer does it?

 

[Daw]

CIV4GameText_MultipleYieldsConsumed.xml has this:

	<TEXT>
		<Tag>TXT_KEY_BUILDING_YIELDS_CONVERSION</Tag>
		<English>[ICON_BULLET]Converts %s1_symbol_list into %F2_yield</English>
		<Russian>[ICON_BULLET]&#1055;&#1077;&#1088;&#1077;&#1088;&#1072;&#1073;&#1072;&#1090;&#1099;&#1074;&#1072;&#1077;&#1090; %s1_symbol_list &#1074; %F2_yield</Russian>
		<French>[ICON_BULLET]Convertit %s1_symbol_list en %F2_yield</French>
		<German>[ICON_BULLET]Wandelt %s1_symbol_list in %F2_yield um</German>
		<Italian>[ICON_BULLET]Converte %s1_symbol_list in %F2_yield</Italian>
		<Spanish>[ICON_BULLET]Convierte %s1_symbol_list en %F2_yield</Spanish>
	</TEXT>

Everything works fine unless there are actually multiple yields consumed somewhere, which makes the %s1_symbol_list thing take up "and" somewhere, and this "and" thus remains untranslated. It is however translated in German, so I figure that an isolated "and" bit is floating lonely somewhere out there in the unexplored depths of the XML files.
No doubt I will eventually come across it, and will then wonder again: "where this particular thing is supposed to go?"

 

[Kailric]

What do you search with is the question? I use Microsoft Visual Studio C++ 2010 Express, it is free to download. Start this program and press ctrl+shift+f, the "Find in Files" option will pop up. Look where it says "Look in", this is where the program will search. Click the little box with the three dots, like ... This will allow you to add file locations. Add the locations of the Python, and XML, and the C++ source files.

Now, for the above mentioned TXT_KEY_CIRCUMNAVIGATE, simply do a ctrl+shift+f and paste in TXT_KEY_CIRCUMNAVIGATE and hit search. 5 locations appear for this xml tag, the important ones being python. Open the Python files by clicking on them and it will show you how this text is being used. In this case it is used as part of a command button mouse help. It is for sailing to the West side of the map, so Circumnavigate means to circle the globe.

Another example, if you see the text in game, "Immune to First Strikes" open the find in files option and enter "Immune". Then in the Find in Files window click the little + box next to "Find options".

Here you have Match Case: only text the matches the Capitalization will be shown so that words like "immune" will not appear.
Then Match whole word: so that words like immunes or immuned or swimmune would not appear. Those are not real words by the way, but if they were they would not appear.
Not exactly sure what the Use option does yet.

Of course you can just search for the whole phrase "Immune to First Strikes" and get better results but sometimes you don't have this option.

Anyway, this will greatly help you find and decipher what the text is suppose to mean or used for.

 

[Kailric]

CIV4GameText_MultipleYieldsConsumed.xml has this:

	<TEXT>
		<Tag>TXT_KEY_BUILDING_YIELDS_CONVERSION</Tag>
		<English>[ICON_BULLET]Converts %s1_symbol_list into %F2_yield</English>
		<Russian>[ICON_BULLET]&#1055;&#1077;&#1088;&#1077;&#1088;&#1072;&#1073;&#1072;&#1090;&#1099;&#1074;&#1072;&#1077;&#1090; %s1_symbol_list &#1074; %F2_yield</Russian>
		<French>[ICON_BULLET]Convertit %s1_symbol_list en %F2_yield</French>
		<German>[ICON_BULLET]Wandelt %s1_symbol_list in %F2_yield um</German>
		<Italian>[ICON_BULLET]Converte %s1_symbol_list in %F2_yield</Italian>
		<Spanish>[ICON_BULLET]Convierte %s1_symbol_list en %F2_yield</Spanish>
	</TEXT>

Everything works fine unless there are actually multiple yields consumed somewhere, which makes the %s1_symbol_list thing take up "and" somewhere, and this "and" thus remains untranslated. It is however translated in German, so I figure that an isolated "and" bit is floating lonely somewhere out there in the unexplored depths of the XML files.
No doubt I will eventually come across it, and will then wonder again: "where this particular thing is supposed to go?"

Ok, doing my above example I did a search for TXT_KEY_BUILDING_YIELDS_CONVERSION, which is found in CvGameTextMgr.cpp. Opening this file you can see how this text is used. But more importantly you can see the xml reference TXT_KEY_AND, there is your AND

Now, search for TXT_KEY_AND and you will be able to easily add your translation. It is actually in the orginal files, so you may need to copy the whole:

<TEXT>
	<Tag>TXT_KEY_AND</Tag>
	<English>[SPACE]and[SPACE]</English>
	<French>[SPACE]et[SPACE]</French>
	<German>[SPACE]und[SPACE]</German>
	<Italian>[SPACE]e[SPACE]</Italian>
	<Spanish>[SPACE]y[SPACE]</Spanish>
</TEXT>

into a M:C text file, such as CIV4GameText_Medieval.xml and then add your Russian. Not sure if you can just do:

<TEXT>
	<Tag>TXT_KEY_AND</Tag>
	<Russian>[SPACE]and[SPACE]</Russian>
</TEXT>

probably not.

 

[Daw]

Your white men dark magic is really creepy But strangely enough I was able to reproduce it and it works!

EDIT:
However, this means that everything that is used from the original text files (like names for terrain types, features, bonuses, resources, yields, improvements, etc. and also main menu buttons text, screen interface text and so on) is to be copied somewhere into the mod's text files. And either I got very lucky or the copied stuff can be dropped actually anywhere, and the game will find it anyway as long as its syntax is not broken. Right?

If it's right, I humbly suggest that new files are created for that if possible. I mean, there's totally no difference from the computer's point of view as to where to search for this or that text bit. But I'm just a human, so I find it convenient when the files names tell me what sort of text is in there without any C++.

Like:
- UnitsTexts for units names, descriptions, pedia entries;
- TerrainTexts for everything about terrain;
- FeaturesTexts - for features;
- BuildingsTexts;
- YieldsTexts;
- PromotionsTexts;
- ProfessionsTexts;
- MainInterfaceTexts;
- etc.

Now if that's agreeable, I can create files but I am not sure if I am able to upload them with GIT. I more or less got used to cloning/pulling/committing/pushing (and really got on Nightinggale's nerves in the process; I'm sincerely very grateful for the patience and the time he spared) but uploading/creating a new file instead of updating an existing one may turn out to be something, which requires more than one attempt if I try to do it.

What do you think?

 

[Kailric]

I am not exactly sure how the code handles text but you can as you say put text in any text file. It would be a huge job to separate all the text as you say and there is probably better things to do with your time. Like the Units, Professions, Buildings, Leaders, etc. all need pedia descriptions filled in, English and all other languages. That would be a better project.

As far as the original text files, I am not sure what the easiest and fastest thing to do is. We could copy all the original files into the mod directory and start to edit them there. That may be a good idea as then you and Night can release a Russian version of vanilla Col for Russian speakers and modders.

 

[Daw]

Like the Units, Professions, Buildings, Leaders, etc. all need pedia descriptions filled in, English and all other languages. That would be a better project.

This is exactly what I'm talking about. Since this all has to be created anyway, why not do it in an orderly fashion as well?

That may be a good idea as then you and Night can release a Russian version of vanilla Col for Russian speakers and modders.

There's good news and bad news on that.

Good news is that other people have already done it for us long ago.

Bad news is that they messed up featured something so that Russian vanilla is incompatible with any EU/US version based mod. And vice versa: you can't run a Russian vanilla based mod on EU/US vanilla version.

Yet another good news is that it still provides a substantial heritage. For one is the Cyrillic character set we now successfully use. Second, they must have all vanilla text translated, so I probably only have to copy/paste it in some way that it fits well with the mod.

As far as the original text files, I am not sure what the easiest and fastest thing to do is. We could copy all the original files into the mod directory and start to edit them there.

I assume it is inevitable that we do it eventually, because if you have EU/US vanilla version, you can't have any Russian text that does not come with the mod. This means all the vanilla text that is still kept in the mod must come with the mod. And if you play Russian vanilla version, you can't have the mod itself (well, you can, but it won't run).

 

[Kailric]

Well, you could start now and keep the entries separated first and that will make it easier to go back and separate out the older text entries.

I just thought of something though, you can't really add the original files to the mod as I changed specific entries to fit with the M:C theme so that would mean some entries would be duplicated, and perhaps show the wrong entry. So, the only thing really to do is when you see text in the game and you find it in the original files you will need to add it to the M:C text xmls.

 

[Daw]

Well, you could start now and keep the entries separated first and that will make it easier to go back and separate out the older text entries.

Yes, I think it's what I'm going to do.

I just thought of something though, you can't really add the original files to the mod as I changed specific entries to fit with the M:C theme so that would mean some entries would be duplicated, and perhaps show the wrong entry. So, the only thing really to do is when you see text in the game and you find it in the original files you will need to add it to the M:C text xmls.

Yes, I meant only transferring those files/bits, which are not duplicated. I plan to do it as I move along. This approach will allow surgically accurate transplantation rather than blind massive dumping, which supposedly would hurt more than help.

BTW, speaking of delayed realizing things talked about some time ago, I recall several times hearing something that, as I now realize, was about this:

<CivilizationInfo>
			<Type>CIVILIZATION_ANGLO_SAXON</Type>
			<Description>[B]Anglo-Saxon[/B]</Description>
			<ShortDescription>[B]Anglo-Saxon[/B]</ShortDescription>
			<Adjective>[B]Anglo-Saxon[/B]</Adjective>
			<Civilopedia>TXT_KEY_CHARL_CIV_ANGLOSAXON_PEDIA</Civilopedia>

referring to that there must be not "Anglo-Saxon" but "TXT_KEY_ANGLOSAXON_SOMETHING". And then somewhere there shall be something like:

<Text>
			<Tag>TXT_KEY_ANGLOSAXON_SOMETHING</Tag>
			<English>[B]Anglo-Saxon[/B]</English>
			...
</Text>

However I found out by trial and error that I cannot just add it (or I accidentally broke something in the process as my Civ appeared to be named literally TXT_KEY_ANGLOSAXON_SOMETHING

). I need to add it properly.

So, any hints on what the proper way is?

 

[Kailric]

If your Civ or what ever shows a text key such as, TXT_KEY_ANGLOSAXON_SOMETHING, in the game then something went wrong when the code tried to find TXT_KEY_ANGLOSAXON_SOMETHING in the XML text files. Perhaps you forgot to add the TXT_KEY or misspelled the text key such as TXT_KEY_ANGLOSAXON_SOMETHNG. There needs to be two places for the TXT_KEY. For example; one in the CIV4LeaderHeadInfos.xml that points to the translation in a text file xml, such as CIV4GameText_Medieval.xml.

Actually, I am not sure if I understand what you are asking but maybe that helps?

 

[Daw]

Yes, it brilliantly does! It helps by proving the point that I had correct overall idea, but...

I accidentally broke something in the process

... due to my kackhandedness.

Now I know I did the right thing only I have to do it more carefully. Thanks!

 

[Daw]

This does not look right. The text bit is collected all right, but Cyrillic bit is shown in only one place of the three, while Latin D (this is for "description" which I marked bits with for myself to see which one comes here) is shown in all of them.

Both Latin and Cyrillic symbols are shown in:
- the pop-up message balloon;
Only Latin shows up in:
- the List of Nations
- after the Leader's name.

All three entries come from the same line in the same file.

Spoiler :

Update:
More of probably the same issue (I removed Latin characters to get pure picture):

1.

None of the Description/Short_Description/Adjective are shown in Your Details box, though collected and placed (and is even selectable):

Spoiler :

2.

I can found the would be glorious city of Лондон (the river is probably the Thames) but the city name does not show in the pop-up "rename" box. It is there, as you can guess based on the cursor position, it's only that it can't be seen.

Spoiler :

3.

My "Лондон" London is named "Лондон" exactly as it is supposed to on the city screen,

Spoiler :

...but the pop-up "rename" box is as expected - no different from when it's called up from the main map screen.

BTW it's quite odd to have population 100 while max population is 5, isn't it?

4.

"History" pedia box title has deserted its usual position and gone lost ("don't panic"(c), probably just got invisible) after it got translated.

Spoiler :

Now, does anyone have enough mana to cast "Reveal" spell on these hideaways?

 

[Kailric]

Hmm, not sure about that. Maybe Nightinggale can help. I figure the popup is done in the dll and the other two are done with python.

 

[Nightinggale]

Not sure if you can just do:

<TEXT>
	<Tag>TXT_KEY_AND</Tag>
	<Russian>[SPACE]and[SPACE]</Russian>
</TEXT>

probably not.

I know you can't. However TXT_KEY_AND makes me wonder. The more we make sentences out of parts of sentences, the harder it is to translate. We might rethink how we do this.

However, this means that everything that is used from the original text files (like names for terrain types, features, bonuses, resources, yields, improvements, etc. and also main menu buttons text, screen interface text and so on) is to be copied somewhere into the mod's text files. And either I got very lucky or the copied stuff can be dropped actually anywhere, and the game will find it anyway as long as its syntax is not broken. Right?

If it's right, I humbly suggest that new files are created for that if possible. I mean, there's totally no difference from the computer's point of view as to where to search for this or that text bit. But I'm just a human, so I find it convenient when the files names tell me what sort of text is in there without any C++.

Like:
- UnitsTexts for units names, descriptions, pedia entries;
- TerrainTexts for everything about terrain;
- FeaturesTexts - for features;
- BuildingsTexts;
- YieldsTexts;
- PromotionsTexts;
- ProfessionsTexts;
- MainInterfaceTexts;
- etc.

You are right that we can place it in whatever file we like. I plan to do precisely this in the near future. I have an idea for a perl script, which will do most of the work.

I also want to copy every single vanilla file into the mod and then delete all unused strings.

I can create files but I am not sure if I am able to upload them with GIT.

Select the file (you may have to unfilter it with the filter buttons in the upper right) and click stage. You can then commit and push new files just like you with modified files. I noticed that you commit files on at a time. It is in fact possible to commit multiple files at once.

BTW, speaking of delayed realizing things talked about some time ago, I recall several times hearing something that, as I now realize, was about this:

<CivilizationInfo>
			<Type>CIVILIZATION_ANGLO_SAXON</Type>
			<Description>[B]Anglo-Saxon[/B]</Description>
			<ShortDescription>[B]Anglo-Saxon[/B]</ShortDescription>
			<Adjective>[B]Anglo-Saxon[/B]</Adjective>
			<Civilopedia>TXT_KEY_CHARL_CIV_ANGLOSAXON_PEDIA</Civilopedia>

referring to that there must be not "Anglo-Saxon" but "TXT_KEY_ANGLOSAXON_SOMETHING". And then somewhere there shall be something like:

<Text>
			<Tag>TXT_KEY_ANGLOSAXON_SOMETHING</Tag>
			<English>[B]Anglo-Saxon[/B]</English>
			...
</Text>

However I found out by trial and error that I cannot just add it (or I accidentally broke something in the process as my Civ appeared to be named literally TXT_KEY_ANGLOSAXON_SOMETHING

). I need to add it properly.

So, any hints on what the proper way is?

I have no idea what you did to break it. However you should be able to do this:

<CivilizationInfo>
			<Type>CIVILIZATION_ANGLO_SAXON</Type>
			<Description>TXT_KEY_CIVILIZATION_ANGLO_SAXON_DESCRIPTION</Description>
			<ShortDescription>TXT_KEY_CIVILIZATION_ANGLO_SAXON_SHORT_DESCRIPTION</ShortDescription>
			<Adjective>TXT_KEY_CIVILIZATION_ANGLO_SAXON_ADJECTIVE</Adjective>
			<Civilopedia>TXT_KEY_CHARL_CIV_ANGLOSAXON_PEDIA</Civilopedia>

Then in any XML file in Text, do

<TEXT>
	<Tag>TXT_KEY_CIVILIZATION_ANGLO_SAXON_DESCRIPTION</Tag>
	<English>Anglo-Saxon</English>
</TEXT>
<TEXT>
	<Tag>TXT_KEY_CIVILIZATION_ANGLO_SAXON_SHORT_DESCRIPTION</Tag>
	<English>Anglo-Saxon</English>
</TEXT>
<TEXT>
	<Tag>TXT_KEY_CIVILIZATION_ANGLO_SAXON_ADJECTIVE</Tag>
	<English>Anglo-Saxon</English>
</TEXT>

I think it would be a good idea to make a description of how Description, ShortDescription and Adjective are used. Right now it would appear that it would be enough to just have one of them. However since there are 3 different strings, there are 3 ways to use it. I think it's something like this:
Description: Great Britain
Short Description: GB
Adjective: British
However it would be nice to be sure.

 

[Daw]

Post #12 updated with more "invisible text" bug data to analyze.

I know you can't. However TXT_KEY_AND makes me wonder. The more we make sentences out of parts of sentences, the harder it is to translate. We might rethink how we do this.

Well, it's up to you. Your idea is generally right, but there will be variables anyway. They have to, otherwise it is not coding but writing a book.

So far I did not find anything that can't be solved elegantly with rephrasing.

You are right that we can place it in whatever file we like. I plan to do precisely this in the near future. I have an idea for a perl script, which will do most of the work.

I also want to copy every single vanilla file into the mod and then delete all unused strings.

I am not sure assorting is good to be done machine way. I am moving from bit to bit manually anyway. This is inevitable because every time I translate a bit, I need to see how it fits the environment it was supposed for (it is really rewarding to see it fits right, so it is an important motivational part for me, not only quality assurance). And as I move along I can as well cut/paste tings to somewhere, thus leaving behind a structure of any design you like.

However, having unused strings deleted by a machine would be a real blessing.

Select the file (you may have to unfilter it with the filter buttons in the upper right) and click stage. You can then commit and push new files just like you with modified files. I noticed that you commit files on at a time. It is in fact possible to commit multiple files at once.

I managed to commit&push my CivilizationsTexts_Medieval.xml without either prior staging (I didn't know I am supposed to do that) or any error messages. I hope I did not break anything again.

I have no idea what you did to break it.

I have no idea myself Breaking things in most unimaginable ways is one of my award winning talents However, I did it again very carefully, and this part works now.

I think it would be a good idea to make a description of how Description, ShortDescription and Adjective are used. Right now it would appear that it would be enough to just have one of them. However since there are 3 different strings, there are 3 ways to use it. I think it's something like this:
Description: Great Britain
Short Description: GB
Adjective: British
However it would be nice to be sure.

Maybe, but not very necessary for Russian part. Language works somewhat differently there.

 

[Nightinggale]

I am not sure assorting is good to be done machine way.

I'm thinking like going through say CIV4UnitInfos.xml and then add strings to all descriptions and pedias. Those strings will then be dumped in a new file for unit names and we will have more or less all strings related to unts in one file and nothing else. Repeat that with all the other files with hardcoded strings.

The main reason is actually to allow translation access to all strings.

However, having unused strings deleted by a machine would be a real blessing.

I know it will be for several reasons. I just need to figure out precisely how to do this and if the translation tool made for civ4 can be used or I have to make my own.



I have investigated the font issues. It turns out that at certain locations, like main menu, the game will not use GameFont. Instead it will use Sylfaen.ttf. However if the character ID is higher than 255, then it will use GameFont. Text color and size will only work if Sylfaen.ttf is used.

To make this worse, this is hardcoded in the exe. We can't even intercept the string and modify IDs based on where the caller is.

I see no alternative to replacing vanilla chars to get room for Russian ones. Bye bye … (that's one character!).

 

[Nightinggale]

I have had a major breakthrough. I think I figured out how the game displays text/icons.

- Sylfaen.ttf is used if the ID is below 256
- GameFont is used if is is higher than 8482
- the ID range between those two numbers appears to be unreachable

The exception to this is city billboards. They always use GameFont.

What this mean for Russian is that the characters should be added to Sylfaen.ttf. We need to add 64 characters and it looks to me that there are 68 unused IDs. Even better the font already has Russian characters meaning it is a matter of changing IDs, not creating new characters.

The unused char IDs lack slots in GameFont. The solution is then to write a function to convert from font IDs to GameFont IDs (shouldn't be difficult). The two functions to set billboard text should then run the conversion before returning the string in question.

This should allow Russian characters in the font using font IDs, which allows resizing, color and all that. When writing on billboards, the IDs from GameFont will be used and it will still look right.

 

[Daw]

I'm thinking like going through say CIV4UnitInfos.xml and then add strings to all descriptions and pedias. Those strings will then be dumped in a new file for unit names and we will have more or less all strings related to unts in one file and nothing else. Repeat that with all the other files with hardcoded strings.

The main reason is actually to allow translation access to all strings.

I know it will be for several reasons. I just need to figure out precisely how to do this and if the translation tool made for civ4 can be used or I have to make my own.

Sounds as good as it is

I have investigated the font issues. It turns out that at certain locations, like main menu, the game will not use GameFont. Instead it will use Sylfaen.ttf. However if the character ID is higher than 255, then it will use GameFont. Text color and size will only work if Sylfaen.ttf is used.

To make this worse, this is hardcoded in the exe. We can't even intercept the string and modify IDs based on where the caller is.

I see no alternative to replacing vanilla chars to get room for Russian ones. Bye bye … (that's one character!).

I still think that formatting issues are minor ones. The character set we currently use is just

- meaning we just haven't found anything BETA yet, and/or haven't bothered to make it BETA, too.

Even now main menu looks mush less scary will ALL CAPS and no Latin in sight to compare with.

Spoiler :

It's far from being a pink of aesthetic perfection, but may become much better with an improved character set, and definitely will do for now.

As for always white text, well... yes, it is unpleasant, but it may be OK if the text is logical enough, and it is definitely better than not having any text at all.

Speaking of which, I am much more worried about that invisible text.

I even thought about if it is possible to create some text boxes of our own and to place them over those refusing-to-cooperate ones. If possible at all, these boxes might be also enabled/disabled by switching to and from Russian language in the menu.

 

[Nightinggale]

Speaking of which, I am much more worried about that invisible text.

I will try to mod the font into showing Russian at the IDs we want it to use and then see if the invisible text will turn visible.

 

[Daw]

I will try to mod the font into showing Russian at the IDs we want it to use and then see if the invisible text will turn visible.

Yeah, I wrote last 3 passages of post #18 before I read your post #17. And now I'm praying to all Gods I can remember names of (and that's a long list) asking them to grant you luck in you noble endeavor and support you all the way to success.