[Dev tool] Civilization 4 XML translation tool

[dbkblk]

Hisuipanda seems to understand what's going on. I've tried to understand what "she" (jade?) where saying in Twitter but haven't understood the whole story. It would be nice to have a discussion here. Maybe she was saying to use the chinese nocd patch.

EDIT: Maybe i'm stupid, but i do not know how to get in touch from twitter without subscribing

EDIT2: The game crash for me with the Chinese NoCD. A lot of python errors, then the executable shuts down.

EDIT3: The cyberfront code seems to work on some sentences, but not on others :S

 

[Milliopolis]

@dbklk
"We want to do the reverse of this, inject japanese into the european version!"
It seems very difficult because English ver originally supports 1-byte character.
If you succeed in coexistence 1-byte and 2-byte character on English ver,it may mean replacing main civ4 contents not only MOD folder.
Playing BTS or other MODs,that change will cause some new error.
Civ4 multi wiki team has a lot of member and thier skill is various and good.
You should contact them not Hisui.

@Nightinggale
もし日本語を読むのに支障がなければ私が英語で話すのが難しかったり遅かったりする事柄をdbkblkに伝えることを手伝ってもらえないだろうか
特に"長期的プロジェクト Civilization4 THE NEXT 10 YEARS ～Quality Up ＆ Preservation Project～について"
http://civ4multi.info/loadmap/
この計画は言語を問わずciv4を愛する人達にとって有益だしその実現は困難に見える
もし日本以外に伝わっていたり同じような計画があるのならマルチwikiとIPA-ZONEのメンバーの助けになると思う

 

[dbkblk]

@Miliopolis: I understand your concern and that's why we do not want to replace default content. We'll try to look if the executable has been modified in the japanese version.
About main contents, we found that enabling UTF-8 in python files allows to display 1-byte non-latin1 languages, so that means the python part can handle UTF-8 correctly (but some code may need to be fixed in the mod). I may be too confident but i don't think this is a problematic part.

EDIT: The problem about getting in touch with Civ4 multi wiki team is I don't understand a word of Japanese .
EDIT2: I've posted a message here: http://civ4multi.info/forums/topic/civ-a-new-dawn/

@Nightinggale: After re-reading what we have done before, i've remembered the GameBryo engine support UCS-2 for two bytes chars, but not UTF-8. The solution might reside in: UTF-8 > UCS-2 > Locale.

 

[Nightinggale]

@Nightinggale
もし日本語を読むのに支障が

Sadly that is not really the case. It's impossible for me to read that without a dictionary and with one... well my reading speed is... let's just say I will not read a whole book

(the text says something about reading Japanese not being a obstacle for me)

http://civ4multi.info/loadmap/

The link is interesting though. The URL contain Engrish though, as it is spelled Roadmap.

見やすいフォント導入、ユニバーサルデザイン

There is a post about fonts: http://forums.civfanatics.com/showpost.php?p=13652618&postcount=67
Any other insights on how to improve fonts would be most welcome.

There are some items on the roadmap I don't really get. Why would you request HD graphics? It works fine for me at 2560x1440.

CPUのマルチコア化 64bitOSへ対応

I have no idea how you plan to convert the 32 bit exe file into a 64 bit file, but if you figure that out, then please let me know
Adding multi CPU core support is possible to do in the DLL code as it is right now. In fact I have done it, but never published the code because it didn't make the game faster at all. Adding multi-core support to something, which isn't designed for it from the start is very hard.

 

[dbkblk]

Sadly that is not really the case. It's impossible for me to read that without a dictionary and with one... well my reading speed is... let's just say I will not read a whole book

I'm sure i read it faster using Google Translate... but it's way less smart

There is a post about fonts: http://forums.civfanatics.com/showpost.php?p=13652618&postcount=67
Any other insights on how to improve fonts would be most welcome.

I forgot to report this but i've made progress on this side. In "Civ4Theme_Common.thm", set

GFont	.Size4_Bold				=	GFont("Trebuchet MS",		"Bold",			15, GFlags(GFontFeature, GFC_FONT_BOLD,	GFC_FONT_ALPHA));

to Trebuchet 15 improve the text rendering on fScale -1.0. I've also limited string size in billboard. Look at rev890 (CvInfos.cpp) http://sourceforge.net/p/anewdawn/code/commit_browser

 

[dbkblk]

I wrote some code to read UTF-8 and store the unicode ID (the real unicode ID). Technically it converts from UTF-8 to UTF-32. Then the iconv tables are used to convert that number back into a single byte using a codepage encoding. If we are to use a two byte codepage (assuming the exe accepts it), the UTF-32 has to be converted into that codepage.

Based on the previously quoted documentation here, i'll try a similar approach as your by writing a on-the-fly converter UTF-8 to UCS-2, because GameBryo apparently supports it.

 

[dbkblk]

....well that is not exactly korean but it shows that the english executable is able to display korean:

I've tried to convert a text file to EUC-KR and bypassed korean by injecting CvString to CvWString. wszNewString.append(CvWString(szTextVal));
Research still need to progress as the first line should be "싱글 플레이어".

I'm not sure the japanese executable has something interesting. I think it only contains translated error messages and that's why it's bigger (more bytes used by letters). But for sure it's just an assumption.

EDIT: On the previous screenshot, i've made a HUGE error. I noticed later that i was in cp932 while working on korean. But that still means it's possible to display japanese chars in game on the english executable. Sadly, these are the 1-byte chars of cp932.

 

[dbkblk]

I have understood why the iconv accepts only two chars. It's because it input UCS and not UTF-8.

That said, i rewrote the parser to use the old method to convert russian, but i've tried to map the unicode chars to cp949 chars (荶 for example), using this table: http://unicode.org/Public/MAPPINGS/VENDORS/MICSFT/WINDOWS/CP949.TXT
The problem is that the game still thinks it is UTF8 and input three bytes. With the XML set to ISO-8859-1 it still input three bytes.
What i can't understand right now is that the game now displays accentued chars, which are in 8859-1 but my codepage is on 949, so it shouldn't.

EDIT: I have an idea but it's unclear in my head. I note it here to not forget it: Set a negative offset to the cp949 numerical html value.

 

[Nightinggale]

Based on the previously quoted documentation here, i'll try a similar approach as your by writing a on-the-fly converter UTF-8 to UCS-2, because GameBryo apparently supports it.

Alternatively we can encode the XML files using UCS-2. Notepad++ supports it with both big endian and little endian. I think windows assumes LE (since x86 is LE), but I'm not sure.

 

[dbkblk]

I've already tried that but the file couldn't be read.

 

[Nightinggale]

It would likely be easiest to use existing code for encoding conversion. I found this one, which could be interesting:
https://gist.github.com/suapapa/2374101

Adding a full blown iconv could be an issue though unless it is statically linked. I don't think people want to add a DLL next to the exe file.

 

[dbkblk]

In fact, i've came to something simplier. Before to try to convert existing text. We already need to try to display A 2-byte char.

wchar c[2];
c[0] = 0xAE;
c[1] = 0xE6;
wszNewString.append(c);

EDIT: This will display two chars: 0xAE and 0xE6 instead of 0xAEE6 :S
EDIT2: Finally, the executable might be the reason of two-bytes display. Interesting fact: Launching the european version with the japanese DLL display no text.

 

[dbkblk]

OK. So i have a good news, and a bad news.
I've replaced the files of the japanese version one by one with the ones from the european. I've found that the python modification are only to enhance the japanese version but not critical. Which brings us to the good news:
I was able to display a korean char using the japanese executable. I still need to investigate but... that brings us to the bad news:
The compatibility with 2-bytes seems indeed linked to the executable. I don't know how we can port that to Colonization unfortunately

I need to find a way to include the modification into our DLL and to provide the modified executable for the asian players while keeping a full compatibility with the european version.

EDIT: There are still problems but...

 

[Milliopolis]

@Nightinggale
"Why would you request HD graphics? It works fine for me at 2560x1440. "
Because IPA-ZONE says civ4's graphics support HD and over that, expands graphics.
So he planned to make FHD graphics and textures.

I introduce this and multi wiki threads to twitter civ4 community in Japan.
@civ4_Multi and @zone_z_official don't react yet.
You should contact them and their member,you want.

 

[Nightinggale]

The compatibility with 2-bytes seems indeed linked to the executable. I don't know how we can port that to Colonization unfortunately

Let's forget about Colonization for now. There is an even bigger problem in regards to replacing the exe. In theory the Japanese exe can be renamed and placed next to the English exe. However the steam version will not accept that and I wonder how many users can figure out to do this and use the correct exe. Getting display of all languages to work will not really gain anything if more than half the users can't install it.

 

[dbkblk]

However the steam version will not accept that and I wonder how many users can figure out to do this and use the correct exe.

That is not THIS problematic. I don't know if you played our mod but i've developed a launcher to make game configuration easier. It also allows easy update (click on a button and wait ). So i can manage to invite the user to replace the executable, or rename it to something different. Then, the user can launch the mod through the launcher which then point to the renamed executable if the language is set to an Asian one.

EDIT: I also opened the executable through ResHacker and discovered the artwork displayed at launch is compiled as a resource in the executable. So we can even change the name and the icons of the exe
About Steam, it deserves a test.

 

[dbkblk]

Now that is a REAL progress (Japanese here):

Another evidence

 

[Milliopolis]

Bravo！

 

[dbkblk]

A Korean screen:

 

[dbkblk]

Some news about the current implementation:

• Chinese (51% translated), Japanese (66%) and Korean (72%) are now implemented in game.
• The launcher is able to change to these languages.
• A small patch is needed to be applied on the European/US version to play but it does not affect any original file (you can continue to play the english version and mods compatible only with this one). It only copies a new executable and generate symlinks.
• If you try to start the game through the launcher with those, it will automatically switch to the asian executable and if the patch is not applied, it will invite the user to apply it.
• It only works with the version 3.19 of the game (you won't be able to play with players which have the latest steam version).
• I still need to implement parts of code to optimize CJK display.
• EDIT: GameFonts are broken for Asian languages at the moment.

我希望你享受游戏的乐趣
私はあなたがゲームをお楽しみくださ
난 당신이 게임을 즐기시기 바랍니다

Bravo！

Merci