1. We have added a Gift Upgrades feature that allows you to gift an account upgrade to another member, just in time for the holiday season. You can see the gift option when going to the Account Upgrades screen, or on any user profile screen.
    Dismiss Notice

Translation / text handling related questions

Discussion in 'Civ4Col - Medieval: Conquests' started by Daw, Jul 9, 2014.

  1. Nightinggale

    Nightinggale Deity

    Joined:
    Feb 2, 2009
    Messages:
    4,354
    I tried adding Russian to Sylfaen.ttf. However because a number of IDs can't be used, there is only 53 free character slots. This mean we lack 11 to fully implement Russian :cry:

    I see two options, none of them perfect.

    1:
    we make a new Sylfaen.ttf where Russian overwrites existing characters. People will then have to install different files depending on their choice of language.

    2:
    We make a reduced Russian set where a number of characters use the latin counterpart. For instance both A and А in XML will use the latin A. We need at least 6 shared characters for this to work, preferably even more.

    Why 1 is better than 2:
    Sorting alphabetically works and 30 characters can change between lower and upper case without extra code. (we want that for 32 characters :()

    Why 2 is better than 1:
    Savegame and multiplayer compatibility issues in 1. Same font file for all languages.

    I don't know what to do now. How should we have Russian work ingame?

    What annoys me the most is that this is a minor coding issue in the exe. If I had the source code, I would have increased the limit from 255 to 8000 and we would be able to add Russian without even modifying the font file. I suspect that would take 5-10 minutes of coding time to make that change.
     
  2. Daw

    Daw Emperor

    Joined:
    Sep 30, 2013
    Messages:
    1,782
    The Rusifier we started with comes with its own Sylfaen.ttf. I see no reason for that but that it has already been amended the way you said. The link I sent you must still be working if you discarded the distributive.

    Shared characters:

    Latin:
    ABCDEFGHIGKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz

    Cyrillic:
    АБВГДЕЖЗИЙКЛМНОПРСТУФХЦЧШЩЪЫЬЭЮЯабвгдежзийклмнопрстуфхцчшщъыьэюя

    EDIT: I has just occurred to me that З character can be shared with 3 numeral.
     
  3. Daw

    Daw Emperor

    Joined:
    Sep 30, 2013
    Messages:
    1,782
    Does this feature really matter that much? I mean:
    - How often it is used (maybe it is possible to do without it at all like in main menu where English text is converted to uppercase while I just entered Russian uppercase in the XML to have the same result);
    - If we are able to precisely locate exact strings for which the conversion is used, I bet there are good chances I can make it so that 30 characters will be enough.

    By the way it reminds me of the missing "Ч" mystery. May be it was not accidentally missed but rather deliberately omitted?.. However I don't understand the reasons for the candidate choice. Omitting "Ъ" would seem more logical: "Ч" is rare to find, but finding "Ъ" is almost impossible.

    Sounds preferable to me unless it appears that alphabetic sorting and lowercase to uppercase conversion are definitely not expendable from coding point of view.

    I prefer 2. However it is probably because I understand it better.

    Well,

    Firstly, probably that's why Russian vanilla is incompatible with EU/US version.

    Secondly, like you told me the other day, "Never modify vanilla, because if people start doing it, it all will end up with incompatibility."

    Thirdly, look at the bright side. The old wisdom puts it that "Even if you've been swallowed, you still have 2 ways out." You have just suggested the 2 ways. And it's definitely better than no way at all. :goodjob:
     
  4. Nightinggale

    Nightinggale Deity

    Joined:
    Feb 2, 2009
    Messages:
    4,354
    I naturally looked at this file and I don't like it. For some reason latin chars are modified to look worse. The Russian chars in the vanilla font looks ok to me. It's just a matter of assigning new IDs to them to make them reachable ingame.

    Because there is the turning to uppercase code (shame on it), we should try to share characters, which are the same in both lower and upper case. That would be A, E, O, P, C and X. That's 6 characters, which mean a reduction of 12 slots. That is more than the needed 11 meaning this should be enough to get this working.

    However the list is interesting for another reason. We can reduce the Russian size in GameFont by 18. Sounds tempting considering it is in the range, which can be used on billboards.

    The only real issue with solution 2 would be that the shared characters would change case while the non-shared ones don't. It will be in main menu and some python files (mainly pedia).

    I think they share A, then add the entire alphabet. The very last character is ÿ, but Ÿ isn't part of the ISO set. Funny enough it is actually reachable in vanilla anyway, but I don't think I would ever have discovered that without my debug display. The exe makes something completely illogical, which gives access to around 20 characters we shouldn't have access to :crazyeye:
    Why they picked Ч and not Ъ? I have no idea. It doesn't look like there is a technical reason for it.

    I can't change vanilla/the exe for this (though it would rule if I could). Russian support needs a modified font file, which is kind of like modifying vanilla. However my goal is to modify the parts vanilla don't use. That way it doesn't matter if people use a modified or non-modified file unless they switch to Russian.
     
  5. Daw

    Daw Emperor

    Joined:
    Sep 30, 2013
    Messages:
    1,782
    It can be avoided if the text in question is uppercase in XML anyway, can't it? This will definitely work for the main menu as text from TXT_KEY_MAIN_MENU_* is (at least as far as I can see) not used anywhere else but in the main menu itself.

    The "python" as I understand it means here:
    1. names of units/professions/yields/etc. which turn uppercase in the respective pedia page header;
    2. names of Nations/Leaders which turn uppercase in the screens that pop-up at the game start for the player to inspect and confirm who he is in game.

    These two are something I don't know what to think about. The only solution I can think of is to have different "TXT_KEY"s for uppercase/lowercase places.

    I assume that's the way they keep aphabetic sorting, as both Latin and Cyrillic start with Aa, so sharing these will not harm anything. Then they added the entire alphabet as having other Latin symbols will place them before any Cyrillics on the list, which will distort the listing.

    The only reason I can imagine is that in vanilla vocabulary there are no words to contain ч to turn uppercase or start with uppercase Ч, while there was some word containing ъ to turn uppercase. :dunno:

    Anyway, this may be very different with mod vocabulary.
     
  6. Nightinggale

    Nightinggale Deity

    Joined:
    Feb 2, 2009
    Messages:
    4,354
    Remember that vanilla files are used unless the mod has a file of the same name.

    Python/Screens/CvHallOfFameScreen.py uses TXT_KEY_MAIN_MENU_SINGLE_PLAYER and TXT_KEY_MAIN_MENU_MULTIPLAYER without changing case. There are a few both in DLL and python using TXT_KEY_MAIN_MENU_*. However those files are few and can be modded if needed. The main issue here is that the main menu generation is in the exe and is hardcoded to use strings like TXT_KEY_MAIN_MENU_SINGLE_PLAYER and convert those to uppercase.

    Good idea, but it seems like a lot of XML work. I wonder if we could take it one step further. Imagine this:

    When the DLL reads a string, it does it twice. First time is the regular time just like it is doing right now.

    Second time it appends _UPPER to the string name and it converts the string to uppercase while reading. As it has the UTF-8 character at this point, it can tell the difference between у and y meaning they can be uppercased to У and Y respectively.

    Now .upper() should no longer be used. Instead you append _UPPER like
    Code:
    localText.getText("TXT_KEY_PEDIA_SCREEN_EXIT", ()).upper()
    should be written
    Code:
    localText.getText("TXT_KEY_PEDIA_SCREEN_EXIT[COLOR="Red"]_UPPER[/COLOR]", ())
    The other variant of using upper would be like this
    Code:
    szText.upper()
    turn it into
    Code:
    (szText + "_UPPER")
    No XML changes, no big coding change and we have put the converting to uppercase into a function we have 100% control over meaning there will no longer be special meanings to certain character IDs.

    There are two bad parts to this though. One is performance and memory, though I suspect it is acceptable (it's nothing major). The other one is the inability to make random text uppercase. We will then not allow code to make say a custom city name uppercase. I think we can live with that (are we even using anything like that?)
     
  7. Daw

    Daw Emperor

    Joined:
    Sep 30, 2013
    Messages:
    1,782
    And since we can't change exe hardcoded to use strings like TXT_KEY_MAIN_MENU_SINGLE_PLAYER, we still can change python so that is uses something like TXT_KEY_MAIN_MENU_SINGLE_PLAYER_LOWER and have two strings then: uppercase for main menu, lowercase for elsewhere.

    I mean, these places, which are hardcoded in exe, are few, aren't they? So it's not like a pile of work to settle an alternative string to such places.

    Well, anyway, currently I'd be happy if text was shown any case at all:
    Spoiler :


    Imagined. Found it tempting. Now wondering if I am able to put it right, not awry.

    Well, I do rename my cities sometimes when playing. It's like having their names reflect production generated there. But having them lowercase in, say, "top cities list" seems like not a big issue compared to I can't rename them at all now.
     
  8. Nightinggale

    Nightinggale Deity

    Joined:
    Feb 2, 2009
    Messages:
    4,354
    I think you are getting demanding :p

    That makes me wonder about keyboard input. Presumably we should eventually be able to write all characters present in GameFont, both vanilla and those we added. I think a good place to start would be to figure out how to even write Russian on a computer.

    I found this elsewhere on the forum and while it I don't think GameFont itself is that bad, it certainly fits adding new font characters.
     
  9. Daw

    Daw Emperor

    Joined:
    Sep 30, 2013
    Messages:
    1,782
    Oh yes but of course. At least I am not the one who dares to talk about something as bourgeois, showy, posh and luxuriant as keyboard input here :p

    Not sure what you mean. I find it to be as easy as breathing :p

    Go to control panel, find "language and regional settings", select "languages and keyboards" tab, click "change keyboard" button, click "add" button, select Russian from the mindboggingly long list of available choices. Then you are able to switch to Russian in the tray/taskbar.


    The tricky part is (or most definitely should be) that your keybord lacks Cyrillics, so you don't know what key to press to get the symbol you need. So here is QWERTY keyboard layout with Cyrillic part red on it:



    EDIT: It occurred to me that since you are in Germany, it is more than likely that you use QWERTZ rather than QWERTY. So the conversion must be looking like this (or something very similar):
     
  10. Nightinggale

    Nightinggale Deity

    Joined:
    Feb 2, 2009
    Messages:
    4,354
    I managed to put all the Russian chars inside sylfaen.ttf and wrote a converter to map UTF-8 into using those new values. It look like it works, though I'm not qualified to say if the output is proper Russian.

    I have encountered an unexpected problem. Now the game crashes when switching language and the debugger is not really helpful. All it does is tell that it crashes in some python interface in the exe. The game remembers which language you switched to meaning it is sort of like a "needs to restart to gain effect".

    My god the polution in Russia must be bad if it is that hard to breath

    As far as I'm aware nobody from Germany works on M:C. Sure I know German, but then again you know English, that doesn't teleport you to England, does it?
     
  11. Daw

    Daw Emperor

    Joined:
    Sep 30, 2013
    Messages:
    1,782
    I can verify that part if you give me a try.

    Sounds quite sour. Any plan of how it can be dealt with?

    Depends on what you're used to. I often suffer from vertigo due to hyperoxia when abroad. :goodjob:

    No it does not. But knowing Russian also does not nail me down to Russia. I didn't know where you were, so I tried my best to be as helpful as I could by providing an alternative keyboard layout. Hope there's nothing wrong with that.
     
  12. Nightinggale

    Nightinggale Deity

    Joined:
    Feb 2, 2009
    Messages:
    4,354
    It was already pushed to SF when you wrote that. I also updated the precompiled DLL.
    https://sourceforge.net/p/colonizationmodcollection/precompiledDLL/ci/master/tree/
    Font file (which is part of the DLL source module):
    https://sourceforge.net/p/colonizat...L/ci/Russian_translation/tree/Fonts/Cyrillic/

    No. However I will treat it like any other bug, which is to try to figure out what goes wrong. I figure since the actual crash tells us nothing, the approach would likely be to go into the DLL and see if the game goes through the functions we would expect it to go through.

    The only thing wrong with that would be that you aren't ruthless enough to be a successful medieval leader :king:
     
  13. Daw

    Daw Emperor

    Joined:
    Sep 30, 2013
    Messages:
    1,782
    No, it does not look like it is proper Russian. At least on my machine.
    Spoiler :

    The good news is that it does take up colors now
    Spoiler :



    I did not encounter that though restarted several times, and switched languages several times, and switched languages then restarted several times... The game started smoothly every time.

    It does remind me however that a few times - prior to these last changes applied - the game did start with default "full screen" setting because it suddenly forgot my "in window" setting. It did not seem like a big deal of an issue, so I did not pay attention then.

    Now that's good news as anyone who qualifies for being a good leader in medieval times also qualifies for spending the rest of his days in jail nowadays. So it's like I well fit my era :goodjob:
     
  14. Daw

    Daw Emperor

    Joined:
    Sep 30, 2013
    Messages:
    1,782
    The unreadable thing looks to be a once solved issue with the characters IDs are not of the expected 84xx - 85xx range, but rater 10xx - 11xx and even 01xx.
    Spoiler :

    I wonder if the solution that worked then may work again.

    EDIT:
    What makes me surprised is that what must be the "'OK' button" text reports 4 character IDs instead of just 2, which is expected to be sufficient for the "OK".
     
  15. Nightinggale

    Nightinggale Deity

    Joined:
    Feb 2, 2009
    Messages:
    4,354
    It looks like you ended up with the wrong DLL. I removed the experimental one, which is outdated. Also you need to install the font into your system.

    When I used the newest DLL with the font installed, it looks like the attached image.
     

    Attached Files:

  16. Daw

    Daw Emperor

    Joined:
    Sep 30, 2013
    Messages:
    1,782
    The attached image looks magnificent. :goodjob:

    However, with the newest DLL and the font installed in the .../windows/fonts folder - which I started with - I was still having what was my 1st and 2nd screen shots.

    Now did it once again, and it does look differently, although still not as good as yours:
    Spoiler :


    It is still unreadable without an effort, but at least can now be decrypted:
    ... goes for И;
    € goes for П
    , goes for Г;
    long - goes for Ы;
    ⧧ goes for Л;
    ¤ goes for Я;
    [SPACE] goes for Ь;
    etc.

    Overall, it's even easier to read than Standard Galactic for at least some characters (those shared with Latin, right?) look Cyrillic allright. :goodjob:
     
  17. Daw

    Daw Emperor

    Joined:
    Sep 30, 2013
    Messages:
    1,782
    Can it be that I ended up with a wrong version of sylfaen, as here is how it is interpreted by... umm... whatever Windows application does it.

    EDIT:
    :hmm: Odd as it can be, although my old Sylfaen and the new Sylfaen are interpreted differrently by Windows, it seems that there no difference at all as to which one is installed when the game starts, as the Main Menu looks the same with either one. :confused:
     

    Attached Files:

  18. Nightinggale

    Nightinggale Deity

    Joined:
    Feb 2, 2009
    Messages:
    4,354
    That looks.... wrong :eek:

    Presumably the font broke in the transfer, either into git or to your computer. I compressed the correct version of the file and attached it here. Hopefully it will look correctly with that one.

    Oddly enough it only happens with debug builds. That's quite strange, but at least releases will not crash.

    It just occurred to me that the most scenario is that the bug is not in the game, but in the system. When I have debugger, fontforge and such running, it use up a lot of memory and there might not be enough on a 32 bit system. When the game requests more memory and the system says no, then the game will most likely crash and investigating the locations where the crash occurs reveals no bugs at all. If that's the case all I have to do is to goto the 64 bit system and it will work.

    It would still not be good that the game temporally use that much memory, but at least it's not a game breaking bug.

    I have experienced that as well as reverting to default resolution while switching mod. I suspect that some mod specific game options are stored together with screen options, which mean it can corrupt settings like that. I'm not sure if we can do anything about it at all. Luckily it's not a critical issue.
     

    Attached Files:

  19. Daw

    Daw Emperor

    Joined:
    Sep 30, 2013
    Messages:
    1,782
    No, it does not look any better this way. Apparently it gets twisted on my end as soon as it gets here, no matter which particular way. As you probably remember we did have an issue where our computers did not agree on how to read what.

    The only good thing seems to be that it gets twisted identically every time, so there must be some [alien perversive] logic involved, so theoretically it should be possible to trace it and do something to it.

    Another thing that might be done is that I harness fontforge myself and we try to see if it works when done the other way around :dunno:
     
  20. Nightinggale

    Nightinggale Deity

    Joined:
    Feb 2, 2009
    Messages:
    4,354
    After investigating this I learned that this is what happens when the modified font IS NOT USED. Somehow you didn't install the font correctly into your system. Maybe you didn't remove the vanilla font first or something. Either way that seems to be the problem.
     

Share This Page