1. We have added a Gift Upgrades feature that allows you to gift an account upgrade to another member, just in time for the holiday season. You can see the gift option when going to the Account Upgrades screen, or on any user profile screen.
    Dismiss Notice

[We the People] Development "diary"

Discussion in 'Civ4Col - We The People' started by Nightinggale, Nov 23, 2018.

  1. Nightinggale

    Nightinggale Chieftain Supporter

    Joined:
    Feb 2, 2009
    Messages:
    4,072
    There was a request for knowing what goes on with development. We don't have a blog, twitter or anything like that, mainly because we do not have anybody to maintain something like that. We are still recruiting more people, which includes people for any task for the mod itself as well as publicity like building a web site.

    For the time being I decided to add this thread to give hints on what is going on regarding development. It's in no way complete and there is way more information available in the git log and the issue system on git. This is just highlights of major events.

    First entry: translation support

    To put it mildly: the vanilla translation support sucks big time. It allows switching the language used in the game, but that's about it for the good part. It doesn't store which language you picked, but rather the index of that language. When reading the text from xml, it will read the language at that index. The setting is stored in CivilizationIV.ini (my documents) and it's called Language.

    One huge flaw in this system is that if you select language 0, you get language 0 in all mods. This has worked out ok so far because the first 5 languages are set by vanilla, hence the same in all mods. However imagine when we add more languages. We could end up with say Russian as language 6 in one mod and 7 in another mod. Also if we load language 7 in a mod with just 5 languages, all text will vanish.

    Another flaw is that adding a new string (TXT_KEY) requires adding it in all languages and the person adding new text usually doesn't speak all the languages well enough to do the translation work as well as programming. This has resulted in adding English text in non-English languages just because the game requires something to be added.

    With the talk of adding Brazilian Portuguese (and now apparently Russian too), I started working on a complete rewrite of how translations work.

    What I have done

    Since most of the code is hidden inside the exe file (can't get modded), the languages still need to be stored as a number in CivilizationIV.ini. The problem is not the number itself, but rather the issue that all mods needs to agree on which language to use for each number. I added a table for this and added an additional 20 languages to it (to a total of 25). If all mods agree on using this table, the selected language will be kept even when switching mod.

    Getting a list of 25 translations is no good, particularly if there is only 2-3 real translations. I added a new xml file, which tells which languages to add to the language menu. This means it's perfectly fine to use language 0, 2, 17 only and they will show up as 3 languages. This allows mods to agree on which language is used for each number while maintaining the ability to not have the same list ingame. One mod can add a new translation while another mod doesn't. It also allows translations to appear at whatever order the xml modder wants regardless of the numbers for the languages.

    If a translation is missing for a certain TXT_KEY or even missing entirely in the mod, English will be used. This avoids the issue of partial or missing translations results in missing text and it also removes the need to add translations for all languages when adding a new TXT_KEY. This makes it much easier to both add new TXT_KEYs and work on or use partial translations.

    Translations for a TXT_KEY can appear in any order as they are located by tag name rather than index. This means it's possible to add English, German and nothing else. Vanilla wants the second language to be French, but now French is missing (not translated) and people using the French translation will get the English "translation". Vanilla's reaction to a TXT_KEY with just English and German is to read German as French due to only using indexes, not tag names.

    An interesting twist to using tag names is it's possible to use the "language" Tag. This will make the text ingame appear as the TXT_KEYs. Often useless, but if you spot say a typo ingame, you can switch and get which key you should look up in the text xml files.

    Anybody interested in the code can read it here: https://github.com/We-the-People-civ4col-mod/Mod/commit/560a80b37e9305408f1dd3d44c265975d29ae54a
    It should be noted that it should be possible to copy it to other mods without issues as it doesn't rely on any mod specific code. In fact it doesn't rely on Colonization specific code, meaning it should work in BTS too.
     
  2. Nightinggale

    Nightinggale Chieftain Supporter

    Joined:
    Feb 2, 2009
    Messages:
    4,072
    Last entry for translation support: UTF-8 support

    There is a character encoding issue with vanilla text xml files. Only ASCII characters are ensured to stay the same because of the mess with character codepages. This means we need to escape characters like:
    Code:
        <TEXT>
            <Tag>TXT_KEY_UNITS</Tag>
            <English>Units</English>
            <French>Unit&#233;s</French>
            <German>Einheiten</German>
            <Italian>Unit&#224;</Italian>
            <Spanish>Unidades</Spanish>
        </TEXT>
    Those &# + some number + ; represent characters. Now the same with the escaped characters decoded:
    Code:
        <TEXT>
            <Tag>TXT_KEY_UNITS</Tag>
            <English>Units</English>
            <French>Unités</French>
            <German>Einheiten</German>
            <Italian>Unità</Italian>
            <Spanish>Unidades</Spanish>
        </TEXT>
    Obviously not having to use html escaped encoded text is way more read friendly. This is where UTF-8 comes into play.

    I changed the reading code so it can read UTF-8 and then convert it to the codepage used by the language in question. This means much easier access for translators and we have greatly reduced the risk of mistakes, both due to reading issues and due to translators of multiple languages messing up the file encoding because they aren't using the same locale.

    In order to avoid doing the task of converting all the files in one go, I made the conversion activate if an xml file contains "utf8" (not case sensitive). If this isn't in the filename, then the vanilla approach will be used. The game doesn't have any issue with mixed content as in some files are UTF-8 encoded and some aren't.

    The actual text ingame is still relying on codepages. No change there. It will use the locale set in windows, which you can change by using this guide. However the game will try to change the locale for the game itself (works on some computers and not others) and often the player needs to use the locale the computer is already set to use. For instance using the French or German translations requires a western European language as locale while Russian requires Russian locale. Often this is the default in windows, meaning it works out of the box. The big issue is if the computer think it's Russian and the player wants to use say German. The guide in the link should be able to fix it, but it's a part of windows, which is annoying and part of why using codepages is not recommended anymore.

    The code now supports translation in any of the single byte codepages. Currently I added conversion tables for western European, eastern European and Cyrillic. Technically this is adding support for codepages 1250-1252. It's possible to add support for codepages 1253-1258 too, which adds Hebrew, Greek and whatever else you can find there. I haven't added them because I can always do it later if needed.

    I don't think I will code more translation support for the time being. I can't think of anything to add, which isn't a major task in itself, like hack in unicode support (which btw would require access to the Japanese exe file, something I don't have).

    The code change. Like the previous one, this isn't mod specific and should work in any mod and in BTS. You can also get the entire change combined.
     
    Last edited: Nov 23, 2018
    devolution and Thomazml like this.
  3. eXPonent_123

    eXPonent_123 Chieftain

    Joined:
    Mar 25, 2017
    Messages:
    105
    Gender:
    Male
    We can not avoid such characters, as the game has no normal support for the Russian language.
    In ISO-8859-1, characters are replaced with question marks "?".
    And in UTF-8 characters are replaced by French counterparts (symbols with accent grave).

    We use this table to encode characters:
     
    Last edited: Nov 24, 2018
  4. Nightinggale

    Nightinggale Chieftain Supporter

    Joined:
    Feb 2, 2009
    Messages:
    4,072
    The game supports Russian if the system locale is set to Russian or any other language using codepage 1251. There is an issue with GameFont, but we can do something about that. Worst case scenario is to make a new GameFont file and make the game pick the right one at startup depending on system locale. There is support for any 8 bit codepage.

    I changed the text loading code to use tags rather than indexes to avoid that problem. Also I added actual text encoding rather than the vanilla approach, which is to assume the value is the same. That approach only works for western European (codepage 1252).

    Interesting, but it's a dead link. It's most likely in your browser's cache or your are logged in or something.
     
  5. Nightinggale

    Nightinggale Chieftain Supporter

    Joined:
    Feb 2, 2009
    Messages:
    4,072
    Here is an example. I added this to an UTF-8 encoded text file.
    Code:
        <TEXT>
            <Tag>TXT_KEY_MAIN_MENU_SINGLE_PLAYER</Tag>
            <English>русские</English>
        </TEXT>
    Next I hacked the DLL to think English is using codepage 1251, meaning it will use the same character encoding as Russian. In other words I use Russian character encoding ingame despite not having a Russian translation. I then switched my computer's locale to Russian, started the game and I added the result as a screenshot here:
    Russian screenshot.jpg

    Looks just fine to me.
     
  6. eXPonent_123

    eXPonent_123 Chieftain

    Joined:
    Mar 25, 2017
    Messages:
    105
    Gender:
    Male
    Last edited: Nov 24, 2018
  7. Nightinggale

    Nightinggale Chieftain Supporter

    Joined:
    Feb 2, 2009
    Messages:
    4,072
    I added a new concept to the source code: CivEffects.
    If you find the name familiar, then it's because I copied the concept from Medieval Conquest. The idea is that it's a new xml file containing entries of CivEffects and it's something a player can own. It then has an effect on the player or something owned by the player, like units. An example could be changing the number of units on the dock in Europe (capped at at least one). For units it could be free promotion to all units, or all units of a certain combat type or whatever. Another example is if a player is able to build a unit or not.

    Players do not gain CivEffects directly. Instead they are gained indirectly though owning something, which grants a CivEffect. This could be a trait, civic, founding father, civilization (country) etc. This means if a feature is added to CivEffects, it can be used by either of those and if we add more xml files, then all we have to do is to make whatever we add provide CivEffects too and then it will instantly gain access to provide whatever is already possible with CivEffects. Over time all the xml tags from the individual classes will likely end up in CivEffects, which will both simplify the code and grant way better xml support. Imagine traits granting abilities currently only available to founding fathers.

    The system is designed to handle a whole lot of CivEffects. They are never looped (except for init and perhaps later for GUI). Instead everything is a cumulative cache. This not only means the game scales much better to big xml files, it also means the game will be faster than it is right now. I added the unit<->unitclass conversion from civilizationinfo into the cache because while it has nothing to do with the CivEffect concept, the checks needs to be done at the same locations. This means it will speed up checking if the unit in question is banned by the civilizationinfo in question while checking if it's affected by CivEffects at the same time. This too is intended to boost performance.

    CivEffects aren't saved, but rather recalculated on load. This means changing the xml files will affect the game instantly when you load something, making the game both much more responsive to xml changes and reduces the risk of bugs due to xml changes. Units go the same treatment for promotions, meaning now if you add a free promotion in xml, existing units will gain it on load.

    While the player won't really notice anything from this (perhaps a little from the performance boosting), it should help xml modding quite a bit and it's the foundation for future additions.
     
    Drunk-Monk, TaylorItaly and Hecur like this.
  8. Nightinggale

    Nightinggale Chieftain Supporter

    Joined:
    Feb 2, 2009
    Messages:
    4,072
    I moved the colony catchment radius to a map option. This means when you start a new game (new game, custom or multiplayer) you will be asked if you want one or two plot catchment in the game in question. This eliminates the need for the need to use two dll files and WTP 2.8 should be able to be released with a single file, which doesn't need moving.

    Despite this being a runtime setup, you still can't switch a game between the two settings once set. It heavily affects how the game is saved and faking loading the wrong one will render savegames unreadable. I will not even attempt to make it an option to switch once a game has started.

    This setting affects the savegame format meaning you won't see this until WTP 2.8 where the savegames will be broken anyway. Since we need to break the savegame for certain features, we decided to gather those features in one big release to prevent breaking savegames over and over. One of the changes is a new savegame format, which will be harder to break, but adding it will by itself break savegames. For now, all you need to know is that 2.7.x can read current savegames, 2.8 can't.
     
  9. Nightinggale

    Nightinggale Chieftain Supporter

    Joined:
    Feb 2, 2009
    Messages:
    4,072
    Development continues and this time the target audience is developers rather than players. I wrote a script to generate enums of xml data, meaning the compiler is gaining awareness of the xml contents (admittedly I copied quite a bit from the script I wrote for the same purpose in Medieval Conquest). This solves the age old problem where some xml files have to match enums in the DLL and if you change the list in one location, you have to change it in the other location. Now if you change xml, it will automatically update the DLL. Not only is this faster, it's also getting rid of one of the less fun tasks and it gives peace of mind that it avoids the DLL vs XML mismatch type of bugs.

    The script also makes test code for all the assumed values. Now when the game starts up and xml is not matching the values assumed in the DLL, the game will tell you. I added a new error popup (error, not assert. It's present even when asserts are disabled) telling the player to either revert the change or compile a new DLL because the current setup won't work.

    When the compiler is aware of the xml data, then the debugger will also be aware. This means if the debugger says unit type 79 tries to use profession 11, then you are a bit lost and have to do some research to figure out what is behind those numbers. Now it will say "UNIT_NATIVE(79)" tries to use "PROFESSION_HUNTER(11)" and all of a sudden it takes a whole lot less time to get the debugger to reveal what the AI is trying to do. Again it removes a time consuming and annoying task, making it more fun to work with the code and you get more done in the same amount of time.

    I also updated the project files to make it easier to pick what to compile. Rather than a bloated list of release, debug etc for 1 plot, 2 plot etc, it's now just assert, debug, profile and release. Next to this setting is the platform setting. This has been stuck at win32 because that's what we have to compile for. However since we use a makefile for the actual compilation, what it says here doesn't matter. The list now contains platforms like "XML hardcoding" and win32 is gone. This makes it much easier to pick what you actually want to compile.

    You might wonder why you want to turn xml hardcoding on and off. Simply put, if you can compile yourself, you want to hardcode to help the debugger and performance. If you compile a dll for an xml modder who haven't got a compiler, then you want to disable hardcoding to grant as much xml freedom as possible. For this reason part of the design requirements for this is that both options have to be available. Future releases will have one specific DLL and no movement of dll files will be needed in order to play. It's possible that there will be an extra dll for changing xml files yourself, but time will tell how we do this.

    There aren't much news for players, but sometimes development is about making development itself easier. It's not unlike the network desync debug output. Players will never use it, not realize it's there and not pay attention. However the end result is fixed desyncs meaning the fact that players won't notice is the goal because there aren't any desyncs to notice. That's the key for most development tools. They can greatly benefit developers, but players won't notice them and only notice the issues occurring from the lack of said tools.
     
    Hecur likes this.
  10. Nightinggale

    Nightinggale Chieftain Supporter

    Joined:
    Feb 2, 2009
    Messages:
    4,072
    I started work on the new savegame format. The goal is to make it easier for modders and more likely that savegames will survive xml/code changes.

    The primary change is that there is no longer an assumed order in the savegame. When reading, an ID is read and then the game knows what is saved next based on the ID. Writing code is somewhat unchanged from a coding point of view, except it adds the ID.

    This approach has several benefits. Since order no longer matter, read and write can no longer go out of sync (this has happened multiple times and it breaks savegames each time). Also anything not mentioned will automatically get the default value, meaning loading a savegame from before a certain variable was added will not be a problem. Removing data from the savegame is not a problem either. Just stop saving it and modify the read code to read whatever variable it is (usually an int) and throw the read data away.

    We need people to type in all the variables for the savegames
    It's not as complex as it sounds. They layout can be seen here. If you are interested in doing this, message me. It's a nice way to get started modding as the task for the most parts isn't complex and you will have to look around in the source code and get an idea of what kind of data is available.

    Other benefits of the new format:
    (not implemented yet, but the code is made ready to add this later)

    The new format has been made ready to handle changing xml indexes. For instance if unit of type 8 is saved and on load it's type 9 because somebody changed xml, then the idea is that on load a conversion table can tell that 8 is now 9. The code is prepared with 3 locations where the conversion should take place by adding a single line of code, but it requires building the conversion table. This task isn't as tricky as it sounds because I already did it in Medieval Conquest where it's working.

    Another benefit is the savegame size. Since the game will use default values for variables not mentioned in the savegame, any variable at the default value can be skipped. When you examine the code, a lot of the variables are never used. Say for instance unit experience. The vast majority of units will never have any number other than 0.

    Another size benefit is that now the savegame will be stored in a byte array in DLL memory and read/written in the exe in one go. This allows compression. While the game currently saves uncompressed (vanilla doesn't compress and to my knowledge no mod ever added compression), having the savegames as byte arrays allows adding something like 7z to greatly reduce the size.

    Why does size matter? It's actually not to save a lot of HD space (most of us have plenty). It's bandwidth. When a player joins a multiplayer game, the game freezes for everybody, the host saves and then transmit the savegame to the player. If the file is smaller, hopefully the freezing time will be shorter.
     
  11. Nightinggale

    Nightinggale Chieftain Supporter

    Joined:
    Feb 2, 2009
    Messages:
    4,072
    Work continues, though not much happened since last time in the eyes of the player. Yields can now be set in xml as opposed to the dll+xml setup in vanilla. This allows more yield changes without recompiling the dll, which is a big help to xml modders.

    Work on the new savegame format is ongoing. It's going well and the code becomes a whole lot cleaner and less likely to break. It also has build in xml conversion on load, meaning it makes a conversion table to update the savegame to the current xml files. For instance if somebody adds a new unit in the middle of the list, then unit 79 (native braves) will become 80. On load if it's 79, the game will know it's supposed to read 80, store 80 in memory and the savegame should still work. This part seems to work fine, but it has had very limited testing, mainly because it doesn't really work until all the savegame code has been converted.

    The savegame conversion is a nice first task in the project. It often doesn't require great programming skills and you get to see the code and get familiar with it. If you want to help out, do tell.
     
    JosEPh_II likes this.

Share This Page