[We the People] Development "diary"

[Nightinggale]

There was a request for knowing what goes on with development. We don't have a blog, twitter or anything like that, mainly because we do not have anybody to maintain something like that. We are still recruiting more people, which includes people for any task for the mod itself as well as publicity like building a web site.

For the time being I decided to add this thread to give hints on what is going on regarding development. It's in no way complete and there is way more information available in the git log and the issue system on git. This is just highlights of major events.

First entry: translation support​

To put it mildly: the vanilla translation support sucks big time. It allows switching the language used in the game, but that's about it for the good part. It doesn't store which language you picked, but rather the index of that language. When reading the text from xml, it will read the language at that index. The setting is stored in CivilizationIV.ini (my documents) and it's called Language.

One huge flaw in this system is that if you select language 0, you get language 0 in all mods. This has worked out ok so far because the first 5 languages are set by vanilla, hence the same in all mods. However imagine when we add more languages. We could end up with say Russian as language 6 in one mod and 7 in another mod. Also if we load language 7 in a mod with just 5 languages, all text will vanish.

Another flaw is that adding a new string (TXT_KEY) requires adding it in all languages and the person adding new text usually doesn't speak all the languages well enough to do the translation work as well as programming. This has resulted in adding English text in non-English languages just because the game requires something to be added.

With the talk of adding Brazilian Portuguese (and now apparently Russian too), I started working on a complete rewrite of how translations work.

What I have done

Since most of the code is hidden inside the exe file (can't get modded), the languages still need to be stored as a number in CivilizationIV.ini. The problem is not the number itself, but rather the issue that all mods needs to agree on which language to use for each number. I added a table for this and added an additional 20 languages to it (to a total of 25). If all mods agree on using this table, the selected language will be kept even when switching mod.

Getting a list of 25 translations is no good, particularly if there is only 2-3 real translations. I added a new xml file, which tells which languages to add to the language menu. This means it's perfectly fine to use language 0, 2, 17 only and they will show up as 3 languages. This allows mods to agree on which language is used for each number while maintaining the ability to not have the same list ingame. One mod can add a new translation while another mod doesn't. It also allows translations to appear at whatever order the xml modder wants regardless of the numbers for the languages.

If a translation is missing for a certain TXT_KEY or even missing entirely in the mod, English will be used. This avoids the issue of partial or missing translations results in missing text and it also removes the need to add translations for all languages when adding a new TXT_KEY. This makes it much easier to both add new TXT_KEYs and work on or use partial translations.

Translations for a TXT_KEY can appear in any order as they are located by tag name rather than index. This means it's possible to add English, German and nothing else. Vanilla wants the second language to be French, but now French is missing (not translated) and people using the French translation will get the English "translation". Vanilla's reaction to a TXT_KEY with just English and German is to read German as French due to only using indexes, not tag names.

An interesting twist to using tag names is it's possible to use the "language" Tag. This will make the text ingame appear as the TXT_KEYs. Often useless, but if you spot say a typo ingame, you can switch and get which key you should look up in the text xml files.

Anybody interested in the code can read it here: https://github.com/We-the-People-civ4col-mod/Mod/commit/560a80b37e9305408f1dd3d44c265975d29ae54a
It should be noted that it should be possible to copy it to other mods without issues as it doesn't rely on any mod specific code. In fact it doesn't rely on Colonization specific code, meaning it should work in BTS too.

 

[Nightinggale]

Last entry for translation support: UTF-8 support​

There is a character encoding issue with vanilla text xml files. Only ASCII characters are ensured to stay the same because of the mess with character codepages. This means we need to escape characters like:

    <TEXT>
        <Tag>TXT_KEY_UNITS</Tag>
        <English>Units</English>
        <French>Unit&#233;s</French>
        <German>Einheiten</German>
        <Italian>Unit&#224;</Italian>
        <Spanish>Unidades</Spanish>
    </TEXT>

Those &# + some number + ; represent characters. Now the same with the escaped characters decoded:

    <TEXT>
        <Tag>TXT_KEY_UNITS</Tag>
        <English>Units</English>
        <French>Unités</French>
        <German>Einheiten</German>
        <Italian>Unità</Italian>
        <Spanish>Unidades</Spanish>
    </TEXT>

Obviously not having to use html escaped encoded text is way more read friendly. This is where UTF-8 comes into play.

I changed the reading code so it can read UTF-8 and then convert it to the codepage used by the language in question. This means much easier access for translators and we have greatly reduced the risk of mistakes, both due to reading issues and due to translators of multiple languages messing up the file encoding because they aren't using the same locale.

In order to avoid doing the task of converting all the files in one go, I made the conversion activate if an xml file contains "utf8" (not case sensitive). If this isn't in the filename, then the vanilla approach will be used. The game doesn't have any issue with mixed content as in some files are UTF-8 encoded and some aren't.

The actual text ingame is still relying on codepages. No change there. It will use the locale set in windows, which you can change by using this guide. However the game will try to change the locale for the game itself (works on some computers and not others) and often the player needs to use the locale the computer is already set to use. For instance using the French or German translations requires a western European language as locale while Russian requires Russian locale. Often this is the default in windows, meaning it works out of the box. The big issue is if the computer think it's Russian and the player wants to use say German. The guide in the link should be able to fix it, but it's a part of windows, which is annoying and part of why using codepages is not recommended anymore.

The code now supports translation in any of the single byte codepages. Currently I added conversion tables for western European, eastern European and Cyrillic. Technically this is adding support for codepages 1250-1252. It's possible to add support for codepages 1253-1258 too, which adds Hebrew, Greek and whatever else you can find there. I haven't added them because I can always do it later if needed.

I don't think I will code more translation support for the time being. I can't think of anything to add, which isn't a major task in itself, like hack in unicode support (which btw would require access to the Japanese exe file, something I don't have).

The code change. Like the previous one, this isn't mod specific and should work in any mod and in BTS. You can also get the entire change combined.

 

[eXPonent_123]

This means we need to escape characters like:
Those &# + some number + ; represent characters.

We can not avoid such characters, as the game has no normal support for the Russian language.
In ISO-8859-1, characters are replaced with question marks "?".
And in UTF-8 characters are replaced by French counterparts (symbols with accent grave).

We use this table to encode characters:

Interesting, but it's a dead link. It's most likely in your browser's cache or your are logged in or something.

 

[Nightinggale]

We can not avoid such characters, as the game has no normal support for the Russian language.

The game supports Russian if the system locale is set to Russian or any other language using codepage 1251. There is an issue with GameFont, but we can do something about that. Worst case scenario is to make a new GameFont file and make the game pick the right one at startup depending on system locale. There is support for any 8 bit codepage.

And in UTF-8 characters are replaced by French counterparts (symbols with accent grave)

I changed the text loading code to use tags rather than indexes to avoid that problem. Also I added actual text encoding rather than the vanilla approach, which is to assume the value is the same. That approach only works for western European (codepage 1252).

We use this table to encode characters

Interesting, but it's a dead link. It's most likely in your browser's cache or your are logged in or something.

 

[Nightinggale]

Here is an example. I added this to an UTF-8 encoded text file.

    <TEXT>
        <Tag>TXT_KEY_MAIN_MENU_SINGLE_PLAYER</Tag>
        <English>русские</English>
    </TEXT>

Next I hacked the DLL to think English is using codepage 1251, meaning it will use the same character encoding as Russian. In other words I use Russian character encoding ingame despite not having a Russian translation. I then switched my computer's locale to Russian, started the game and I added the result as a screenshot here:


Looks just fine to me.

 

[eXPonent_123]

There is an issue with GameFont, but we can do something about that. Worst case scenario is to make a new GameFont file and make the game pick the right one at startup depending on system locale.

You when the wrote:
I know why the GameFont icons are messed up when switching to Russian. Better yet, I know how to fix it. I will make two new tga files in the near future, which should display Russian correctly without breaking anything else.

 

[Nightinggale]

I added a new concept to the source code: CivEffects.
If you find the name familiar, then it's because I copied the concept from Medieval Conquest. The idea is that it's a new xml file containing entries of CivEffects and it's something a player can own. It then has an effect on the player or something owned by the player, like units. An example could be changing the number of units on the dock in Europe (capped at at least one). For units it could be free promotion to all units, or all units of a certain combat type or whatever. Another example is if a player is able to build a unit or not.

Players do not gain CivEffects directly. Instead they are gained indirectly though owning something, which grants a CivEffect. This could be a trait, civic, founding father, civilization (country) etc. This means if a feature is added to CivEffects, it can be used by either of those and if we add more xml files, then all we have to do is to make whatever we add provide CivEffects too and then it will instantly gain access to provide whatever is already possible with CivEffects. Over time all the xml tags from the individual classes will likely end up in CivEffects, which will both simplify the code and grant way better xml support. Imagine traits granting abilities currently only available to founding fathers.

The system is designed to handle a whole lot of CivEffects. They are never looped (except for init and perhaps later for GUI). Instead everything is a cumulative cache. This not only means the game scales much better to big xml files, it also means the game will be faster than it is right now. I added the unit<->unitclass conversion from civilizationinfo into the cache because while it has nothing to do with the CivEffect concept, the checks needs to be done at the same locations. This means it will speed up checking if the unit in question is banned by the civilizationinfo in question while checking if it's affected by CivEffects at the same time. This too is intended to boost performance.

CivEffects aren't saved, but rather recalculated on load. This means changing the xml files will affect the game instantly when you load something, making the game both much more responsive to xml changes and reduces the risk of bugs due to xml changes. Units go the same treatment for promotions, meaning now if you add a free promotion in xml, existing units will gain it on load.

While the player won't really notice anything from this (perhaps a little from the performance boosting), it should help xml modding quite a bit and it's the foundation for future additions.

 

[Nightinggale]

I moved the colony catchment radius to a map option. This means when you start a new game (new game, custom or multiplayer) you will be asked if you want one or two plot catchment in the game in question. This eliminates the need for the need to use two dll files and WTP 2.8 should be able to be released with a single file, which doesn't need moving.

Despite this being a runtime setup, you still can't switch a game between the two settings once set. It heavily affects how the game is saved and faking loading the wrong one will render savegames unreadable. I will not even attempt to make it an option to switch once a game has started.

This setting affects the savegame format meaning you won't see this until WTP 2.8 where the savegames will be broken anyway. Since we need to break the savegame for certain features, we decided to gather those features in one big release to prevent breaking savegames over and over. One of the changes is a new savegame format, which will be harder to break, but adding it will by itself break savegames. For now, all you need to know is that 2.7.x can read current savegames, 2.8 can't.

 

[Nightinggale]

Development continues and this time the target audience is developers rather than players. I wrote a script to generate enums of xml data, meaning the compiler is gaining awareness of the xml contents (admittedly I copied quite a bit from the script I wrote for the same purpose in Medieval Conquest). This solves the age old problem where some xml files have to match enums in the DLL and if you change the list in one location, you have to change it in the other location. Now if you change xml, it will automatically update the DLL. Not only is this faster, it's also getting rid of one of the less fun tasks and it gives peace of mind that it avoids the DLL vs XML mismatch type of bugs.

The script also makes test code for all the assumed values. Now when the game starts up and xml is not matching the values assumed in the DLL, the game will tell you. I added a new error popup (error, not assert. It's present even when asserts are disabled) telling the player to either revert the change or compile a new DLL because the current setup won't work.

When the compiler is aware of the xml data, then the debugger will also be aware. This means if the debugger says unit type 79 tries to use profession 11, then you are a bit lost and have to do some research to figure out what is behind those numbers. Now it will say "UNIT_NATIVE(79)" tries to use "PROFESSION_HUNTER(11)" and all of a sudden it takes a whole lot less time to get the debugger to reveal what the AI is trying to do. Again it removes a time consuming and annoying task, making it more fun to work with the code and you get more done in the same amount of time.

I also updated the project files to make it easier to pick what to compile. Rather than a bloated list of release, debug etc for 1 plot, 2 plot etc, it's now just assert, debug, profile and release. Next to this setting is the platform setting. This has been stuck at win32 because that's what we have to compile for. However since we use a makefile for the actual compilation, what it says here doesn't matter. The list now contains platforms like "XML hardcoding" and win32 is gone. This makes it much easier to pick what you actually want to compile.

You might wonder why you want to turn xml hardcoding on and off. Simply put, if you can compile yourself, you want to hardcode to help the debugger and performance. If you compile a dll for an xml modder who haven't got a compiler, then you want to disable hardcoding to grant as much xml freedom as possible. For this reason part of the design requirements for this is that both options have to be available. Future releases will have one specific DLL and no movement of dll files will be needed in order to play. It's possible that there will be an extra dll for changing xml files yourself, but time will tell how we do this.

There aren't much news for players, but sometimes development is about making development itself easier. It's not unlike the network desync debug output. Players will never use it, not realize it's there and not pay attention. However the end result is fixed desyncs meaning the fact that players won't notice is the goal because there aren't any desyncs to notice. That's the key for most development tools. They can greatly benefit developers, but players won't notice them and only notice the issues occurring from the lack of said tools.

 

[Nightinggale]

I started work on the new savegame format. The goal is to make it easier for modders and more likely that savegames will survive xml/code changes.

The primary change is that there is no longer an assumed order in the savegame. When reading, an ID is read and then the game knows what is saved next based on the ID. Writing code is somewhat unchanged from a coding point of view, except it adds the ID.

This approach has several benefits. Since order no longer matter, read and write can no longer go out of sync (this has happened multiple times and it breaks savegames each time). Also anything not mentioned will automatically get the default value, meaning loading a savegame from before a certain variable was added will not be a problem. Removing data from the savegame is not a problem either. Just stop saving it and modify the read code to read whatever variable it is (usually an int) and throw the read data away.

We need people to type in all the variables for the savegames​

It's not as complex as it sounds. They layout can be seen here. If you are interested in doing this, message me. It's a nice way to get started modding as the task for the most parts isn't complex and you will have to look around in the source code and get an idea of what kind of data is available.

Other benefits of the new format:
(not implemented yet, but the code is made ready to add this later)

The new format has been made ready to handle changing xml indexes. For instance if unit of type 8 is saved and on load it's type 9 because somebody changed xml, then the idea is that on load a conversion table can tell that 8 is now 9. The code is prepared with 3 locations where the conversion should take place by adding a single line of code, but it requires building the conversion table. This task isn't as tricky as it sounds because I already did it in Medieval Conquest where it's working.

Another benefit is the savegame size. Since the game will use default values for variables not mentioned in the savegame, any variable at the default value can be skipped. When you examine the code, a lot of the variables are never used. Say for instance unit experience. The vast majority of units will never have any number other than 0.

Another size benefit is that now the savegame will be stored in a byte array in DLL memory and read/written in the exe in one go. This allows compression. While the game currently saves uncompressed (vanilla doesn't compress and to my knowledge no mod ever added compression), having the savegames as byte arrays allows adding something like 7z to greatly reduce the size.

Why does size matter? It's actually not to save a lot of HD space (most of us have plenty). It's bandwidth. When a player joins a multiplayer game, the game freezes for everybody, the host saves and then transmit the savegame to the player. If the file is smaller, hopefully the freezing time will be shorter.

 

[Nightinggale]

Work continues, though not much happened since last time in the eyes of the player. Yields can now be set in xml as opposed to the dll+xml setup in vanilla. This allows more yield changes without recompiling the dll, which is a big help to xml modders.

Work on the new savegame format is ongoing. It's going well and the code becomes a whole lot cleaner and less likely to break. It also has build in xml conversion on load, meaning it makes a conversion table to update the savegame to the current xml files. For instance if somebody adds a new unit in the middle of the list, then unit 79 (native braves) will become 80. On load if it's 79, the game will know it's supposed to read 80, store 80 in memory and the savegame should still work. This part seems to work fine, but it has had very limited testing, mainly because it doesn't really work until all the savegame code has been converted.

The savegame conversion is a nice first task in the project. It often doesn't require great programming skills and you get to see the code and get familiar with it. If you want to help out, do tell.

 

[raystuttgart]

Hi guys,

we want to assure community that actual work on WTP is progressing steadily.
We are not just discussing new ideas in the forum.

A lot of work has been invested since last Beta-Release (which was less than 3 weeks ago).

• Many small improvements, fixes and corrections
• AI improvemements
  
• Overhauls of technical base systems like e.g. "New Save Game Format", "Translation System and Language Support", ...
  
• Perparing technical concepts considering new features and overhauls for the planned Feature Release 2.8 (which will need some time though)
  
• Contacting other former modders for Civ4Col that might consider to join us or at least be willing to support a bit from time to time
• Dicussions on how we could better involve community and let you participate more
• Internal ogranisation optimization and a bit of cleaning up
  
• ...

We will try to publish a "Public Release" soon again that contains all the improvements we already implemented and tested.
We still however want to finish some of the assinged issues and we will also need to do more internal testing of course.

We - the team - really believe that WTP still has a lot of potential for the future and that it is still fun to play.
We just hope that you - the community - do as well and stay with us. We need you!

Let us have a lot of fun together modding, discussing and playing WTP.

 

[raystuttgart]

Figuring out plans and necessary team support for Release 2.8:

Spoiler :

All team members will of course choose their own priorities but we need to inform each other and agree.
And of course we will continue to work on issues and improvements. (But there is no real planning needed for that.)

Thus I give it a start and will write down my personal work plan for Release 2.8.
(We do not have to set a Release Date yet - we will see anyways how fast or slow we will progress.)

@community:
This is just meant to clarify a "short term" plan internally in the team.
And of course more concepts might be implemented in later releases.

Please do not start discussions on the specific features in this thread but instead use the feature threads.
This is currently just informing community about plans and clarifying in team if those plans are ok with everybody.

Ray's Plan for Release 2.8:
--------------------------------------

By reading the feedback in the threads and checking the polls, the following concepts are pretty much my top todo list for Release 2.8.

All of these seem to have a very high rate of agreement and I personally like them enough to invest my effort.
I also purposely tried to first pick smaller changes that should not have too much impact on gameplay.

Already Done:

Spoiler :

• Return of the Town Guard (DONE)
• Rebalancing Lumber Mill (DONE)
• Unique Units for Russian Expert Priest and Expert Missionary (DONE)
• Restored "European Line Infantery" to normal Expert that can change Professions (DONE)
• Merging Treasures (DONE)
• Big Refactoring of Events (including reactivation of deactivated Events (DONE)
• Making Slavery more interesting (DONE)
• Trading Posts in Native Villages (DONE)
• Additional Citizen Help Information - LbD and Modifiers (DONE)
• Only one Colonist at a time living in a Native Village (DONE)
• Russian Founding Fathers (DONE)
• Happiness vs. Unhappiness (DONE)

Still Todo:

• nothing considering features (for this Release - otherwise it will get too big and risky)

It would however be nice if we could also finish and merge these improvement branches:

• scrollable Score List
• BTS style city Icons and effects
  
• K-mod pathfinder improvements
• New Savegame Format

--------------------------------------

All the rest of the suggestions and concepts will probably not become part of 2.8 because they are either:

• too big for Release 2.8 considering effort (on top of the list for 2.8)
  
• not yet clarified or agreed enough
• dependent on another implementation first

So probably for 2.9, 2.10, and following:

• Upkeep for Military Units
• Water Improvement Fishing Nets
  
• Improving Feature Health
• Founding New Cities Mechanics --> Will need help with AI
• Changing Cannons From Units to Professions --> Will need help with AI and graphics (Buttons in Atlasses)
  
• Captured City Art
• Pirate Nation and Pirate Raids

• Nation specific Unique Units and Unique Buildings
• Religions of the New World
• Large Rivers
• Techs?? (no game concept yet)
• Civics?? (no game concept yet)
• Rebuild of Military System (no game concept yet)

But maybe other team members are interested or they might be contained in a later release.
Let me know if there is then support needed somewhere.
---------

@team:
Please let me know if there are any concerns or disagreements considering my personal todo list for Release 2.8.
(We can also discuss internally in our chat. I will then adjust this post wherever necessary.)

It would be great if the rest of the team could inform as well, what they are currently planning to work on for Release 2.8.
Please also mention implementations where you are willing to give support or will need support yourself from other team members.

 

[Schmiddie]

Please let me know if and when you need any graphic support and how we can handle this. I'm currently not familiar with the current modding tools and the home of the mod and don't know how to provide any new graphic.

 

[raystuttgart]

Please let me know if and when you need any graphic support and how we can handle this.

Awesome.
Let us just discuss it in the team Slack-Chat.
It is a bit easier and will not confuse the rest of community.

I'm currently not familiar with the current modding tools and the home of the mod and don't know how to provide any new graphic.

We can and will explain everything to you, don't worry.

 

[raystuttgart]

Update on the current status.

We have made pretty nice progress since the last published beta.

• We have fixed many small issues and implemented a few more improvements (UI, texts, balancing, AI, exploits, performance, ...)
• We also did some more testing and code review. Thus we also found and eliminated some yet undiscovered small issues from last releases.
• Additionally we improved some core systems really deep in the code and built new modding tools to give us more possibilities for the future, increase our efficiency and let us find issues faster. (e.g. Savegame System, Python Event System, Translation System, Language and Gamefont Support, ...)
• We have prepared concepts and prototypes we will need for the next Major Feature Release.
• We also added a few more Python Events and Quests.

-----
Achievements

We have now achieved to have the most stable and clean release we ever had. (Fixed bugs, improved balance, corrected texts, ...)
We also eliminated exploits and improved AI. (But we are a bit careful with this and will not release all of it at once before we did more testing.)
It also contains several improved core systems that will make it easier and more efficient for us modders to build on and publish further releases.
We now also have new modding "tools" that will make it easier for us modders to discover issues (e.g. missing TXT_KEYs).

Well, we also created a few things you will notice considering gameplay (e.g. new Quests and Events). But it was not our focus.
We wanted to clean up and have a solid base to start the next big phase for expanding the mod with features.
We also prepared the next Feature Release of course by creating feature concepts, technical concepts and a general implementation roadmap.

-----
Release Strategy

The current "Improvement Release" will most likely be published latest this weekend.
It might not have much new gameplay but it is a really important step on our roadmap.
It will also still be savegame compatible. Which we wanted to have as well. Thus no new features yet.
(All new Units, Buildings, Improvements, Bonus Ressources, Yields, ... would have broken existing savegames.)

Then we will start to work on our next "Major Feature Release" which will most likely take a couple of months.
It will of course have much more new gameplay but also necessary fixes and improvements for issues reported.
It will most likely not be savegame compatible to previous releases (only to later following releases) since we will change the Savegame System.
(The change of the new Savegame System will allow us to add new features in the future without ever breaking savegame compatibility again.)

-----
Summary:

Stay tuned. Our next "Improvement Release" is on the way.

We did invest a lot of effort to make modding for us modders easier and more efficient.
This was not really visible to you as players but it was necessary to speed up things and improve quality for future releases.

We also needed to do a "clean up and consolidation" again before taking the next big steps considering gameplay.
This you as players will also notice in quality, performance and stability.

But now since that has happened we can now focus more efforts again to improve gameplay by more features and game mechanic overhauls.
We fully understand that you are looking forward to that as well and promise that you will get more nice new features in a couple of months.

 

[Mr. ZorG]

and after that I planned to start translating Colonization again)
it will be necessary to somehow cooperate so that the text already translated into Russian remains in the new releases of the mod.

 

[Swarbs]

Release Strategy

The current "Improvement Release" will most likely be published latest this weekend.

Has this been released yet? All I can see from the 2.7.2 beta from December

 

[raystuttgart]

Has this been released yet?

No.

There is still one issue that needs to be fixed. (Broken special characters - it only occurs on some installations though.)
But Nightinggale, the only team member who can fix it, is currently sick. (Corona)

But do not worry:
Internal work still continues. (e.g. testing, bugfixing, further small improvements, ...)
In the meantime we also already work and prepare features for the next release. (e.g. concept work, collecting graphics, prototyping, ...)
So at least the upcoming release will be even more improved and the release after the current one will be finished a bit faster.

 

[raystuttgart]

Hi guys,

I am going to start my "small feature branch" this weekend - either this evening or tomorrow.
(Unless I get confirmation that we can publish "current improvement release" this weekend - which I originally wanted to build on in its published version.)

We have simply already collected so many graphics and code prototypes that I do not want to be "idle" any longer just fixing and improving small stuff.
(I am really getting impatient and I am eager to finally implement some features again.)

Generally it should be no problem to later merge whatever will still be implemented in the "current improvement release" with "small feature branch".
But of course we should still try to publish the "current improvement release" at some reasonable point of time soon.
(Thus it is probably be best to keep the 2 separated for now so "small feature branch" will not delay "current improvement release".)