Mini-engine progress

[snowern]

That would still pose major potential compatibility challenges to lots of mods with Python components - from what I understand, in the current implementation, they should be compatible with the new engine out of the box.

Oh yes, it's potentially feasible with an engine that won't support most mods anyway, but not so useful for a more fully featured engine.

 

[Jerrybi]

we could have two separate branches
one with "full" compatibility and one with "new features" such like python3

 

[snowern]

we could have two separate branches
one with "full" compatibility and one with "new features" such like python3

Yes, branches. But the idea of using python 3 is to not use python 2 any more.

I've also got a C++ modules version of Cv4MiniEngine sitting around. That could be a separate branch, but it's not like I'd want to maintain both branches. If modules worked well, there would be only modules.

*Unless... a python 3 branch would have the performance to warrant its existence.

 

[Jerrybi]

this thread

Thread 'Civ4 - Savegame Library and Editor (c4lib)'

Sep 24, 2024

Hello everyone.

Brief Project Overview
This thread is for a Civ4 savegame library and accompaning editor which I've been working on for the past several months. I'll use the acronym Civ4SL from now on to refer to the library.

The library will provide a C-style programming API which will enable the programmatic editing of savegame files, including the zlib-compressed data. The library implements the Civ4 checksum function. A Boost Property Tree-style interface is planned which will facilitate use of the API.

Included in the project is an editor, Civ4Edit.exe. The editor will be...

• hankinsohl
• Replies: 19
• Forum: Civ4 - Project & Mod Development

• 

does not help with info on save file?

 

[snowern]

this thread

Thread 'Civ4 - Savegame Library and Editor (c4lib)'

Sep 24, 2024

Hello everyone.

Brief Project Overview
This thread is for a Civ4 savegame library and accompaning editor which I've been working on for the past several months. I'll use the acronym Civ4SL from now on to refer to the library.

The library will provide a C-style programming API which will enable the programmatic editing of savegame files, including the zlib-compressed data. The library implements the Civ4 checksum function. A Boost Property Tree-style interface is planned which will facilitate use of the API.

Included in the project is an editor, Civ4Edit.exe. The editor will be...

• hankinsohl
• Replies: 19
• Forum: Civ4 - Project & Mod Development

• 

does not help with info on save file?

Well, that's interesting.

 

[snowern]

Oh, it's great. I can be defeated in my unconventional games so much faster.

 

[black_imperator]

as someone with a mod with a non negligible amount of both python and dll work, i'm more than willing to go through whatever hoops i'll be required to to get to those sweet performance boosts and non-hardlocked engine.

So, if python3 gets us something relevant but i need to spend a couple of weeks hunting for division signs in python to get it, i'll do it happily (and help others if needed)

 

[snowern]

Yes, if python 3's free-threading would have performance benefits, then it may be worthwhile. Or if Python 3 is simply faster by itself. There's not much python to thread though. Plot reveal is the one thing I've seen, there aren't many other python callbacks that take up too much time.

And to be extra safe, it may be a good idea to enforce a file extension to recognise python 3/free-threading compatible scripts. *Or have the VFS get scripts from a "Python3" folder.

Another thing is the VFS. Need to improve startup performance of that. Could either rename directories to force casing, or do case-insensitive physical file ops.

Might want to rip out all the string stuff too and replace it with UTF-8 everywhere, but that would change the regular build, and I want the regular build to have only minimal changes, but, one would hope changing strings won't change game hashes.

 

[Nightinggale]

Yes, if python 3's free-threading would have performance benefits, then it may be worthwhile. Or if Python 3 is simply faster by itself. There's not much python to thread though.

When using vanilla tools, C++ is much faster than python when writing precisely the same code in both. For this reason I would expect all code with performance impact to be in C++. While modern python might reduce this advantage, python will never beat something compiled with a bunch of optimization flags enabled.
Conclusion: faster python is good, but it can't become good enough to not recommend putting performance intensive code in C++.

Might want to rip out all the string stuff too and replace it with UTF-8 everywhere, but that would change the regular build, and I want the regular build to have only minimal changes, but, one would hope changing strings won't change game hashes.

WTP stores the text xml files in UTF-8. It's much easier for translators that way because it removes the need for character escapes and it allows using more than one codepage, through it's still max one per language. There is a plugin for Notepad++, which converts the character escapes, which makes the process of converting the files easier. UTF-8 is highly recommended.

If we worry about EXE compatibility in that regard, I suppose it wouldn't be that hard to rip out my custom made UTF-8 to Codepage 1252 converter and make it sort of a standard addon to any mod, which then assumes all the xml files to be UTF-8 encoded.

 

[Walter Hawkwood]

WTP stores the text xml files in UTF-8. It's much easier for translators that way because it removes the need for character escapes and it allows using more than one codepage, through it's still max one per language. There is a plugin for Notepad++, which converts the character escapes, which makes the process of converting the files easier. UTF-8 is highly recommended.

Is it a standalone plugin or something that comes bundled? I am a Notepad++ user, but IIRC there's no easy UTF-8 conversion functionality in it natively.

 

[snowern]

When using vanilla tools, C++ is much faster than python when writing precisely the same code in both. For this reason I would expect all code with performance impact to be in C++. While modern python might reduce this advantage, python will never beat something compiled with a bunch of optimization flags enabled.

What I've seen is that when the AI trades maps on gigantic maps, a lot of time is spent doing the plot reveal event, which BUG mod handles. This is what could be threaded.

WTP stores the text xml files in UTF-8

Okay, so it looks like I would have to check encoding: <?xml version="1.0" encoding="UTF-8" standalone="no"?>

UTF-8 everywhere might not actually be everywhere though. The TUI loves the property that string.size == width. Ripping out Cv[W]String is half way there though. Replace with maybe u8string, u16string, and string for ASCII. Keep text in UTF-16 to avoid needing to change TUI stuff.

 

[a pen-dragon]

What I've seen is that when the AI trades maps on gigantic maps, a lot of time is spent doing the plot reveal event

Not only the AI, as when I trade a map on such settings, the map takes a noticeable amount of time to update, probably due to this (it takes longer the more tiles are new).

This can take up to a few seconds (in RFC DoC), and it sometimes triggers OOM crashes, meaning it probably takes quite a bit of RAM as well.

Btw. is there a good reason why this is in python and not the dll?

 

[Nightinggale]

Btw. is there a good reason why this is in python and not the dll?

Whenever somebody asks that, the only answer I have ever been able to get is to get mac support because the mac version doesn't support dll modding. The slowness is likely due to the python-dll interface being dead slow and possibly memory hungry. Looping through all plots in the DLL to update visibility and similar is not memory intensive and practically instant. In fact I do believe vanilla will do this automatically when trading maps, so I don't know what BUG is doing in python.

Is it a standalone plugin or something that comes bundled? I am a Notepad++ user, but IIRC there's no easy UTF-8 conversion functionality in it natively.

Sadly it seems it's no longer installed in my notepad++ and I can't find it again. Maybe it was removed from the list of approved plugins for reasons I don't know about. That sucks.

 

[snowern]

Not only the AI, as when I trade a map on such settings, the map takes a noticeable amount of time to update, probably due to this (it takes longer the more tiles are new).

This can take up to a few seconds (in RFC DoC), and it sometimes triggers OOM crashes, meaning it probably takes quite a bit of RAM as well.

Btw. is there a good reason why this is in python and not the dll?

It's in python because that's what BUG mod is written in. I don't think vanilla BTS does anything with the event, but the DLL will still invoke it.

 

[snowern]

What is the encoding of the INI file? If I stick non-ascii in my name, it appears to be saved as Windows-1252. But it could vary depending on OS language. I think I should just make a new ini file that is always UTF-8 encoded.

Then there's the save file format. At least the mod name is byte encoded, but are there any mods out there with non-ascii folder names? If Cv4MiniEngine loads a save, what encoding should it assume. Maybe I should just assume UTF-8 and error out on any extended ascii mod names.

 

[Nightinggale]

Then there's the save file format. At least the mod name is byte encoded, but are there any mods out there with non-ascii folder names? If Cv4MiniEngine loads a save, what encoding should it assume. Maybe I should just assume UTF-8 and error out on any extended ascii mod names.

CvDLLUtilityIFaceBase has Compress() and Uncompress(). It seems that apart from the first call to CvGameCore::write(), all calls to write will be compressed with whatever compression method hides behind those two functions. This is why you are not seeing that it seems all names seems to be saved in plain text. Even worse it seems to be saved using the codepage the computer is currently using so transferring savegames between continents works, but the city names (among other things) will look really weird. Log entries are also saved as the resulting text, not the TXT_KEYs, so you better hope whoever made the savegame you are looking at used the same codepage and language as you do or the log will look corrupted.

On top of that, it seems that the exe is saving data outside of what the DLL is saving. Among other things it seems to save an array of unit and building types, which means if you change the number of those in xml (as in adding a new unit info), the savegame gets corrupted even if the DLL is modded to handle such a change. WTP has somewhat managed to get around that with a dirty hack, but it does reveal that we do not have all the savegame code.

I'm not sure it's worth it to try to stay savegame compatible with Firaxis' exe file. It's no worse than people having to start a new game, which a lot of people will likely do anyway. Savegame compatibility is a luxury problem compared to get everything else working.

 

[snowern]

but the city names (among other things) will look really weird

City names? m_szName is a CvWString. Maybe the encoding is messed up elsewhere? Are the XML files even read correctly?

For player names, in CvInitCore, latest commit serialises them without conversion to CvString, and encodes as UTF-8 in the INI. So Unicode names appear to work.

Log entries are also saved as the resulting text

Yes, it's why CvTranslator::kFirstSymbolCode uses the original value.

I think this was CvTalkingHeadMessage/CvPlayer::getGameMessages, but the text is stored as CvWString, which should be unicode.

I'm not sure it's worth it to try to stay savegame compatible with Firaxis' exe file

Useful for testing. And it's easy to do anyway. As long as there are no string encoding problems. Would be nice though to really do "UTF-8 everywhere" in save files, but not all that worth it.

 

[Nightinggale]

text is stored as CvWString, which should be unicode.

Unfortunately that's not the case. Both CvString and CvWString saves using the current codepage. This means while CvWString uses two bytes per character, half of the bytes are 0x00. The only reason to use CvWString is that GameFont IDs starts at 8483, hence GameFont will use two bytes. The number 8483 is hardcoded in the exe and I have no idea why they picked that one.

Useful for testing. And it's easy to do anyway. As long as there are no string encoding problems. Would be nice though to really do "UTF-8 everywhere" in save files, but not all that worth it.

If you know it's 8 bit encoded characters, then the game will work even if you end up reading using the wrong codepage. It will just provide weird names. It's fairly easy to convert from a known codepage to UTF-8. Iconv is a free library, which can do it. Or you can just copy the converter from WTP, which is partly based on iconv.

 

[snowern]

Oh, you piece of archaic... And it just works! Somehow...

*It just works because maybe this is unicode.

 

[Nightinggale]

Or you can just copy the converter from WTP, which is partly based on iconv.

I take that part back. WTP converts from UTF-8 to the currently used codepage. You need codepage to UTF-8. That is however still easy to do. Get iconv, use the windows-1252 decoder. It converts a string into UTF-32. Next you need the UTF-8 encoder header, which converts from UTF-32. You don't need the entire library and you don't need to use options. Just those two header files and make a single call to each for each string.