Mini-engine progress

[snowern]

Wait, this is ISO-8859-1, so it should be decodable as UTF-16, right?

Windows-1252 - Wikipedia

en.wikipedia.org

ISO/IEC 8859-1 - Wikipedia

en.wikipedia.org

Latin-1 Supplement - Wikipedia

en.wikipedia.org

Civ4 wouldn't let me type in other characters, like € from Windows-1252.

 

[Nightinggale]

Wait, this is ISO-8859-1, so it should be the decodable as UTF-16, right?

For reasons which is still beyond me, the vanilla xml files claims to be ISO encoded in the header, but they are actually windows-1252 encoded and the exe/dll ignores the header and assumes windows-1252. All translations in vanilla are languages, which uses windows-1252, so the game never properly supported leaving western Europe or anywhere else using western European languages. If windows is set to a language not using windows-1252, then vanilla can have issues.

 

[snowern]

Okay, I did PyPlayer(0).getCityList()[0].setName(u"1€2¬3¦4\u266b") and Civ4 does preserve Unicode:

*Wait... 0x80 is not valid unicode. This is a mishmash? Script was Windows-1252 encoded.
*Or maybe the script was interpreted as ISO-8859-1.

 

[snowern]

Possibly switching over to pugixml. It appears to handle encodings better: https://pugixml.org/docs/manual.html#loading.encoding

Using this to test:

        <Tag>TXT_KEY_CITY_NAME_BERLIN</Tag>
        <English>Berlin£€Â‡&#x266b;</English>

TinyXML2 just takes the bytes and returns them through element->GetText(). I can then interpret as either codepage or UTF-8, but which? And TinyXML2 will happily expand your unicode character entity in codepage text.

Pugixml interprets the file as ISO-8859-1 and decodes to UTF-8 for you.

Not completely correct though as it will pass through all bytes to UTF-8 encoding, so you potentially get all those high control bytes in https://en.wikipedia.org/wiki/Latin-1_Supplement#Character_table.

This will only be a problem if there are Windows-1252 XML files, with those Windows-1252 characters. This occurs in:

• Beyond the Sword\Mods\Rhye's and Fall of Civilization\Assets\XML\Text\CIV4GameText_FourthRoundTranslation.xml
  • "¡Las ciudades"
  • "nom qu’elles"
• Beyond the Sword\Mods\The Road to War\Assets\XML\Text\Civ4GameText_WW2_ADDON1.xml
  • Lots of hyphens
• Beyond the Sword\Mods\The Road to War\Assets\XML\Units\CIV4UnitInfos.xml
  • A hyphen

Not a whole lot. And could be worked around in text drawing, because these bytes will get decoded to characters that shouldn't be used.

 

[Git-384]

TinyXML2 just takes the bytes and returns them through element->GetText(). I can then interpret as either codepage or UTF-8, but which?

The XML header should have the encoding of the file and you could use that. However if XML files include Windows-1252 characters but the header says ISO-8859-1 or something else this doesn't work.

Converting all the XML files to UTF8 and fixing codepage issues in the process might be a solution that doesn't require a bunch of workarounds in the code.

 

[snowern]

Not really a workaround, it's a solution. It decodes to UTF-8 as an XML parser should. But the odd Windows-1252 character would need workarounds, if one were to support those mods. Like the python 3 idea, I would prefer to avoid shipping a whole collection of patches or patched assets.

Or, you could patch at runtime. Run encoding detection or assume from the header. That would probably work. But, how do you get the header without writing a mini XML parser.

*Could probably patch string values as they are returned, and with pugixml, it tells you whether the file is "latin" or unicode.

 

[Nightinggale]

Or, you could patch at runtime.

You probably have to figure out a fix for realtime because you better support unmodified vanilla files. All the vanilla files are encoded in windows-1252 regardless of what the header says.

 

[snowern]

Yes, it's something I could add support for, but it's 99% ISO-8859-1, if my list of exceptions is correct. The only Windows-1252 I've found is in mods that you would not be able to use anyway.

 

[snowern]

Who wants a binary release?

Maybe it'll work. Then I can upgrade to actual v1.0.

Compiled with Clang for max performance. With a quick fix for gigantic maps.

Grab https://www.python.org/downloads/release/python-2718/ and https://learn.microsoft.com/en-us/cpp/windows/latest-supported-vc-redist (VS 2022).

Latest commits: https://github.com/fortsnek9348/Cv4MiniEngine/commits

There is an odd lingering quirk though that gigantic maps (huge 5x) seem to have less resources in the capital. Not sure if that's a bug or I'm just seeing things.

 

[Gameboy2396]

Seriously I cannot believe we got to this point. One year on, and look what you made! My save had generated ridiculously fast - it took only a second or something to generate a Large size map. Even just taking terms feels, like, instant. I'm so proud of you, @snowern! Now it's all about getting graphics and, like, an engine that doesn't use command prompt or something.

 

[1SDANi]

I'm struggling to compile with MSVC. Any chance we might get a step by step compilation guide?

 

[Git-384]

Compiled with Clang for max performance.

Have you tried using PGO or PGO+LTO for the Clang build?

 

[snowern]

Seriously I cannot believe we got to this point. One year on, and look what you made! My save had generated ridiculously fast - it took only a second or something to generate a Large size map. Even just taking terms feels, like, instant. I'm so proud of you, @snowern! Now it's all about getting graphics and, like, an engine that doesn't use command prompt or something.

India? It's like you're playing with the same random seed I always used.

I'm struggling to compile with MSVC. Any chance we might get a step by step compilation guide?

There are steps with cmake *for Linux. With VS, the project should be ready to go once you update the dependencies. On Windows, I think you would need to build pugixml, zlib, native file dialogs. You'd probably use cmake-gui. And you have to build the INSTALL target to get the proper directory layout the project expects.

Have you tried using PGO or PGO+LTO for the Clang build?

Doesn't seem you can do that on Windows clang-cl. I'll have to try it on WSL.

*Okay, couldn't do it on WSL for some reason. But I could do it in VS by using the command line options directly. The 520x320 turn 650 save I have improved from 23.82s to 22.7913s for turn 651, which isn't really a whole lot.

 

[1SDANi]

India? It's like you're playing with the same random seed I always used.


There are steps with cmake *for Linux. With VS, the project should be ready to go once you update the dependencies. On Windows, I think you would need to build pugixml, zlib, native file dialogs. You'd probably use cmake-gui. And you have to build the INSTALL target to get the proper directory layout the project expects.


Doesn't seem you can do that on Windows clang-cl. I'll have to try it on WSL.

*Okay, couldn't do it on WSL for some reason. But I could do it in VS by using the command line options directly. The 520x320 turn 650 save I have improved from 23.82s to 22.7913s for turn 651, which isn't really a whole lot.

Ah, INSTALL is a target. That explains a lot. Wasn't sure if it was just a different name for the build folder or the INSTALL file or something else. Thank you so much for the help!

 

[1SDANi]

Ah, INSTALL is a target. That explains a lot. Wasn't sure if it was just a different name for the build folder or the INSTALL file or something else. Thank you so much for the help!

Why does compiling on Windows always have to some convoluted ritual? I wish 99% of the difficulty of modding wasn't just getting started

 

[snowern]

Why does compiling on Windows always have to some convoluted ritual? I wish 99% of the difficulty of modding wasn't just getting started

It's the same on Linux really. Download the binaries you can and build the rest with cmake. I had to figure it out too. But it would be nice if binaries were available for everything. Updated versions, Ubuntu.

This video certainly enlightened me on how building should work:

 

[<Nexus>]

So there is no "ready-to-be-played" version at the moment?

 

[1SDANi]

It's the same on Linux really. Download the binaries you can and build the rest with cmake. I had to figure it out too. But it would be nice if binaries were available for everything. Updated versions, Ubuntu.

This video certainly enlightened me on how building should work:

At least in my experience from when I ran Linux before that laptop got bricked by crashing in the middle of updating the distribution I was running, most programs on Linux just require running a list of commands in sequence, whilst Windows often asks you to recursively follow readmes while hunting down insanely specific versions of programs on the internet, at times resulting in having to then get insanely specific versions of build tools, and all the while having to deal with the fact that, if you make a single misassumption MSVC will spit out the least helpful error messages, and all the while you are hoping everything uses ninja, cmake, msvc, or literally anything other than make because somehow despite being the most common way to build programs it also has the most weird edge case scenarios. I've seen far more compilation processes that are 10+ steps on windows and literally a 1 liner on Linux than I've seen the inverse.

 

[snowern]

So there is no "ready-to-be-played" version at the moment?

Binary release candidate above.

I've seen far more compilation processes that are 10+ steps on windows and literally a 1 liner on Linux than I've seen the inverse.

In this case, you can use commands too, because you can tell cmake to build and install from the command line. I don't do that for libs though, I use the GUI. For Cv4MiniEngine, I do have a script to build and install that's similar to the Linux script:

Spoiler :

if "%VSINSTALLDIR%"=="" (
    echo Not in VS Developer Command Prompt. Exiting.
    exit /b 1
)

mkdir BuildRegular
mkdir BuildAVX2
mkdir BuildAVX512
mkdir Cv4MiniEngine
set LIBS=D:\Lib

set CMAKE_PREFIX_PATH=%LIBS%\SFML-3.0.2\lib\cmake\SFML;C:\Python27;%LIBS%\nativefiledialog-extended-1.2.1\install\lib\cmake\nfd;%LIBS%\pugixml-1.15\Install\lib\cmake\pugixml
REM set CMAKE_PREFIX_PATH=%CMAKE_PREFIX_PATH%;%VSINSTALLDIR%Common7\IDE\CommonExtensions\Microsoft\TeamFoundation\Team Explorer\Git\mingw64\bin

set COMMON_OPTIONS=-T ClangCL ^
    -DPYBIND11_INC_DIR="%LIBS%\pybind11-2.9.2 clean\include" ^
    -DCMAKE_PREFIX_PATH="%CMAKE_PREFIX_PATH%" ^
    -DS3TC_DIR=%LIBS%\s3tc-dxt-decompression-master ^
    -DZLIB_ROOT=%LIBS%\zlib-1.3.1\install ^
    -DSTB_INC_DIR=%LIBS%\stb-master ^
    -DENABLE_NFD=ON

cmake %COMMON_OPTIONS% -S ..\CmdCiv4 -B BuildRegular || exit /b %errorlevel%
cmake --build BuildRegular --config Release || exit /b %errorlevel%
    
cmake %COMMON_OPTIONS% -S ..\CmdCiv4 -B BuildAVX2 -DENABLE_BUILD_ENGINE=OFF -DENABLE_GAMECOREDLL_ENHANCEMENTS=ON || exit /b %errorlevel%
cmake --build BuildAVX2 --config Release || exit /b %errorlevel%
    
cmake %COMMON_OPTIONS% -S ..\CmdCiv4 -B BuildAVX512 -DENABLE_BUILD_ENGINE=OFF -DENABLE_GAMECOREDLL_ENHANCEMENTS=ON -DENABLE_AVX512=ON || exit /b %errorlevel%
cmake --build BuildAVX512 --config Release || exit /b %errorlevel%

cmake --install BuildRegular --prefix Cv4MiniEngine || exit /b %errorlevel%
cmake --install BuildAVX2 --prefix Cv4MiniEngine || exit /b %errorlevel%
cmake --install BuildAVX512 --prefix Cv4MiniEngine || exit /b %errorlevel%

 

[Nightinggale]

At least in my experience from when I ran Linux before that laptop got bricked by crashing in the middle of updating the distribution I was running, most programs on Linux just require running a list of commands in sequence, whilst Windows often asks you to recursively follow readmes while hunting down insanely specific versions of programs on the internet, at times resulting in having to then get insanely specific versions of build tools, and all the while having to deal with the fact that, if you make a single misassumption MSVC will spit out the least helpful error messages, and all the while you are hoping everything uses ninja, cmake, msvc, or literally anything other than make because somehow despite being the most common way to build programs it also has the most weird edge case scenarios. I've seen far more compilation processes that are 10+ steps on windows and literally a 1 liner on Linux than I've seen the inverse.

I will say the main difference is not compiling itself, but rather that Linux tend to come with a package manager meaning "hunting down insanely specific versions of the programs" becomes something like "apt-get program" and that command will not only give you the program itself, it will automatically also give you everything it depends on. MacOS has 3rd party package managers while I'm not aware of any such managers for windows unless you use some of the tools to make Windows more Linux like, which is often not what you want to do if you want peak performance.

Microsoft started adding the ability to mix code compiled by different versions of Visual Studio in 2015, but it's not backward compatible meaning if one part is compiled before 2015, everything has to be compiled with that version. Linux doesn't suffer from this, which is another reason why Windows feels more tricky to set up at times. All civ4 releases are compiled with visual studio 2003, which is why that specific version is needed for compiling mods. This sucks because it means the result is optimized for 2003 hardware and doesn't benefit from hardware improvements from the last 20 years such as 64 bit support, more CPU registers etc. There is ways around that (at least in theory), but then you are really talking about difficulties setting up for compiling.

It is possible to do some sort of package management on Windows. WTP does that in order to provide everybody with the correct version of the compiler, through it's homemade, so it's cloning a repository I made.