Nightinggale
Deity
- Joined
- Feb 2, 2009
- Messages
- 5,269
A question came up in the quick modding questions thread and since the code in question happens to be in the exe, this line showed up:
Why do we want the source code?
Being able to mode everything
There are some code in the exe, which we would like to mod. Most noteworthy is the Main Menu, including all the "submenus" (new game, custom game etc) and the diplomacy screen.
Bugfree and stable mods
Accessing everything will not only allow modding, but also bugfixing. This is not only bugs in the exe itself, but also bugs in the DLL caused by assuming something about the exe, which seems correct at the time, but isn't. Also debugging anything where the exe is involved will also be a whole lot easier.
There are also some undocumented limitations, such as some enum values are hardcoded in the exe. More than once have modders added something, which results in an offset of one, hence broken code.
Better performance
The exe has inefficient code when communicating with the DLL, like it loops and calls a DLL function to get a bool to tell if it's a certain enum of not, meaning we often run into unneeded loops.
Compiling the exe and dll into one file will not only allow just using the enums, it will also allow some compiler/linker optimization, which is not possible today because the dll can't optimize for something in the exe.
Modern tools
Right now we are using a compiler from 2003 with boost and python from 2004. This is problematic because there is no support for those. For instance I looked at boost threads and it turns out it was changed significantly in 2007 (for the much better), meaning getting info on how to do stuff prior to 2007 is near impossible today. Also the dll is unavailable today meaning you have to compile yourself, which turned into a nightmare.
Using a newer compiler also means more advanced algorithms for optimization and ability to optimize for modern hardware. Modern CPUs have more registers and more instructions, which sits idle when running old code. Just compiling something, which is intended precisely for the CPU in question will make the code execute faster.
Modern C++
When looking up how to do stuff in C++, what you find online today is C++14 or perhaps C++11. We use C++03, which doesn't support all the cool stuff people talk about today. Using a modern compiler will allow usage of C++14.
64 bit application
32 bit applications are limited to 4 GB of memory. 64 bit has no realistically reachable limit in address space. This allows bigger mods.
IPv6
The game uses IPv4 for multiplayer, which is in itself ok. The problem is that we have run out of IPv4 addresses and more and more people can't get a public IPv4. IPv6 has more addresses than there are grains of sand on the planet, meaning everybody should be able to get one and will eventually. Adding support for this will ensure that people will not lose multiplayer. IPv4 will not be dropped, though mixing IPv4 and IPv6 in one game will likely never be supported
Speaking of multiplayer, all players act sort of like a server. Say A hosts a game and B and C joins, C can then be refused because A can be contacted, but C can't connect directly to B. With firewalls and stuff, which can become problematic. Access to the source of the exe can allow the host to be relay between B and C if that is the case.
Modern graphics engine
Luckily there is a plan B. Rewriting the graphics interface to use a freely available engine. Unity has great features for this and AFAIK should be free to use until you sell a certain amount of copies. Since we won't be selling copies at all, it should be free to use for us (must verify to be 100% sure. Didn't read everything in their terms). Using a different engine allows modern graphical engine with modern features.
A new graphics engine will also get rid of engine limitations, such as limited to 128 (or was it 127?) rows of plots being drawn when zoomed out.
Cross platform support
With a switch to a cross platform engine like unity, it might be possible to compile for different platforms. We might finally be able to make mac/linux mods, which mods the DLL. This isn't top priority though, more like a possible bonus.
Now what?
The question is who is interested in doing anything for this and how do we do it?
This sparked multiple posts about the topic, which in turn resulted in this thread.@firaxis: Isn't it high time you released the CivIV EXE to a fully open source licencing??? Surely the agreement with Gamebryo has a limited term...
Why do we want the source code?
Being able to mode everything
There are some code in the exe, which we would like to mod. Most noteworthy is the Main Menu, including all the "submenus" (new game, custom game etc) and the diplomacy screen.
Bugfree and stable mods
Accessing everything will not only allow modding, but also bugfixing. This is not only bugs in the exe itself, but also bugs in the DLL caused by assuming something about the exe, which seems correct at the time, but isn't. Also debugging anything where the exe is involved will also be a whole lot easier.
There are also some undocumented limitations, such as some enum values are hardcoded in the exe. More than once have modders added something, which results in an offset of one, hence broken code.
Better performance
The exe has inefficient code when communicating with the DLL, like it loops and calls a DLL function to get a bool to tell if it's a certain enum of not, meaning we often run into unneeded loops.
Compiling the exe and dll into one file will not only allow just using the enums, it will also allow some compiler/linker optimization, which is not possible today because the dll can't optimize for something in the exe.
Modern tools
Right now we are using a compiler from 2003 with boost and python from 2004. This is problematic because there is no support for those. For instance I looked at boost threads and it turns out it was changed significantly in 2007 (for the much better), meaning getting info on how to do stuff prior to 2007 is near impossible today. Also the dll is unavailable today meaning you have to compile yourself, which turned into a nightmare.
Using a newer compiler also means more advanced algorithms for optimization and ability to optimize for modern hardware. Modern CPUs have more registers and more instructions, which sits idle when running old code. Just compiling something, which is intended precisely for the CPU in question will make the code execute faster.
Modern C++
When looking up how to do stuff in C++, what you find online today is C++14 or perhaps C++11. We use C++03, which doesn't support all the cool stuff people talk about today. Using a modern compiler will allow usage of C++14.
64 bit application
32 bit applications are limited to 4 GB of memory. 64 bit has no realistically reachable limit in address space. This allows bigger mods.
IPv6
The game uses IPv4 for multiplayer, which is in itself ok. The problem is that we have run out of IPv4 addresses and more and more people can't get a public IPv4. IPv6 has more addresses than there are grains of sand on the planet, meaning everybody should be able to get one and will eventually. Adding support for this will ensure that people will not lose multiplayer. IPv4 will not be dropped, though mixing IPv4 and IPv6 in one game will likely never be supported
Speaking of multiplayer, all players act sort of like a server. Say A hosts a game and B and C joins, C can then be refused because A can be contacted, but C can't connect directly to B. With firewalls and stuff, which can become problematic. Access to the source of the exe can allow the host to be relay between B and C if that is the case.
Modern graphics engine
This seems to be the biggest concern and rightfully so. Graphics engines are big and complex to develop, meaning companies who have done so keep them close to themselves. We might not be able to get the graphics engine at all.I mentioned Gamebryo because from all the chatter on the forums here, best I can tell is that the main hangup that may never allow it is the Gamebryo graphic engine contract.
Luckily there is a plan B. Rewriting the graphics interface to use a freely available engine. Unity has great features for this and AFAIK should be free to use until you sell a certain amount of copies. Since we won't be selling copies at all, it should be free to use for us (must verify to be 100% sure. Didn't read everything in their terms). Using a different engine allows modern graphical engine with modern features.
A new graphics engine will also get rid of engine limitations, such as limited to 128 (or was it 127?) rows of plots being drawn when zoomed out.
Cross platform support
With a switch to a cross platform engine like unity, it might be possible to compile for different platforms. We might finally be able to make mac/linux mods, which mods the DLL. This isn't top priority though, more like a possible bonus.
Now what?
The question is who is interested in doing anything for this and how do we do it?
Last edited: