Python alternative to Callbacks

[OrionVeteran]

Through some exhaustive testing, I have noticed how having callbacks can really slow down performance. The performance lag time is very annoying. I want to try and get some speed back.

My Question: Without using SDK, is there a Python method that can be used as a replacement for the USE_CAN_TRAIN_CALLBACK or USE_CANNOT_TRAIN_CALLBACK?


Orion Veteran

 

[davidlallen]

Those routines *are* the callbacks for these. There are no alternatives. Some of the callbacks are called very often. canMoveInto is probably the worst. canConstruct and canTrain are also pretty bad.

If you comment out the body of the callbacks temporarily, do you get the speed back? Sometimes the thing causing the slowdown may not be what you expect.

It is possible that the routines can be sped up by rewriting the code. Sometimes there are operations which are constant, and they can be computed once at the start of the game. For example, the number of peaks around a city is "unlikely" to change unless you also have some sort of terraforming mod.

 

[Afforess]

Without using SDK,

There is a bright and beautiful world out there, and you are missing it.

 

[OrionVeteran]

Those routines *are* the callbacks for these. There are no alternatives.

No Alternatives... Drat!

Some of the callbacks are called very often. canMoveInto is probably the worst. canConstruct and canTrain are also pretty bad.

Those callbacks are the ones I'm using.

If you comment out the body of the callbacks temporarily, do you get the speed back?

Yes, The performance hit is definitely coming from the callbacks, as most my python code has been optimized.

Thanks for the reply.


Orion Veteran

 

[OrionVeteran]

There is a bright and beautiful world out there, and you are missing it.

I think my resistance is futile and I may eventually become assimilated!

This just means you will have to teach me how to code in SDK and how to compile the DLL.

Orion Veteran

 

[davidlallen]

You probably know the first steps. Get refar's guide from the modiki, get the 3.19 makefile from the modiki, and compile the vanilla sdk. If there are specific callbacks you are targeting for speedup, the sdk function names are probably similar to the python names. A line-by-line translation probably isn't right, but at least it will point you into the right places in the sdk code.

 

[OrionVeteran]

You probably know the first steps. Get refar's guide from the modiki, get the 3.19 makefile from the modiki, and compile the vanilla sdk. If there are specific callbacks you are targeting for speedup, the sdk function names are probably similar to the python names. A line-by-line translation probably isn't right, but at least it will point you into the right places in the sdk code.

The problem we have with compiling a new DLL for Civ4 is that the DLL is no longer standard and for the most part only the game designed to use the new compiled DLL can use it.

I have a inovative idea. As you know, BUG has allowed us to modularize the Python files and functions, so that multiple files can all be called; even though these files may have the same functions in them. This allows me to run say, Mine Warfare with the Inquisitions mod, where both python files may have code that needs to run within the same function in say a XXXgameUtils.py file. This along with using WoC for XML is ideal for separating out the files and functions for the respective mods. Let's go one step further.

Is there a way to split out the SDK into sub routines in sub DLLs that can be called by a new standard DLL? For example: I want to place all of the SDK requirements for Dales Combat Mod into it's own sub DLL. I also want to place all of the SDK requirements for Mountains back in Service into it's own sub DLL. Lastley, I want the main DLL to look in sub folders for any existing sub DLLs and subsequently run them if they exist. Modular DLLs like we have modular Python --- Can it be done?


Orion Veteran

 

[Afforess]

The problem we have with compiling a new DLL for Civ4 is that the DLL is no longer standard and for the most part only the game designed to use the new compiled DLL can use it.

I have a inovative idea. As you know, BUG has allowed us to modularize the Python files and functions, so that multiple files can all be called; even though these files may have the same functions in them. This allows me to run say, Mine Warfare with the Inquisitions mod, where both python files may have code that needs to run within the same function in say a XXXgameUtils.py file. This along with using WoC for XML is ideal for separating out the files and functions for the respective mods. Let's go one step further.

Is there a way to split out the SDK into sub routines in sub DLLs that can be called by a new standard DLL? For example: I want to place all of the SDK requirements for Dales Combat Mod into it's own sub DLL. I also want to place all of the SDK requirements for Mountains back in Service into it's own sub DLL. Lastley, I want the main DLL to look in sub folders for any existing sub DLLs and subsequently run them if they exist. Modular DLLs like we have modular Python --- Can it be done?


Orion Veteran

No. And even if it could be done; I would strongly advise against it. The code in the SDK is too tightly woven together. If you, for example, used two dll's, like Better AI and Mountains back to service together, you would have conflicts. You are going to merge everything manually and use only one.

 

[davidlallen]

The problem we have with compiling a new DLL for Civ4 is that the DLL is no longer standard and for the most part only the game designed to use the new compiled DLL can use it.

I am sure that is true. I was describing the first few steps you should take, if you want to become an sdk wizard. Before compiling your custom dll, make sure you can compile the standard one.

 

[EmperorFool]

What you describe about having modules that the DLL can call that are plugged in at runtime are exactly what DLLs are designed to do. In olde times, all of the code was compiled and linked into a single application (EXE) with no configurability. DLLs--dynamic link libraries--were designed to allow you to decide at runtime to link to functions from different modules.

I suspect it would be technically possible to break out extra DLLs for the modules that you would link in at runtime, but a) I'm not sure a DLL can link to another DLL (probably can) without the EXE getting involved, and b) as Afforess said, most of the DLL mods have code that weaves in to far too many functions to be pluggable.

Now, this would be perfect for mods that just want to recode callbacks and events in C++ to make them faster. It wouldn't be too hard to make a system of callbacks that would first call a configured set of C++ callbacks before calling the Python one (or instead or optionally, whatever). When I say this I mean conceptually as the idea is already there in the code. Writing the actual code would not be easy. It's code I've done in Python, but there it's a snap.

Unfortunately, I have zero knowledge of true Windows programming. I know many of the concepts like MFC and DLLs, but I have never used them myself. I would be quite hesitant to tackle a project of this difficulty and magnitude--given all the other things I'm juggling--knowing that I'd have to bumble through much of it, especially with the looming fear that I'd get 80% in only to find out that it is, in the end, technically impossible.

If someone can figure out how to have the DLL dynamically load other DLLs on-the-fly and wants to build this project, I'm happy to help out with technical design work and some coding. It sounds like a neat project, but I wonder how many modders will make use of it. I really don't see it expanding beyond porting callbacks from Python to C++, but then maybe I'm not seeing the big picture.