Python--C++ bridge

Can someone explain, how the C++ and Python code correspond with each other?

Inter alia:
- how Cy***.h/.cpp corresponds with the Python code? There are no DLLExport-s, so are they loaded directly into Python?

- why the interfaces of C++ classes used in the Python code do not match? Like, there is getPlayer() method called on a CyGlobalContext object in a python file, but in the C++ class definition there is no such.

- does the python identify objects, like units or cities, from the C++ code by pairs played's id + object's id, or by IDInfo objects? And if the second, can the IDInfo's structure be changed?

And btw question
- does all communication between CvGameCoreDLL and the engine goes through Python?

n47 said:

- how Cy***.h/.cpp corresponds with the Python code? There are no DLLExport-s, so are they loaded directly into Python?

Click to expand...

It's done through boost:ython library.

- why the interfaces of C++ classes used in the Python code do not match? Like, there is getPlayer() method called on a CyGlobalContext object in a python file, but in the C++ class definition there is no such.

Click to expand...

Python only has access to functions that are exposed to it in interface files, and those match perfectly. getPlayer is defined in CyGlobalContextInterface1.cpp and references getCyPlayer, which in turn is declared in CyGlobalContext.cpp. Other definitions are usually more logical (i.e. they match the methods in the original Cv* classes), but everything is defined in interface files with .def function, which exposes C++ functions under a specific name in Python (and normally wrapper functions are exposed i.e. methods from CyPlayer class, which then call those in CvPlayer).

- does the python identify objects, like units or cities, from the C++ code by pairs played's id + object's id, or by IDInfo objects? And if the second, can the IDInfo's structure be changed?

Click to expand...

Python gets whatever objects it receives from C++ using any particular method. It doesn't need to identify them by anything, other than the function arguments.

I'm pretty sure you can't and don't really need to modify IDInfo class (since it's used by the engine).

- does all communication between CvGameCoreDLL and the engine goes through Python?

Click to expand...

No, what is declared as DLLExport is available directly to the engine, and what is defined in Cy* interface files is available to Python, which runs embedded in the game engine.

Still would like to precise.

embryodead said:

Python gets whatever objects it receives from C++ using any particular method. It doesn't need to identify them by anything, other than the function arguments.

Click to expand...

So it identifies them mainly by integer ids? I'm aware it is done by methods, but as the class definitions didn't matched to the python code, I was not sure, what are the types of the arguments.

embryodead said:

I'm pretty sure you can't and don't really need to modify IDInfo class (since it's used by the engine).

Click to expand...

Actually it would be nice do it. I've started to look closer to C2C mod. It is such immense, that referring everything by IDInfo-s, in the way it is done now, may be one of the main problems for the performance. If IDInfo-s are required by the engine, I guess, I will just try to remove it from the inner CvGameCoreDLL communication.

embryodead said:

No, what is declared as DLLExport is available directly to the engine, and what is defined in Cy* interface files is available to Python, which runs embedded in the game engine.

Click to expand...

OK. Still, there are many functions marked as DLLExport with the comment "Exposed to Python". Does it mean, they are used only by Cy* files or they may be in use be the engine either?

And still some questions
- from what I see, ids are generated in CvGameCoreDLL. Am I right?
- ... I forgot
Will ask, when I remind myself.

Again, thank you embryodead. It could take a lot of time to realize by myself, how it works.


edit
I remember.
- is there a lot of the engine--CvGameCoreDLL and Python--CvGameCoreDLL communication? Precisely -- would logarithmic time of getting an object (a unit, a city) from CvGameCoreDLL by the engine or Python be too much?

n47 said:

So it identifies them mainly by integer ids? I'm aware it is done by methods, but as the class definitions didn't matched to the python code, I was not sure, what are the types of the arguments.

Click to expand...

They are sometimes called by IDs, but most common are helper classes such as getUnitList which return a list of player units generated by firstUnit & nextUnit methods from C++. Another common method is getting units from a plot e.g. plot.getUnit(index). But Python itself doesn't reference them by anything, it just gets the object from C++.

Actually it would be nice do it. I've started to look closer to C2C mod. It is such immense, that referring everything by IDInfo-s, in the way it is done now, may be one of the main problems for the performance. If IDInfo-s are required by the engine, I guess, I will just try to remove it from the inner CvGameCoreDLL communication.

Click to expand...

I don't quite understand what you're getting at, or you have a wrong idea about the whole thing... IDInfo is just player ID + array index. There is nothing wrong with it. Use a DLL profiler to see what actually takes time. Generally it will be bad python callbacks, trade networks calculations, pathfinding, AI decision-making, stuff like that.

OK. Still, there are many functions marked as DLLExport with the comment "Exposed to Python". Does it mean, they are used only by Cy* files or they may be in use be the engine either?

Click to expand...

No, it means they can be used internally, by the game engine and by Python.

- from what I see, ids are generated in CvGameCoreDLL. Am I right?

Click to expand...

Yes, they are array indexes.

- is there a lot of the engine--CvGameCoreDLL and Python--CvGameCoreDLL communication? Precisely -- would logarithmic time of getting an object (a unit, a city) from CvGameCoreDLL by the engine or Python be too much?

Click to expand...

No, Python calling functions from the DLL is virtually instant.

On the other hand, DLL calling Python by a callback is very slow, though it's a completely different topic. However, most mods don't use them and even disable some default callbacks, because they're not really needed.

embryodead said:

I don't quite understand what you're getting at, or you have a wrong idea about the whole thing... IDInfo is just player ID + array index. There is nothing wrong with it. Use a DLL profiler to see what actually takes time.

Click to expand...

There are quite many references to objects stored as IDInfo-s in the dll. As retrieving those objects involves FFreeListTrashArray, I thought about replacing the whole mechanism of storing objects with something more effective. However, it actually doesn't seam to elevate significantly the performance. Still, I will need more research to be sure.

embryodead said:

No, it means they can be used internally, by the game engine and by Python.

Click to expand...

Yes, it is obvious. I do not ask about, where they are visible, but does the engine indeed use them.

embryodead said:

No, Python calling functions from the DLL is virtually instant.

On the other hand, DLL calling Python by a callback is very slow, though it's a completely different topic. However, most mods don't use them and even disable some default callbacks, because they're not really needed.

Click to expand...

You aren't in algorithms, are you? Asking about the algorithmic time, means I would like to make the Python called functions from the dll not instant anymore. I've wanted to speed up the dll at the cost of performance of a part of communication Python-->DLL and engine-->DLL. In fact, I've wanted to know, how many times the dll methods are called from the outside, not how long does it takes now.

However, it is not actual anymore. I know how to do what I wanted, without this performance lost. Still, it's a lot of work and I'm afraid, it won't bring a big speed up. So probably I will give up the idea.