Thoughts on a community DLL

[Pazyryk]

GameInfoTypes is just a simple Lua table indexed by strings. So no speed problem there at all. GameInfo.Unit is actually of type userdata, meaning it is calling some kind of C++ somewhere. GameInfo.Unit() returns an iterator method. But like any C++ method passed to Lua, you can't see the contents of that method (from Lua), so that part is black box unless it happens to be in our DLL somewhere.


Edit: Accessing GameInfo.Units[1] is also very slow. That construct will get you a table. But it's not a table that was sitting around already in Lua. The GameInfo.Units part is sending you to C++ which (somehow) is generating a table for you. Again, I suspect it is doing this by querying the DB in real time rather than cached values. Actually I quite sure of that. Not only is it very slow, but you can do tests and see that GameInfo.Units[1] returns values that reflect post-init changes in DB.

 

[S3rgeus]

Yeah, I did not think you were. However, I have less faith in the game developers. I think they are doing this for GameInfo.Unit() type iterators passed to Lua. Hence my line of questions above.

This is a tremendous burden on game runtime even in un-modded civ, given all the times these run each turn just for the Top Panel UI.

This could be an easy performance gain for the community DLL if changing those iterators to cached values isn't *too* complicated. I'll take a look at the iterators at some point, but I've got a full plate with other features for now.

The db and all mods are loaded [we jump in here with our changes] and a memory cache is built for GameInfoTypes.

Additions to existing DB tables are definitely cached by GameInfoTypes. I've used GameInfoTypes.FEATURE_BLIGHT and GameInfoTypes.UNIT_HORN_HUNTER from FireTuner many times for testing, both of which were newly added for my mod.

And sure, the code in WhateverMyFunction needs to be changed to use those values, check them, etc. But it had to be either way. Data storage is pretty straight forward. I guess I'd have to see specific examples of what data we'd want to store and retrieve as part of extending the xml tag system, that wouldn't be just easily get/set paired. In a slightly fancier way, that is all GameInfoTypes is doing isn't it?

GameInfoTypes only caches IDs, so it just always returns integers. (That may be where its performance gains come from, actually, but of course we won't know for sure without looking at the implementation.) I believe there is in the tutorial forum about GameInfoTypes. If I remember correctly, any table in the DB that has an integer "ID" field and string "Type" field gets loaded into GameInfoTypes? I'm not sure though.

 

[Hambil]

I've been adding iterators. The ones I've added so far have all had memory structures behind them and did not go to the database. I can only speak to the ones I've looked at so far, though.

I'm confused about how iterators matter, as they are just another method of accessing data. Either we can access it or not. Cache it or not. After that, there is no magic to using it.

 

[S3rgeus]

GameInfoTypes is just a simple Lua table indexed by strings. So no speed problem there at all. GameInfo.Unit is actually of type userdata, meaning it is calling some kind of C++ somewhere. GameInfo.Unit() returns an iterator method. But like any C++ method passed to Lua, you can't see the contents of that method (from Lua), so that part is black box unless it happens to be in our DLL somewhere.

Ah, if GameInfoTypes is just a Lua table then the context switch from Lua to C++ might be what causes the performance degradation between that and the iterators. (And given there's a context switch at every iteration in a loop, it's potentially quite a lengthy process.)

 

[S3rgeus]

I'm confused about how iterators matter, as they are just another method of accessing data. Either we can access it or not. Cache it or not. After that, there is no magic to using it.

I believe Pazyryk (whose name is very hard to spell ) is asking if the iterators provided to Lua can be sped up from the DLL given that they are so much slower than GameInfoTypes for accessing similar data.

 

[Pazyryk]

And GameInfoTypes is not always an option. You can always use it if you want ID. But if you want any other info (Description, Cost, etc...) you have to use GameInfo.Units[ID or str] to get it. [Edit: unless you cache values in Lua, which I actually do for many time critical functions.]

I added in post edit above but probably missed:

Accessing GameInfo.Units[1] is also insanely slow. This construct will get you a table. But it's not a table that was sitting around already in Lua. The GameInfo.Units part is sending you to C++ which (somehow) is generating a table for you. Again, I suspect it is doing this by querying the DB in real time rather than cached values. Actually I'm quite sure of that. Not only is it very slow, but you can do tests and see that GameInfo.Units[1] returns values that reflect post-init changes in DB.

 

[Pazyryk]

Funny thing is, I could actually code this in Lua and probably speed up base UI by 20x or more. GameInfo is just a global (like any other) and could be replaced by a Lua table. Only problem is that GameInfo.Units needs to be (sometimes) a table and (sometimes) a function. But metamethods would solve that I think. Then cache all GameInfo into Lua. And Presto! Massive speed increase.

But really, it's silly that converting base C++ methods to modder Lua methods could speed things up. But I'm quite sure it's true.

 

[Hambil]

I believe Pazyryk (whose name is very hard to spell ) is asking if the iterators provided to Lua can be sped up from the DLL given that they are so much slower than GameInfoTypes for accessing similar data.

Most likely the developers decided that the tables in question where too large to memory cache, or used to infrequently (in their work).

I understand that there is overhead in a context switch from LUA to C++. What I'd be very curious about is how that overhead compares to a DB read. If it really is *that* slow to make the context switch than an iterator will always be slow, unless cached in lua as Pazyryk suggests.

If the context switch is negligible then the problem would be strickly one of data not being cached. Even if they read it, they should time cache it so the next read doesn't have to go to the database again, especially since it's static data.

If this does turn out to be the issue (they are doing reads not caching) then I'd suggest a fix as above (timed cache, where elements expire and can be tossed after x seconds). Then, in the case of an iterator, you'd have one slow read up front then fast after that. Hopefully the issue isn't in the context switch from lua to c++ and back itself. That would suck.

 

[Hambil]

I'm running LuaJit, but, a simple speed test did not give me the results I expected:

speedtest: Simple method call finished 100,000 times in 0.011000000000024
speedtest: GameInfo call finished 100,000 times in 0.0079999999999814
speedtest: GameInfoTypes call finished 100,000 times in 0
speedtest: Iterator call finished 100,000 times in 0.079999999999984

That's 100,000 times through each loop. If there is a performance issue anywhere here I'm missing it. Please tell me what to add to speed test to find it.

Edit: found small bug in test removing and retesting before anyone complains (if not too late lol)
Edit2: Okay, bug fixed, results/post/attachment updated.

 

[Pazyryk]

OK...

speedtest: GameInfo call finished 100,000 times in 0.0079999999999814
speedtest: GameInfoTypes call finished 100,000 times in 0

Here you were comparing access to the same single value using either GameInfo.Units.UNIT_PROPHET.ID or GameInfoTypes.UNIT_PROPHET. The first is passed through GameInfo.Units (userdata which is a C++ call); the second is a simple Lua table access. Consistent with the results of my speed tests, accessing a Lua table value is much faster than using GameInfo (so much so that you didn't really even measure it here).

speedtest: Iterator call finished 100,000 times in 0.079999999999984

This was not a test of a GameInfo type iterator. pPlayer:Units() does pass an iterator, yes, but it isn't one that has any thing to do with DB data. Try something like "for unitInfo in GameInfo.Units() do" instead.

speedtest: Simple method call finished 100,000 times in 0.011000000000024
speedtest: GameInfo call finished 100,000 times in 0.0079999999999814

This comparison does surprise me. I thought something like pPlayer:GetID() would be faster. It's slower than a single GameInfo access (which is the surprising part to me) and it is light years slower than a simple Lua table access (maybe that part shouldn't be surprising).


Edit: I'd be interested in how much LuaJit mattes here. My prediction (which could easily be wrong) is not at all.


Also, you should always localize your variables like start and iSkip. (Yes, I know those were outside the loop so have no effect on your results whatsoever. )

 

[Hambil]

So it's not having an iterator, it's whether or not that iterator accesses the database that matters. I'll add one of those to the test.

Still, if it's anything like the other results I'd have to say it just doesn't matter. You'd have to make millions of those calls in your code in one doturn for the user to even notice.

 

[Hambil]

speedtest: Simple method call finished 100,000 times in 0.012
speedtest: GameInfo call finished 100,000 times in 0.012
speedtest: GameInfoTypes call finished 100,000 times in 0
speedtest: Iterator call finished 100,000 times in 0.039000000000016
speedtest: Iterator DB call finished 100,000 times in 0.61500000000001

Well that last number makes my radar, because multiply that by 8 players and you are starting to have an issue. But that's at 100,000 calls. Really inefficiently designed ones, two

 

[Pazyryk]

Well, my old computer (on which I did my speed tests) took 0.748 seconds to do your last test for only 1000 iterations (well, I had a "Cost < 200" test in there too but that doesn't add much). I wonder if LuaJit is really that good, or if my old computer was really that slow. (It was 2004 vintage, so perhaps the latter -- especially with the strain of running Civ5 at the same time). However, it could do the same iteration over a Lua cached Units table in 0.011 sec (68x faster).

 

[Hambil]

Probably the older computer matter more. You are correct, results will be computer dependent. I'm just curious about what any code that is doing enough to slow down the system, is doing.

 

[Pazyryk]

Darn, just found out that GameInfo has a protected metatable. If not for that, I could write some code to cache frequently used GameInfo tables and generate iterators on them. Would have been cool to be able to do that across all Lua states with one file.

It can still be done by replacing GameInfo with an impostor GameInfo. But the substitution would have to be done in each state (at least one line of code) where you wanted the optimization. Still may be useful in my mod with 20000 lines of Lua running mostly in one state.

 

[whoward69]

The class would use a generic getter and setter (e.g. CvDBExt.Get('MY_NEW_TAG')).

Be very wary of using strings as keys into tables/arrays in C++ within the DLL. The version of the stl (std:: ) library being used does not support ordered_map, so every such look-up comes down to a linear array scan, letter by letter, for the key.

This explains why Firaxis don't do CvDefines::GetValue(char* szKey) and then read the GameDefines values out of a table, but actually have a class with a data member and getter for every individual value, eg GC.getBARBARIAN_CAMP_IMPROVEMENT(), GC.getBARBARIAN_CAMP_ODDS_OF_NEW_CAMP_SPAWNING(), GC.getBARBARIAN_CAMP_FIRST_TURN_PERCENT_OF_TARGET_TO_ADD(), etc.

I've just had to re-write my on/off switch code (which initially used a generic getter and a table of values), as even with only 70 keys in the table we were seeing turn times double.

Any getters must also be declared as inline (to avoid the overhead of creating a stack frame)

 

[Hambil]

So strcmp is slow? Good to know. Unfortunate. I'll have to think now on how to implement this generically. CvDBExt.Get(5) just doesn't have the same ring to it. And if we put a bunch of #defines in to get rid of the magic numbers then it's not generic.

 

[whoward69]

So strcmp is slow?

strcmp() and strncmp() have never been fast - they are a necessary throwback to the way that C (not C++) handles strings - ie as a sequence of charaters (either with a 0 byte terminator or with a count prefix) and not as an encapsulated object.

Compared to modern hash-comparision techniques they are positively glacial. However, without a string object to cache the associated hash-code into and efficient implementions of maps / sparsely populated arrays as B-trees, any gain from hashing will be lost on subsquent lookups and/or the linear scans over the array of values.

 

[Hambil]

Proposed Feature:
Multiple leaders per civilization. Each leader has his/her own UA but the civs UU and UB remain unchanged. I assume this is a major feature addition, as an interface for such a thing doesn't even exist and we'd have to decide how to do the UI for it.

Note: Whoward does something like this. I've yet to determine (time, time, time) if it does what I want exactly, but if it does this Proposal is off the list also.

 

[Hambil]

Proposed Feature:
Post-Industrial faith can be used to 'hurry' wonders. It could maybe be done in lua, but the issue seems to be one of the AI using it...

Okay, been poking around at this. I can get the AI to use it, and if I can figure out how to add a new choice (Wonders) to automatic faith purchases drop down I think I can code this all in lua. It seems more elegant also, to have the game automatically finish a wonder for you if you have enough faith and have it set to auto-complete wonders.

Okay, I think this one is off the proposed list and onto my growing 'mods I want to write' list.