Using scriptData vs. new variables in DLL - performance

[embryodead]

Yeah! What exactly did you do? (It would probably be harder for me to find it in the next RFC Marathon update.)

I moved all get/set functions to StoredData (well, tested with Stability so far) + what I've explained in the last post, i.e. no instantiation. All modules use "from StoredData import sd" (it's not initialized in Event Handler).

EDIT: to make it clear, scriptDict is member of StoredData created on __init__ but since there's just one instance of StoredData (sd) it's accessible across all modules. In other words, modules import one same instance, not a class. sd.load() and sd.save() do the pickling.

You mentioned a sort of "modders RFC edition" in a post awhile ago? What do you say about we join up and make RFC Marathon the next generation of RFC? That would mean minimal changes to actual gameplay (except balancing the different levels of play) but added support for modding and customization. (I believe this is already what you're doing, but I'm yet to actually look at your work.)

What I'm looking to do myself is to start going through Rhye's Python code and see what improvements could be done for performance. And also divide key features into easily accessible methods, instead of the 100s of lines of code that needs to be processed as is now. Unless you're already doing this I could share with you the fruits of my own labor. And then I'd be able to use RFC Marathon for my own future modding. (The final version of PyScenario would be a add-on to RFC Marathon, obviously.)

Frankly, RFCMarathon was just a request, what I've been doing is that Middle Eastern mod which is done but with 36 civs and some extra stuff it's a bit slow hence the attempts to improve turn times. We can work on general RFC improvements though, for sure. The two main things that help are the CAR mod and this de-pickling I guess. I also disabled a few more python callbacks in the SDK. With modding-friendly setup it's a matter of un-hardcoding hundreds of functions and values scattered over different SDK files though I guess it's a topic for RFC modmod forum.

 

[Baldyr]

I moved all get/set functions to StoredData (well, tested with Stability so far) + what I've explained in the last post, i.e. no instantiation. All modules use "from StoredData import sd" (it's not initialized in Event Handler).

EDIT: to make it clear, scriptDict is member of StoredData created on __init__ but since there's just one instance of StoredData (sd) it's accessible across all modules. In other words, modules import one same instance, not a class. sd.load() and sd.save() do the pickling.

I'm not quite sure how you managed to create a StoredData instance in the StoredData module. (That line would appear in the code before the class definition or the init definition.) Also, isn't every module importing the module StoredData creating a new instance of the StoredData class? (And if they are only importing the sd value, would that be a global value then? When/where exactly is the instance created?)

This is confusing stuff, for sure... But you learn a lot by trying to wrap your head around how it works.

Frankly, RFCMarathon was just a request, what I've been doing is that Middle Eastern mod which is done but with 36 civs and some extra stuff it's a bit slow hence the attempts to improve turn times.

Ah, I forget about the Middle-Eastern thing you're working on...

We can work on general RFC improvements though, for sure. The two main things that help are the CAR mod and this de-pickling I guess. I also disabled a few more python callbacks in the SDK. With modding-friendly setup it's a matter of un-hardcoding hundreds of functions and values scattered over different SDK files though I guess it's a topic for RFC modmod forum.

Regardless, I can't even see how any future work on RFC would be done without the Marathon mod incorporated (once its fully tested and updated). Its everything any RFC player would ever hope for and anything done with another version of the mod will mostly be ignored.

How difficult would it be to transfer RFC Marathon to the BUG setup, by the way? I'm yet to look into either project for real, but it does seem like a big step forward...

Also, such a RFC - the Next Generation version could incorporate future fixes. I know there are some reported already...

 

[embryodead]

I'm not quite sure how you managed to create a StoredData instance in the StoredData module. (That line would appear in the code before the class definition or the init definition.) Also, isn't every module importing the module StoredData creating a new instance of the StoredData class? (And if they are only importing the sd value, would that be a global value then? When/where exactly is the instance created?)

No, modules are not instantiated, only classes are.

The " sd = StoredData() " line appears after all definitions, outside the class, in the main scope.

In practice sd is global, that is, it's directly accessible in all modules that import it, e.g. sd.getStability(i) works in any context.

Regardless, I can't even see how any future work on RFC would be done without the Marathon mod incorporated (once its fully tested and updated). Its everything any RFC player would ever hope for and anything done with another version of the mod will mostly be ignored.

I'll try to fix the 2 remaining issues this weekend, that should finish the GameSpeed-only purpose of the mod.

How difficult would it be to transfer RFC Marathon to the BUG setup, by the way? I'm yet to look into either project for real, but it does seem like a big step forward...

Same as with regular RFC. I'm not very familiar with BUG's code, but it sounds like a lot of work, possibly SDK too. Just merging would easier, but if you want the modular, more object-oriented setup of BUG it means basically rewriting RFC, I think...

---

BTW I had the same problem with SdToolKit conflicting with StoredData, so I ported Mercenaries to use the latter, but it should be the other way round I guess, since this is just silly:

				'mercenaryData': {
					"AvailableMercenaries" : {},
					"HiredMercenaries" : {0:{}, 1:{}, 2:{}, 3:{}, 4:{}, 5:{}, 6:{}, 7:{}, 8:{}, 9:{}, 10:{}, 11:{}, 12:{}, 13:{}, 14:{}, 15:{}, 16:{}, 17:{}, 18:{}, 19:{}, 20:{}, 21:{}, 22:{}, 23:{}, 24:{}, 25:{}, 26:{}, 27:{}, 28:{}, 29:{}, 30:{}, 31:{}, 32:{}, 33:{}, 34:{}},
					"MercenaryGroups" : {},
					"MercenaryNames" : {},
					"UnplacedMercenaries" : {},
					},

That's the hardcoded mercenary dictionary, with SdToolKit these values are created on the fly. Making it dynamic would also save from adding tens of 0's every time a new civ is added etc.

A quick test with a 200 turn autoplay confirms your tests so far, about 10% speed increase from early to mid game, should be even better late game.

 

[embryodead]

This is of course skewed with randomness, but my 282 turn start, previously ~20 min, just took 11 mins.

 

[Baldyr]

The " sd = StoredData() " line appears after all definitions, outside the class, in the main scope.

In practice sd is global, that is, it's directly accessible in all modules that import it, e.g. sd.getStability(i) works in any context.

Ah, this explains it. I was originally toying with the same concept but failed for some reason to make it work, so I dropped it.

But this means that you need to alter every instance of self.getStabilty to sd.getStability, right? That's a lot of work, considering the sheer number of functions included...

I, on the other hand, used find-replace to get rid of the statements responsible for the pickling. It took like 4 minutes to process the whole mod.

I'll try to fix the 2 remaining issues this weekend, that should finish the GameSpeed-only purpose of the mod.

This is great news! Will you be updating it regularly or are you pretty much dropping it once done?

BTW I had the same problem with SdToolKit conflicting with StoredData, so I ported Mercenaries to use the latter, but it should be the other way round I guess, since this is just silly:

Yeah, I didn't even bother with it myself. So I used a CyPlayer instance to store the scriptDict instead...

A released version would of course merge everything to a SdToolkit setup or similar, but that just seemed like a lot of work... (And I'm converting to RFC Marathon myself, eventually.)

A quick test with a 200 turn autoplay confirms your tests so far, about 10% speed increase from early to mid game, should be even better late game.

Good to hear! I was actually expecting greater gains originally, but 10+% is still good.

 

[embryodead]

But this means that you need to alter every instance of self.getStabilty to sd.getStability, right? That's a lot of work, considering the sheer number of functions included...

I, on the other hand, used find-replace to get rid of the statements responsible for the pickling. It took like 4 minutes to process the whole mod.

I have find-replace as well though OK, this took maybe 30 minutes with all the typos I've made.

This is great news! Will you be updating it regularly or are you pretty much dropping it once done?

I don't know really, as such there's no reason to update it - but other projects can be started from there.

Good to hear! I was actually expecting greater gains originally, but 10+% is still good.

It's 45% on the second test, but I'm not jumping to conclussions before I have a larger sample.

 

[Baldyr]

I don't know really, as such there's no reason to update it - but other projects can be started from there.

The best way forward would probably be a community maintained new version of RFC, like RFC 2 or something. With Rhye's blessing, of course. The initial release would pretty much be the final version of RFC Marathon but it could also include some fixes. Requests from the community for changed or additional content could perhaps be included as independent add-ons or patches.

It's 45% on the second test, but I'm not jumping to conclussions before I have a larger sample.

Thats actually a lot, but you're right - it might not be very common...

I'm just pleased to know that the mod isn't doing all that pickling, all the time. I actually believe that if you ask the game to do it too frequently not even the cPickle module can cope with the workload and there will be additional lag generated from this. Because I got some pretty weird timing when I unintentionally made the mod pickle the scriptDict some 20 times in a row. The lag was more than 20 times what ever it takes to do one of those pickle.dumps(). Perhaps it was the combination of pickle.loads() in between every iteration? (So the game was continously writing and reading the same scriptData...)