Optimization suggestions

[AnotherPacifist]

Shame on me, but I still haven't tried it yet. But for Russia's UP, have you taken into account barbs and independents (which may not necessarily show up as war "officially")?

 

[musicfreak]

Shame on me, but I still haven't tried it yet. But for Russia's UP, have you taken into account barbs and independents (which may not necessarily show up as war "officially")?

Actually, I realized that there's no way to check if you are at war with someone without looping through every civ and checking them independantly, so there goes that suggestion.

 

[kravixon]

Pacifist, you should definitely try it. As of now, the only thing that has changed is that it runs in a faster format. He hasn't gotten around to changing how the UHV's register yet.


Anyway, as of 1450, the game seems to run a lot faster. I'll have to try it later as America and see how long the turns are later on.

Thank you for your changes. This really makes Rhye's a lot more enjoyable for me.

 

[musicfreak]

Damn, I'm going to have to put up a download for RFC RAND; I just tried the same changes and the game is about 4x faster. Crazy. Honestly, I don't feel like I'm playing RFC anymore. The slowness is still felt in the normal RFC though, but at least from what I've noticed, my changes helped.

I'll be working some more on this, but right now I can't really make too many changes because I get an EOF error whenever I make any big changes...do any of you programmer/techie people have any clue as to why that is?

 

[rapiduser]

Update!

I found more files that could use this optimization. I've also done a couple other small things that I'm not going to try to explain. Again, this is for BTS only, other version will come soon, don't worry.

This is great thank you very much. I love to play this fast mod

 

[kairob]

Thanks so much, this makes the game so much faster. I can finally play into the modern era!

 

[musicfreak]

Thanks guys! I've posted these changes for RFC RAND as well, if you wanna check it out.

 

[Metal Alloy Man]

Hey Musicfreak, any chance of showing and explaining to us these changes when your done as I would love to know what your doing being a beginner modder and all?

 

[musicfreak]

Yes, gladly. All I did was go through every Python file (except in the screens\ and EntryPoints\ directories; no need to go there) and replace this:

import pickle

with this:

import cPickle as pickle

(It's found near the top of most of the Python files.)

Basically, pickle is a Python module that turns Python data into readable text that can be saved somewhere. RFC uses this module to store everything that isn't built into the game (stability, mercenaries, plague, etc).

However, pickle is really slow, so I've replaced it with cPickle. It's the exact same thing, only written in the C programming language, making much faster (up to 1000 times faster according to the Python documentation). The as pickle part just renames it to "pickle" instead of "cPickle", so I don't have to manually go and change every line that says pickle to cPickle.

Does that help?

You can do this on any version of RFC (normal, RAND, multiplayer) for any version of Civilization and it will do the same thing. In fact, you can do this for any mod. cPickle and pickle are the exact same thing, it's just that one is much faster than the other. If this isn't clear, you can open up one of the files I included (for example, StoredData.py), and near the top you'll find a line that ends with # musicfreak. You can compare it to the original file to see the change, if my explanation didn't make any sense. Yes, that one line made the mod twice as fast. Imagine what rewriting the entire mod would do.

Also, I edited the AIWars.py and CityNameManager.py files with a different change. Basically (don't worry if you don't understand this), the original files had tables that had the whole map laid out. For example, CityNameManager.py had a tuple (a list that you can't change) that stored each plot on the whole map for each civ. Whenever a civ would settle a city, Python would look up the civ and the coordinates in the table and would name the city based on the result. The problem with this approach is the size (there were 124x68x27 entries total...ouch). Even plots that didn't have city names, and plots that couldn't even have cities in the first place (oceans, mountains, etc) still had entries in the table. All I did was removed those extra entries, and sorted the table by civ->x coordinate->y coordinate.

In case you're a visual person, I'll show you. Here's a section from the original file:

#Vikings
((	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	),
(	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	),
(	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"Narvik",	"Hammerfest",	"Hammerfest",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	"-1",	),

Notice how every square is stored, for every civilization, regardless if it has a city there or not. You think that was bad? That was only 3 rows (out of 68), and only for the Vikings. Here's what I turned that section into:

11: {   28: {47: 'Fort Casimir', 48: 'Fort Kristina'},
			29: {47: 'Älfsborg', 48: 'Uppland', 49: 'Torne'},
			30: {   47: 'Fort Christina',
					  48: 'Nya Korsholm',
					  49: 'Norumbega',
					  50: 'Norumbega'},
			31: {49: 'Norumbega', 50: 'Norumbega'},
			32: {37: 'Sankt Thomas', 55: 'Markland', 56: 'Markland'},
			33: {56: 'Markland'},

Notice how I sorted everything visually with coordinates. I used a dictionary, which is like a list that doesn't have to be in any order (so I don't have to include every square). Sorry if you don't understand that, it's a handleful, I know. Plus, I don't know how much Python you know, so I don't have a good starting point. These things probably won't actually help with teaching you how to mod, but these things are good practices when trying to make a fast and efficient mod. For example, why store every plot when you can only store the ones needed? Also, the links I scattered around the post are great references; this is a great Python tutorial (that's where I learned it).

Hope that helps. Good luck with your modding! If you have any questions, or you want me to start you out with a "modding tutorial" or something, feel free to PM me. I've had a lot of free time lately.

 

[kairob]

on the AI wars you have changed the grid like you showed, but left it with "import pickle" rather than "import cpickle as pickle" like the other files, is this deliberate or can I change it?

 

[Úmarth]

Somebody suggested using dictionaries before, I forget why it wasn't implemented. Maybe Rhye concluded it wouldn't make that much of a difference.

 

[Metal Alloy Man]

Thanks Musicfreak, that was a very helpful tutorial as I prefer to implement changes myself rather than download them. I currently have very little python knowledge but I completely understood your tutorial.

Thanks.

 

[Fierabras]

Great stuff, musicfreak!

Very informative thread: pickle vs. cPickle (vs. marshal?), tuple vs. dictionairy

 

[musicfreak]

on the AI wars you have changed the grid like you showed, but left it with "import pickle" rather than "import cpickle as pickle" like the other files, is this deliberate or can I change it?

Oh, I did? My bad, yes you may change it.

Somebody suggested using dictionaries before, I forget why it wasn't implemented. Maybe Rhye concluded it wouldn't make that much of a difference.

That was me. Yeah he didn't think it was worth the work. Although I personally think it's much easier to maintain (just add the coordinates)...but maybe that's just me.

Thanks Musicfreak, that was a very helpful tutorial as I prefer to implement changes myself rather than download them. I currently have very little python knowledge but I completely understood your tutorial.

Thanks.

Good, I'm glad. I was afraid you wouldn't understand any of it, if you just started learning Python, but I'm glad it made sense.

Great stuff, musicfreak!

Very informative thread: pickle vs. cPickle (vs. marshal?), tuple vs. dictionairy

Thanks! About marshal...I've tried using it, but I get an EOF error every time...don't know why, but I'm guessing it has something to do with Rhye's change in the way script data is saved. The problem is, if I change it back, I also get an EOF error. I need to figure out what's causing it.

Thanks for all the replies, guys. I think I'm going to add the attachment to the first post, so it's easier to find.

 

[Wessel V1]

Absolutely amazing work man!
Extra bonus: I can now finally load the CityNameManager, it was too big to load it without a crash.

I've attached some more cPickle files, the game runs even faster now.

 

[Khoukharev]

thanks for this optimization service pack ~_^

 

[Fierabras]

Thanks! About marshal...I've tried using it, but I get an EOF error every time...don't know why, but I'm guessing it has something to do with Rhye's change in the way script data is saved. The problem is, if I change it back, I also get an EOF error. I need to figure out what's causing it.

I'm not an expert, but usually you get an EOF error when using a wrong loop direction:

while not EOF
take next record
do something

instead of:

do something
take next record
while not EOF

I've encountered this problem a lot over the years in web-scripting. About still having the same error when changing it back: maybe you have to do a fresh restart (clear cache?) of civ, otherwise you get old errors. It's something I noticed when changing Python, while keeping the game open.

 

[Úmarth]

Absolutely amazing work man!
Extra bonus: I can now finally load the CityNameManager, it was too big to load it without a crash.

I've attached some more cPickle files, the game runs even faster now.

It crashes IDLE but you shouldn't have any problem opening it in a normal text editor.

 

[musicfreak]

Absolutely amazing work man!
Extra bonus: I can now finally load the CityNameManager, it was too big to load it without a crash.

I've attached some more cPickle files, the game runs even faster now.

All those files are included in my attachment...check the first post.

I'm not an expert, but usually you get an EOF error when using a wrong loop direction:

while not EOF
take next record
do something

instead of:

do something
take next record
while not EOF

I've encountered this problem a lot over the years in web-scripting. About still having the same error when changing it back: maybe you have to do a fresh restart (clear cache?) of civ, otherwise you get old errors. It's something I noticed when changing Python, while keeping the game open.

Well I think the reason for it is Rhye's change to the way Civ stores script data. The original DLL uses the ReadString and WriteString functions, which should (theoretically) work without problems (although it doesn't seem to play well with RFC). Rhye changed it to a loop that first reads an integer (the size of the string), and then loops through once for every character and uses Read() to read in every character, one at a time. He commented the block as "bugfix", although I'm thinking that's the cause of the bug. However, like I said, when I change it back to how it is in the original DLL (using ReadString), it gives an EOF error no matter what the Python code actually does...

Basically, Rhye's "bugfix" works normally, without changing any of the Python code, but if I try to change any of the script data, it fails. However, without the "bugfix" (using the original method), nothing works at all, even if I don't change the Python code.

I have no idea why this is happening, because theoretically Civ should just be reading and writing a string, I don't see how the sizes could mismatch.

It crashes IDLE but you shouldn't have any problem opening it in a normal text editor.

For me it took ages to load in any editor with syntax highlighting, and I have a pretty decent computer. I imagine that on a slower one it would just hang. Notepad works fine but I have to have syntax highlighting.

 

[Riker]

With this modifications, the roman uhv works even if you lose one of your aqueducts? Maybe it's just sleep deprivation...