Love the Mod... A concern

[gunburned]

Just want to first state how much I appreciate everyone's work and I think this Mod has gone such a long way to give CIV 4 fundamental improvements in almost every way.
Thank you for this amazing MOD!
Over the years I've been playing Rise of Mankind, its addon A New Dawn, and now this great addition Caveman to cosmos. Its been constantly been adding great new content and additions to the game to improve it.
Now here is where my concern comes in.
In the last year I've started to notice significant load times especially upon entering the industrial era. I have a I7, 6GB Ram and 1GB video card, and play on the lowest settings and my turn times starting in that era average about 3mins (no larger than large maps).
Is my Rig too old, or is that most peoples turn times?
Has there been any thought that maybe too much content has been added and sacrificed too much performance?
I go though the game not even building 50% of some of the units from the main line of the tech tress. Maybe some buildings/units can be merged and or cut to improve preformance.

It makes me sad that I simply cannot bare the turn times anymore. Maybe its my Rig or maybe the game. Please take my post with consideration im shure that there are many more people like me here. If you cannot help me, I understand.
Thank you again for this most terrific game, easily top 10 in my Heart.

 

[Dancing Hoskuld]

Much has been done to improve turn times but as fast as they are improved more content is added that slows things down again.

Currently one of the team, Sparth, is going through as getting rid of duplicates in the graphics which should help somewhat in reducing the load on the machine.

 

[Thunderbrd]

Again... turn times can be greatly improved if you play a multiplayer game (use the IP connection) as a single player and allow simultaneous turns.

Due to some recent growing concerns regarding turn times (most of which is some really amazing AI programming eating up processing time) I've been considering going through and seeing how many ways I can improve processing. I recently did some things that apparently slow processing considerably and I'll be starting there.

Yes, a lot of assets do add to turn times but there's often ways that some good coding can improve on that more effectively than just taking those assets out.

 

[Nimek]

Again... turn times can be greatly improved if you play a multiplayer game (use the IP connection) as a single player and allow simultaneous turns.

I didnt know about that can you give a link to tutorial how to start that kind of game?
Internet connection is required or i can use localhost IP (127.0.0.1)?

 

[Thunderbrd]

You don't need any connection. Just go into multiplayer, then select Direct IP and hit the start game button and setup a game there. All you have to do is make sure you've set the AI players to be AI rather than Open and leave only yourself as the only human player when you start the game. Easy peasy. There's an additional option tab you won't be used to seeing for the multiplayer settings. Leave all that on default and you'll have simultaneous turns which is the trick to getting the game to run a bit faster. (But beware - in game the AI is taking their turns as you are so you might find they move still be able to move a unit after you move yours. You can turn this 'almost realtime strategy' into a positive however with some interesting defense tactics.)

 

[gunburned]

Nice to see you guys working soo hard all the time. Will all the performance improvements be implemented into the next patch?
I will try simultaneous turns, i was just always afraid I'd be annoyed with the AI moving with my moves.

 

[Thunderbrd]

No amount of optimization efforts will ever be complete. There's an ocean of workload we could address there and far more skilled individuals out there than myself at addressing them. I can make a slight impact here and there but nothing as powerful as Koshling had been able to now and then. It's an ongoing process to work on that to the extent that at least we don't lose much ground as we add more corrections and functions into the mix.

 

[gunburned]

Is it possible to do it to an already existing single player game? the option is greyed out.

 

[masaykh]

Here some useful info (imho) i got while resolving "too long load up".

Almost all time (16 minutes) game use cpu and almost not use hdd at all (this data rechecked using process monitor\process explorer and perfmon (windows 7 tool)). All work all 16 minutes done by cpu. My dual core cpu have 1Ghz per core, after i overclocked cpu to have 1.2 Ghz per core, load time decreased to 14 minutes. After 14 minutes 3-5 minutes going hard pressure on hdd to read all game data (in this thread you suggested that upgrade of hdd will improve situation but it isnt fully true). Game starting utilize only one core. In-game i see utilization of both cores.

You can state somewhere that to enjoy C2C comfortably, gamer should have cpu with minimum 2ghz per core or one-core cpu with such frequency. i think will be good to post somewhere such minimum hardware requirements.

And some small questions:
Can you supply python files with compiled versions (pyd files)?
it will reduce cpu work by some grade. I asking becouse game scan all folder for .pyd files so it can use them and looking for them.
Game scan all folders under mod folder for supported files. Can someone tweak or better replace such practice with better approach - make one list with all files and make game read such file?!

I hope you consider my proposotions.
it will reduce turn times!!

 

[Thunderbrd]

Here some useful info (imho) i got while resolving "too long load up".

Almost all time (16 minutes) game use cpu and almost not use hdd at all (this data rechecked using process monitor\process explorer and perfmon (windows 7 tool)). All work all 16 minutes done by cpu. My dual core cpu have 1Ghz per core, after i overclocked cpu to have 1.2 Ghz per core, load time decreased to 14 minutes. After 14 minutes 3-5 minutes going hard pressure on hdd to read all game data (in this thread you suggested that upgrade of hdd will improve situation but it isnt fully true). Game starting utilize only one core. In-game i see utilization of both cores.

You can state somewhere that to enjoy C2C comfortably, gamer should have cpu with minimum 2ghz per core or one-core cpu with such frequency. i think will be good to post somewhere such minimum hardware requirements.

And some small questions:
Can you supply python files with compiled versions (pyd files)?
it will reduce cpu work by some grade. I asking becouse game scan all folder for .pyd files so it can use them and looking for them.
Game scan all folders under mod folder for supported files. Can someone tweak or better replace such practice with better approach - make one list with all files and make game read such file?!

I hope you consider my proposotions.
it will reduce turn times!!

I've read your proposition before and was hoping someone else would comment. What you need to understand is nobody on this team (with its present and not-distracted by other realms of life roster) currently understands any of this suggestion enough to implement it. We would welcome your assistance if you do.

 

[Dancing Hoskuld]

Can you supply python files with compiled versions (pyd files)?
it will reduce cpu work by some grade. I asking becouse game scan all folder for .pyd files so it can use them and looking for them.
Game scan all folders under mod folder for supported files. Can someone tweak or better replace such practice with better approach - make one list with all files and make game read such file?!

I hope you consider my proposotions.
it will reduce turn times!!

1) I am not sure if Civ IV is set up to use compiled python or not.

2) Both the Python and XML files are modular.

Some are only modular because one of us is working on it so having it separate makes it easier to not step on each others toes. Those modules will eventually move into the core area.

Some are modular to make it obvious where we got stuff and sort of give credit where it is due beyond that in the Credits Pages. These could probably be merged in unless they are optional.

Others are modular because they are optional and not used by everyone in every game. If they are not used then usually they are not read either.

3) We (well AIAndy and Alberts2 mostly) have replaced the reading of the XML files with our own system which stored the result of reading the XML in the cache files. This means that the first load is slower but that subsequent loads will be faster, if there have been no changes to any XML files.

 

[masaykh]

i found some clues:

python have extenision py_compile that can make compiled bytecode.
base python distrib used in game stored in bytecode (.pyc)

if maximum of .py files will be stored in .pyc some load times will be cutted off.
i tried to invoke it in in-game console - but as i see it can operate only with supplied commands and not full python.

i dont see why game dont create .pyc by default - but if we force it to ceate it - will be good)

 

[Harrier]

Yes, a lot of assets do add to turn times but there's often ways that some good coding can improve on that more effectively than just taking those assets out.

I started programming in the 1960s (machine code). In those days you had to consider the values you would probably come up against, in a program.

i.e. A * B. BUT should it be B * A ? The end result is the same, but the computing time - very differant. (In those days we had only a few Kilobytes of memory, and had to be memory efficient)

If A = 5 and B = !00.

A * B means 100 additions of 5. (100 opperations)
B * A means 5 additions of 100. (5 opperations) WOW - much faster.

Unfortunately modern programming languages, allthough more efficient in many respects , (and probably easier to program ) they seem to ignore the core basic rules of programming.

So know matter how good a programmer you are, you are let down by the program language you use.

 

[Thunderbrd]

I started programming in the 1960s (machine code). In those days you had to consider the values you would probably come up against, in a program.

i.e. A * B. BUT should it be B * A ? The end result is the same, but the computing time - very differant. (In those days we had only a few Kilobytes of memory, and had to be memory efficient)

If A = 5 and B = !00.

A * B means 100 additions of 5. (100 opperations)
B * A means 5 additions of 100. (5 opperations) WOW - much faster.

Unfortunately modern programming languages, allthough more efficient in many respects , (and probably easier to program ) they seem to ignore the core basic rules of programming.

So know matter how good a programmer you are, you are let down by the program language you use.

You would think with C++ that the auto-compiler would determine the best configuration in situations like these (since it encodes everything anyhow.)

Wow... that kind of complexity of consideration would have to go way back. I remember the FIRST programming I ever did was on an Apple C+. Using Basic. And yeah, it didn't take all that long (though it seemed like miles of code to us at the time) to run the system right out of memory.

Most optimization in the current dll can take place by limiting the looping procedures to going through only what is pertinent to evaluate. In MANY cases these are done rather carelessly - and I would've never known anything about this if not for Koshling and AIAndy, both of whom are FAR better at optimizing than I.

 

[Harrier]

You would think with C++ that the auto-compiler would determine the best configuration in situations like these (since it encodes everything anyhow.).

C++, is a good language (for current usage - the best for programmers) though I know the basics - being retired I never studied it in depth.

It's modular concept revolutionised programming. But an "auto-compiler" (using your words) can only reduce to machine code what you tell it. It does not optimize.

 

[Harrier]

You would think with C++ that the auto-compiler would determine the best configuration in situations like these (since it encodes everything anyhow.).

It does not, it just translates what you tell it into machine code.

It's big benefit is that it makes difficult programming projects more feasable, and their modification much easier.

Year's ago you would have to start from the begining each time. Now you just add on and modify. As with C2C.

Most optimization in the current dll can take place by limiting the looping procedures to going through only what is pertinent to evaluate. In MANY cases these are done rather carelessly - and I would've never known anything about this if not for Koshling and AIAndy, both of whom are FAR better at optimizing than I.

The looping process is as I stated. A * B, or B * A.

Computers can not multiply or divide. All thet do is repeat additions or subtactions.

So 3 times 2, is infact either ( 3 added 2 times) or ( 2 added 3 times). as per my previous post.

 

[Thunderbrd]

Well... yeah that makes sense.

You can also prefilter A or B to make sure that it's even necessary to run through those items, then apply that prefiltering wherever such items are looped. That seems to provide nice gains.

Koshling has been an absolute caching master - unfortunately in many places this is very dangerous for OOS errors in multiplayer mode. And for the life of me I suck at following the caching unless I'm VERY patient to figure out how it's setup and works. In most cases, imagining doing it myself is somewhat baffling.

 

[God-Emperor]

Harrier, you have multiple factual errors in your posts when it comes to modern computers (anything since the 1980s, really). Here are two:

1) Computers do multiply. It is not done via a series of additions, as such. It has been this way since the late 1970's for the most part. At least some of the computers in the 1960s could do actual integer multiplications, I think. These days the actual algorithm is generally still basically a "shift and add" type scheme (like the basic partial products method used to do multiplication on paper, which is very quick and easy in base 2), but it has some refinements and is done in a single piece of multiplier hardware which does it all in a very small number of steps, it doesn't use multiple calls to the regular "bit shift" or "add" hardware. Your basic current generation Intel CPU has a 32 bit integer multiply (IMUL r32) with a latency of 3 clock cycles (used to be 10, but they updated it back around the Core 2 Duo time frame - it probably takes 4 times as many transistors to make it take 1/3 the time, although they may have also switched to a different algorithm) and a throughput of 1 (it takes 3 cycles to get any specific operation done, but has to wait only 1 cycle before the same instruction can be issued to the unit again - fun with pipelining). It doesn't matter what the operands are, it always takes that long. This is why the CPUs have so many transistors: so it can do a lot of things at the same time (like calculating all of the partial sums at the same time) resulting each operation taking very few clock cycles compared to what it used to take. Here is a place to start, if you want to know more about this.

2) Compilers do, in fact, do a lot of optimization. They rearrange code quite a bit, unroll loops, and at least a dozen other things. One such optimization is that multiplying by 2, or probably any power of 2, is typically converted to a bit shift operation since that takes only 1 clock cycle to do instead of 3 (or whatever it takes your CPU to do multiply - the last few generations of Intel CPUs are very efficient in terms of cycles per operation, AMD's tend to take more but can run at higher clock rates). They also can "inline" code from functions in order to eliminate the function call (C++ has a keyword you can add to a function declaration that tells the compiler that this is a good candidate for inlining, but it is just advisory), and I don't even know what all else they do these days. The optimizing compiler page at wikipedia.