PerfectMongoose (PW3 Civ4 Port)

LunarMongoose said:

Okay. Well like I said, I'm 90% sure that's just how PW3's rainfall system works. Ceph made a big deal about mountains creating "rain shadows" when he wrote it, among other effects, and this was heavily discussed in the Civ5 PW3 thread.

I did specifically allow PM to use the PW2 climate system (temperature and rainfall) with the PW3 terrain system (better-looking landmasses) because I recognized the major difference with the new climate system, and I knew that if I might want the old system, then others might too, heh.

But I don't think it's anything I did wrong in the port.

Click to expand...

After extensive debugging I found the bugs:
1.) Integer division instead of floating point division in the latitude from y calculation
2.) Only 6 monsoon neighbor plots (which is Civ5 hex setup) instead of 8 neighbor plots (as it should be in Civ4)
3.) Geostrophic rain deactivated (was commented out but disfunctional with the other bugs anyway)
4.) Bugs in the plot list generation for geostrophic rain (sorted geo map)

I have attached the file I debugged it on but since it is an adaption for C2C it has several other changes as well.

Awesome, thank you very much! I'll try to post an updated version in the next few days.

This would explain why it seemed to mostly work, but I had to change the terrain type threshold values to get the best results.

Hopefully there's nothing wrong with the landmass generator code. I'll double-check a few things based on these bugs.

p.s. -- Sorry for screwing up, lol. I hate making mistakes. It's harder to avoid when you're working on someone else's code, and you understand most of how it works but not most of what it's doing. At least I didn't claim I was certain there were no bugs. Thanks again for your help, I'm really glad to be able to improve this.

Actually it seems the landmass generator has the same hex neighbor problem. It does not use the right part of the perlin noise which definitely changes the way landmasses look, a lot.

I have also done some profiling and it seems that both bonus placement and starting plot finding is very inefficient and claims the majority of the time it takes to create a map, especially with a mod that adds a lot of bonuses, features and improvements.
Every single time it calculates the value of a plot (and it does that not only once but a lot of times for the same plot), it analyses the technology of the mod (feature removal, improvements buildable). That is very expensive as it is done with a loop through all feature and improvement infos.

AIAndy said:

Actually it seems the landmass generator has the same hex neighbor problem. It does not use the right part of the perlin noise which definitely changes the way landmasses look, a lot.

Click to expand...

Heh, yeah I kinda figured. I'm just wall-to-wall busy at the moment with my parents visiting and us doing a lot of misc work on the condo. I'm hoping to be able to work on this very soon.

I really like how the landmasses look as it is though, so maybe I'll leave the current version as an option lol.

AIAndy said:

I have also done some profiling and it seems that both bonus placement and starting plot finding is very inefficient and claims the majority of the time it takes to create a map, especially with a mod that adds a lot of bonuses, features and improvements.

Click to expand...

Unlike the previous issues, this is definitely not my fault, haha. I do know PW3 already generates maps a lot faster than PW2 did, but if the speed can be further improved that'd obviously be good. (I think Ceph may have even mentioned this somewhere before... not sure, my memory's a little fuzzy on it.)

Once again, a deep and heartfelt thanks for your work on this, btw. I'm not a debugging expert. I should be, but I'm not. (I've always been able to fix problems the hard way, by reading the code til I realize what's wrong. This mapscript was one of the first times I'd ever worked on something this complex, that'd also been written by someone else originally. Oh well, a little public embarrassment is good for the soul. )

My memory of this stuff is very fuzzy since its been years since I looked at PW2, but the value of a land plot is very relativistic. For example, if you have a cluster of gold resources way out in the desert, you can't give them all the same value because the available food might only let you work 1 or 2 of them. Also, it can happen that a resource is actually on a different continent than the city site being evaluated. It's all in the context of potential city building.

It would be cool if you guys could improve the speed, but I'll warn you that the problems with starting plot finding on a map like this are non-trivial. I never thought much about efficiency because I figure if I'm willing to spend 10 hours playing Civ I could handle waiting 10 minutes for the map.

When you create the 100th map without playing to find bugs and look how the whole bunch of extra terrain looks like you tend to start to like efficiency (Although I guess you went through all that as well).
And the way the starting plot assignment is done is good and I don't plan to change that.
Profiling just showed me that on a standard size map the computer evaluated a plot over 50000 times. So each plot was checked over 50 times on average and every single time there is a loop over all features and improvements. That might not be much in standard Civ4 but I adapted it for C2C which has loads more.
So I have already done some plot value caching but a good part of that city and plot value checking is done in the loop that adds extra resources to city spots and that loop actually changes the value of plots so simple caching is not enough. Some incremental change of cached plot values would be needed here.
Simplifying the mod technology analysis and moving it out of the plot value function would also reduce the amount of time needed a great deal. Not entirely simple though if you want to store that information efficiently for use in the plot value analysis.

That reminds me that I still need to check why in my changed Mongoose mod the starting plot finder tends to fail nearly every single time (might be because of technology analysis issues).

AIAndy said:

That reminds me that I still need to check why in my changed Mongoose mod the starting plot finder tends to fail nearly every single time (might be because of technology analysis issues).

Click to expand...

Interesting. I wouldn't mind knowing that myself, heh. I've noticed something fails with PM in unmodified-MM like 10-20% of the time, which wasn't enough of a problem to force me to look into it, but it's probably this.

p.s. - Hi Ceph! *hugs*

LunarMongoose said:

Interesting. I wouldn't mind knowing that myself, heh. I've noticed something fails with PM in unmodified-MM like 10-20% of the time, which wasn't enough of a problem to force me to look into it, but it's probably this.

p.s. - Hi Ceph! *hugs*

Click to expand...

Thinking about it the reason is probably that in MM (modified and unmodified) the removal of forest and most improvements are not in the starting era.

Wow, 50000 times for one plot? I don't remember what I was doing, but I would have never imagined so many. That's gotta be a bug. The most I would expect would be once for each city location that contained the plot in the fat cross. Inefficiencies might double that, but 50000 seems way way off.

cephalo said:

Wow, 50000 times for one plot? I don't remember what I was doing, but I would have never imagined so many. That's gotta be a bug. The most I would expect would be once for each city location that contained the plot in the fat cross. Inefficiencies might double that, but 50000 seems way way off.

Click to expand...

A profiling on my current version (starting plot finding only):
4024193 function calls in 52.129 CPU seconds

ncalls tottime percall cumtime percall
21711 0.155 0.000 0.221 0.000
C2C_PerfectMongoose_v310:4101(getPlotPotentialValue)

39074 36.417 0.001 47.659 0.001
C2C_PerfectMongoose_v310:4112(getPlotPotentialValueUncached)

6 0.209 0.035 45.169 7.528
C2C_PerfectMongoose_v310:4208(boostCityPlotValue)

That is already with caching of the plot value except in the boost city part when plot value is increased.
So 60000 calls to evaluate a plot in total, the majority called by boostCityPlotValue. For some reason it evaluates 5000 plots each of the 6 times it boosts a city.

I found the reasons.
1) The mod had the bNormalize flag set for very few resources so the loop that sets the bonuses ran all the way to the end usually with loads of city and plot evaluation without actually setting a bonus.
2) The mod has a city radius of 3, but not at the start. Still the code returned the city plot number at 37 instead of the 21 it should be. So a lot more evaluation per city evaluation.

Fixing both of that (in addition to the caching) reduced the time the starting plot finding to few seconds.

Another thing I noticed in my original code is that when boosting plots I call getCityPotentialValue for each plot within a city site because the loop in question might change the value of the city. I'm sure there's alot of room for improvement there.

cephalo said:

Another thing I noticed in my original code is that when boosting plots I call getCityPotentialValue for each plot within a city site because the loop in question might change the value of the city. I'm sure there's alot of room for improvement there.

Click to expand...

Indeed, that is what I thought about when I first looked at it. Caching it in an incremental way (meaning the placing of the bonus changes the cache value but does not do a full reevaluation). But with the changes it tends to run a LOT less so the improvement is not really needed any more. The run time of the starting plot finding is now down to very few seconds.
Next I rather look at the bonus placement.

AIAndy said:

1) The mod had the bNormalize flag set for very few resources so the loop that sets the bonuses ran all the way to the end usually with loads of city and plot evaluation without actually setting a bonus.
2) The mod has a city radius of 3, but not at the start. Still the code returned the city plot number at 37 instead of the 21 it should be. So a lot more evaluation per city evaluation.

Click to expand...

Heh, oops. However turning normalization off was intentional... I didn't want players getting any more resources than the basic terrain would specify, if possible. Which ones did you turn bNormalize back on for? Just curious.

LunarMongoose said:

Heh, oops. However turning normalization off was intentional... I didn't want players getting any more resources than the basic terrain would specify, if possible. Which ones did you turn bNormalize back on for? Just curious.

Click to expand...

Actually that is a misunderstanding. Your mod has that flag set for many bonuses. It was the C2C mod I was profiling on which has that flag set on very few bonuses.

Perfect World 2 and Perfect mongoose are great scripts. AIAndy thanks for going back over them and tweaking them out.

I play on the 60x40 map sizes pretty regularly (almost exclusively), and use these 2 scripts. They create some awesome worlds!

Can't wait to see how it works when you are done Andy

Sorry if I post this in the wrong spot. I'm a tech dweeb and not sure how mapscripts work. I just downloaded the PW3 Mongoose, is there anywhere that tells me a step by step on how/where to put these files? I normally run games with BAT, LoR, or RevDCM and I've got those set up. Just trying to make sure if these scripts work there. Is there a read me or something? I got the mod loading thing down, just have never done anything with maps or mapscripts. Thanks for any help.

Good Evening,
is there a way to get updated map script mentioned here? I mean the one with bug fixes coded by AIAndy.
Thank you very much for your work, really.

And said:

Good Evening,
is there a way to get updated map script mentioned here? I mean the one with bug fixes coded by AIAndy.
Thank you very much for your work, really.

Click to expand...

I fear I never applied them to normal BtS.
The main version I maintain is in the Caveman2Cosmos mod and contains quite some extra code for the new terrain and everything.
I have also made a version for the MongooseMod which contains most of the fixes (I use that one for some multiplayer games with friends). It should not be too far off from a version for normal BtS (but still requires some removal of code). I'll attach it.

Oh, I see, I guessed I could use it with normal BtS... Maybe LM will be able to update it one day. Thank you very much for the quick answer!