PerfectMongoose (PW3 Civ4 Port)

Serious, major, heavy-duty apologies for the delay - Real Life™ kinda eggsploded on me the last 6 months. (Even though I still don't have a job, it's amazing how busy I've been... sigh.)

I am eggspecting to be able to work on Civ stuff again starting this weekend (and next week at the latest if I'm wrong, which I usually am, lol), and this mapscript update is first on the list.

Btw, thanks for making an update for MM already, Andy; I didn't anticipate that, and it'll make doing the plain BtS version a lot easier. I was gonna do some other things too though, in terms of merging the two files in my download so all the generator options are in one script, and add an option for using the existing "broken" hex-square evaluation even after I apply your fix, since I really like it the way it is.

Looking forward to the update!

Tholal said:

Looking forward to the update!

Click to expand...

Whelp, as you might easily have guessed, I was right about being wrong, lol. I really am almost done now though. I screwed up my big Black Friday Zazzle order which took a whole extra day to fix, people keep buying the stuff I have listed on Amazon which takes time to package and carry over to the post office every day, AND I've been eating too much lately which, in turn, makes me sleep more than usual. (Not to mention I've had to keep a close eye on the stock market given how volatile it's been lately.)

... That was in case you cared about the details, which I doubt... though I'm telling you anyway.

Should be starting this tomorrow afternoon (or possibly tonight if my Brain Gerbils spin their wheels really fast), and I don't really expect it to take more than a couple full days to do. The usual disclaimer still applies though - I am terrible at making time predictions on my own work. Just saying. Sorry again for the delays, and thanks for your interest.

IT! ... HAS! ... BEEEGUNNN!!!

/end Mortal Wombat Announcer

Okay so I'm another 4 days late. I stopped to watch the whole golf tournament since Tiger was, you know, mostly not sucking for once... Then I discovered a wonderful tv show (The Big Bang Theory) which was on during ad breaks in the golf... Then the stock market continued to require my attention Monday. Umm... Blame Europe, not me.

Anyway, I'm officially working on the PM update now - gimme a couple days. A MongooseMod update should follow sometime in the next 6 weeks, since I have to get that done before Diablo 3 comes out on January 17... But at least the real-life pressure is reduced and I'm feeling relatively happy and unstressed now, finally.

('Course I'd be a lot HAPPIER if I weren't single, but any potential gf would have to be Dying for Diablo(tm) as a prerequisite, so what can ya do... I know, I'm hopeless.)

Alright, I'm pretty sure I'm rambling again, so I'm going to stop typing now. Feel free to continue on with your lives.

Quick update:

I've gone through all of AIAndy's code, and made all his quick obvious bugfixes. I also found one additional place where there was an int that was supposed to be a float, though I don't know if it actually matters in this case. I've added the new option to use the old hex tile evaluation math in the landmass generator if the user wants to. @AIAndy: Really clever idea to use the sync'd DLL function to seed the Python random generator so it can still be used in multiplayer games!

I'm just getting into Andy's rainfall fixes and feature / bonus placement code changes. The latter was just for efficiency and the changes are extensive, so I could skip it if I had to, but I guess I'll muck through it.

Then I still have to merge in the PW2 landmass generator. It's gonna be a couple more days before this update is ready, but I'm working on it!

Edit - @AIAndy again: I'm curious about the drainList North/South swap you made in generateRiverMap, cuz that should've been copied verbatim from PW2, but for now I'll assume you're right. Also, I'm pretty sure the gc.getNUM_WHATEVER() calls you replaced with literals are wicked fast - they're mapped to one-line DLL functions that read "return self.constant;" basically. So I'm not sure you're saving more than a few microseconds per pass with this, but I understand why you did it - they're obviously in the heart of the beast as far as the loops go.

Edit 2 - Yep, you were right, but it doesn't matter: generateRiverMap() assumes wrapX is true and wrapY is false, so if the map settings don't match that, it will get into trouble b/c GetIndex() will return -1 for out-of-bounds queries. I've changed all four drainList neighbor checks to check for ii being -1. This will allows rivers to link on the top and bottom of a Toroidal map, for example, and avoid an averageHeightMap[-1] query on a Flat map, for example. Though the latter should be causing a crash and it isn't, which is weird...

LunarMongoose said:

... and avoid an averageHeightMap[-1] query on a Flat map, for example. Though the latter should be causing a crash and it isn't, which is weird...

Click to expand...

Negative array indexes in Python are valid. They count back in from the end of the array, so if array A has items 0 through 9 then item -1 is item 9, -2 is 8, and so on. (Item "-0", if you can get such a thing, is still item 0.) Essentially it is the same as having wrap in the coordinates of a map.

God-Emperor said:

Negative array indexes in Python are valid. They count back in from the end of the array

Click to expand...

Ahhh. Well I'd about concluded they had to be valid since it wasn't causing a crash, but I assumed they just returned a value of -1 or something. Figures it'd be some silly safety mechanism for a non-compiled scripting language lol. Also goes to show my limited experience with scripting languages. Anyway, thanks for the info! (My ii -1 checks will still work though cuz they're a simple way to allow x and y wrapping when and only when the map allows them.)

Progress-wise, I've made some more very minor code optimizations in several places, and turned on the geostrophic system (which was probably disabled b/c I guess I disabled it in the original port when I couldn't get it to work due to the other bugs... and then I promptly forgot about it b/c I didn't understand how important it was... oops). I need to run some tests and be sure I'm happy with the fine-tuning of the terrain percent constants now that the system is actually working properly.

I'm also doing some testing on various combinations of settings for the new "messed-up-math hex evaluator option", which it turns out is even more broken in PM 3.1 than Andy and I thought (b/c range(1, 7) includes the N and S directions that GetNeighbor() ignores, so it wasn't even evaluating all 6 hex directions lol).

Then I have to muck through Andy's feature / bonus rewrite, and merge in PW2. Still gonna be a couple more days.

Making good progress. I've finished merging in Andy's code, and am most-of-the-way done with tweaks and adjustments and checks. I also found one nasty little copy-paste-type bug in makeHarbor(), which is from the original PW 2.0.6 code so all versions of this mapscript have had the problem for a long time, heh.

I still need to merge in PW2's LMG and then convert the file for vanilla BTS, and my parents are arriving tomorrow which will get me busy with real-life stuff again, but I'm still gonna try to finish this update over the weekend... Keep your feathers crossed.

@AIAndy:

I disagree with you removing the "no bonuses of different types in adjacent tiles" rule; this has always been a standard rule in Civ4, and is present in the DLL's default bonus placement code if a mapscript doesn't provide any, so I'm leaving it in.

All the MongooseMod feature removal Builds (Chop Forest, Chop Jungle, Dredge Marsh, Scrub Fallout) have no tech prereq (b/c their tech prereqs are in the feature-removal subsections only). However I don't need to consider them in getPlotPotentialValueUncached(), b/c Cottages will be available in the minimum era range (Classical), and those plus Mines can go anywhere that Jungles and Marshes can (and any improvement is automatically better than just a chop). SO, I'm leaving this line of code the way it was as well, since skipping over the chop builds is another slight speed boost.

My generation time with a Huge map on PM 3.2 averages around 4min 10sec, whereas on PM 3.1 it's around 5min 20sec. Thus, I love your caching. Taking into account your info about most of the time being spent in getPlotPotentialValueUncached() however, I'm going to try exposing the DLL's version of this function to Python and calling that instead in the MM Edition. I don't remember if you can run all the DLL's code from mapscript Python, but if it works it should be a lot faster. More importantly it's also a much better version of the function computationally, as it's a lot longer and includes a lot of BetterAI work by Fuyu and others. So it'd be worth using anyway.

LunarMongoose said:

@AIAndy:

I disagree with you removing the "no bonuses of different types in adjacent tiles" rule; this has always been a standard rule in Civ4, and is present in the DLL's default bonus placement code if a mapscript doesn't provide any, so I'm leaving it in.

Click to expand...

For MongooseMod that is fine. I mainly decided to remove it to make bonus placement easier for C2C as it has a lot more resource types so placement fails to find proper spots often for some bonus types.

My generation time with a Huge map on PM 3.2 averages around 4min 10sec, whereas on PM 3.1 it's around 5min 20sec. Thus, I love your caching. Taking into account your info about most of the time being spent in getPlotPotentialValueUncached() however, I'm going to try exposing the DLL's version of this function to Python and calling that instead in the MM Edition. I don't remember if you can run all the DLL's code from mapscript Python, but if it works it should be a lot faster. More importantly it's also a much better version of the function computationally, as it's a lot longer and includes a lot of BetterAI work by Fuyu and others. So it'd be worth using anyway.

Click to expand...

If calling that function is fine from C++ at that point, then calling it from Python will work as well.

@AIAndy again: I'm curious about the drainList North/South swap you made in generateRiverMap, cuz that should've been copied verbatim from PW2, but for now I'll assume you're right.

Click to expand...

Iirc that code had north and south confused and did not prevent wrapping in y direction as it should which could then later cause an infinite loop in river placement.

I'm almost done with the update, just got swamped with other stuff to do the last few days. Since my last post I've improved sea ice placement significantly, and finished testing and tweaking everything... I just need to finish the PW2 LMG merge and adjust the file for vanilla BTS. If I can get some more blocks of time to work on it it'll be done pretty soon.

AIAndy said:

For MongooseMod that is fine. I mainly decided to remove it to make bonus placement easier for C2C as it has a lot more resource types so placement fails to find proper spots often for some bonus types.

Click to expand...

Yep, I kinda figured that might be why.

AIAndy said:

If calling that function is fine from C++ at that point, then calling it from Python will work as well.

Click to expand...

Right... I just know there were some restrictions on calling DLL functions in some places (like the Civ4 main menu areas, and the initialization phase of a new game right after the mapscript finishes running) and I didn't remember for sure if this was one of em.

AIAndy said:

Iirc that code had north and south confused and did not prevent wrapping in y direction as it should which could then later cause an infinite loop in river placement.

Click to expand...

Yep, I said in "Edit 2" that you were indeed right, but it turned out to not matter cuz I had to rewrite that code anyway. Good find on the north/south lines being mixed up though; it got me thinking about all the edge-checking code in the script.

LunarMongoose said:

Right... I just know there were some restrictions on calling DLL functions in some places (like the Civ4 main menu areas, and the initialization phase of a new game right after the mapscript finishes running) and I didn't remember for sure if this was one of em.

Click to expand...

You mainly have to be careful that everything the function you call uses has been initialized. In case of doubt, add the call and use a debug DLL to see if the C++ code runs fine.

Toldja not to trust my time estimates of my own work. Sorry (again); whenever my parents are here we get a lot of work done on the condo, so I got swamped this past week (again).

I finally got to work on this again today. After much frustrating effort that felt like trying to attach a classic car's body to a modern car's chassis (which was a cool IWCC episode!), I have successfully forced the PW2 landmass generator to live, work, eat and play in my PM3 code environment.

I had to write some kludgy code to get it working, which I need to go back and update now that I trust it. I also need to convert it to vanilla BTS, and check a few final things, but there's a good chance it'll be ready later tonight... gimme a few more hours.

I'm actually not going to bother with the code cleanup I mentioned... It'd take too long, I'm a little burnt out on this thing at the moment, and it works fine the way it is. Besides, there's a lot more redundant functions to consolidate than just the new PW2 LMG stuff. I might do it in a minor future update, but for now just be aware there's an annoying "if mc: em = e3, else: em = e2" assignment at the top of almost every function, lol.

Since my last post I've further improved floating sea ice placement beyond what I did before (it's really awesome now), further tweaked the climate constants, made a speed optimization of my own (very minor, but it takes two lines out of a loop like Andy did in another spot, so I'm proud of it ), updated the version history for 3.2, and rewrote my feature placement code (separately from Andy's rewrite of it, but with the same effect - fixing a bloated mess).

At the moment I'm tackling a problem I discovered yesterday, which is that the PW3 climate system yields temperatures much higher than they should be when used with the PW2 landmass generator, and I haven't figured out why yet. (The other combinations all seem to be working properly: 3 LMG + 3 climate, 2 LMG + 2 climate, and 3 LMG + 2 climate as was available in PM 3.1.)

So anyway, if I can get it fixed tonight or tomorrow I will, otherwise I just need to remove the MongooseMod stuff and it'll be out the door. (At which point Ceph and Andy can figure out the problem for me. )

After 2 weeks of on-and-off work, followed by a week of 12-to-16-hours-a-day work, followed by a 20-hours-of-work-in-a-single-day day, followed by 12 more hours of work today... Version 3.2 has been posted.

This file has now been heavily tested, and I am very confident everything is as it should be. There are several major improvements in this update, and I even managed to further reduce the time the script takes to run beyond what Andy did (from 4min 10sec to 3min 30sec on average on my computer) with some of my most recent code (though at the moment I'm not sure exactly which changes caused this, lol).

I missed Christmas by 2 days, but I only missed the one-year anniversary of the PerfectMongoose 3.1 release by 5 hours and 27 minutes, so I feel like I did okay. Enjoy your holiday present guys!

p.s. - I am burnt out to hell with working on this thing, and completely drained mentally and exhausted physically. PLEASE don't find anything wrong with this file, or think of anything you would like me to change or add, for at least another 9 months. Thanks.

p.p.s. - As promised I will be resuming my work on the also-much-delayed MongooseMod 3.2 update soon, so the MME version of PM 3.2 will be released later with the rest of the mod. I've already made several mod-specific improvements in it, like Marsh placement that is rainfall-sensitive instead of random.

awesome work man, you deserve some rest while we players reap the rewards of your toil

[to_xp]Gekko;11156373 said:

awesome work man, you deserve some rest while we players reap the rewards of your toil

Click to expand...

Thank you, I'm glad you like it. I may actually release a 3.2.1 update here in a bit btw... In addition to getting a lot of sleep, stopping to relax, and noticing how much I missed out on in the stock market over the last 2 weeks (gah, srsly?! grr...), I've updated Oasis placement to be rainfall-sensitive (and to have a slightly wider minimum spacing), and changed Jungle (and Deciduous vs Evergreen Forest while I was at it) temperatures over from absolute values to being percent-based (like I already did with Snow and Tundra in v3.2).

The latter change may be somewhat controversial since it will create jungles at higher latitudes than normal if necessary, but I've decided I really like having consistent amounts of every type of terrain on the map each time, including Jungle, even if it happens to create a world with no major landmasses in the tropics. It's good for gameplay balance, and it's not really THAT different from how South America is on Earth anyway. Feel free to debate this though.

I have tested the new script, the results are awesome. Thank you

so far my experience, on a standard size map, pw3 settings seem to generate alot of deserts. The pw2 settings seem more balanced.

by the way i love the pw seies of maps

modifieda4 said:

so far my experience, on a standard size map, pw3 settings seem to generate alot of deserts. The pw2 settings seem more balanced.

Click to expand...

Nope. In 3.2 they will both generate exactly 18% of total landmass as Desert. I'm not saying that JUST b/c of my code; I also wrote some diagnostic code that counts up each type of terrain at the end and prints the actual percentages to the debug text file. I ran it a whole bunch of times on different sizes, and the values were always within 1% or so of what they were supposed to be.

I left it in, actually. You can find it at the bottom of "def generatePlotTypes()"; just uncomment it and enable Python logging.

I'm getting errors when trying to use this with Master of Mana

hopefully this can be fixed easily cuz I'd love to use the script with MoM