Modder's Documentation

Not exactly a modding question, but I figured this would be the best place to ask. If you prefer the question elsewhere, just say so and I move the post.

Anyway, I'm currently trying to merge your brilliant graphical paging feature into my mod. Unfortunately, I have encountered some problems:

- moving around on the game map a lot can lead to a memory allocation failure ("bad allocation")
- when moving the viewscreen by moving the mouse at the edges of the screen, tiles are not revealed as soon as they get into view. This is not a loading time issue, nothing happens until they've been further moved into the center of the screen.

So I'm wondering where I've made a mistake or left something out. I mostly copied the following methods over from the C2C sources:

CvGame
- m_iLastLookatX and m_iLastLookatY
- m_iXResolution and m_iYResolution
- m_bWasGraphicsPagingEnabled
plus getters/setters, initialization and serialization where appropriate

CvGlobals
- m_bGraphicalDetailPagingEnabled plus getter/setter
- getGraphicalDetailPageInRange()

CvPlot
- EvictGraphicsIfNecessary()
- pageGraphicsOut()
- notePageRenderStart(int iRenderArea)
- setShouldHaveFullGraphics(bool bShouldHaveFullGraphics)
- shouldHaveFullGraphics(void) const
- void destroyGraphics()
- inclusion of shouldHaveFullGraphics() checks into updateSymbolDisplay(), updateSymbolVisibility(), updateSymbols(), updateCenterUnit(), updateFeatureSymbolVisibility(), updateFeatureSymbol(), updateRouteSymbol(), updateRiverSymbol(), updateFlagSymbol() and updatePlotBuilder()
- pageGraphicsOut() in the CvPlot destructor

I did not copy over the shouldHaveGraphics() method, because it seems to be related to the viewports feature. Instead, I changed shouldHaveFullGraphics() to the following:


Can anybody help me out here? What did I do wrong?

You'll have to PM Koshling on that I think. He's not likely to see this post and he's the only one who understands the magic of the graphics paging.

Magic it is Didn't know who of you did the work on this, I'll PM him right away, thanks!

Leoreth said:

- moving around on the game map a lot can lead to a memory allocation failure ("bad allocation")

Click to expand...

I don't get this but I do occasionally get CTDs moving around the map or clicking on the minimap which I suspect are graphics-paging-related. Then again, playing C2C I save the game a lot more often than on vanilla (I actually aim for every half-hour of play), and I even restart the game before hitting end of turn every time (mind you a turn in Industrial is a good 12 hours of playing time for me and may take days).

Leoreth said:

- when moving the viewscreen by moving the mouse at the edges of the screen, tiles are not revealed as soon as they get into view. This is not a loading time issue, nothing happens until they've been further moved into the center of the screen.

Click to expand...

This is normal behaviour for graphics paging. Of course I'm no expert but I'm thoroughly used to seeing it working that way. Is that a problem?

I believe that once they've been revealed, they stay that way or at least reveal more easily for the rest of the session, but perhaps not.

Yudishtira said:

I don't get this but I do occasionally get CTDs moving around the map or clicking on the minimap which I suspect are graphics-paging-related. Then again, playing C2C I save the game a lot more often than on vanilla (I actually aim for every half-hour of play), and I even restart the game before hitting end of turn every time (mind you a turn in Industrial is a good 12 hours of playing time for me and may take days).

Click to expand...

Oh wow. Turn times in DoC aren't nearly as bad, I had actually hoped to get away from the 1-2 an AI turn can take in the late game which is often already frustrating.

The MAFs in DoC are definitely the result of the graphical paging (never had this problem before), although I cannot say if the fault is in my implementation or not. If it's a general issue of the feature maybe we can isolate it easier in DoC so that both of us could benefit from the fix.

This is normal behaviour for graphics paging. Of course I'm no expert but I'm thoroughly used to seeing it working that way. Is that a problem?

I believe that once they've been revealed, they stay that way or at least reveal more easily for the rest of the session, but perhaps not.

Click to expand...

I don't know, I don't play C2C regularly. But when I tried the feature for a while it was a lot smoother than it turned out in my implementation.

@Leoreth,

I wouldn't take Yudishtira's turn times as the avg/mean or even median turn times even in Industrial era.

I play on Huge and sometimes Giant Maps with 12 AI and My turn times are nowhere near as long as Yudishtira.

Now when we generally talk about "turn Times" it's a reference to how long the End of Turn wait is (EoT), and not generally how long it takes to play a turn.

For Yudishtira to take 12 hours to complete 1 Turn of Play ( not EoT) in Industrial there would be many factors to get it to take that long. And again I would say that is Not the norm.

- when moving the viewscreen by moving the mouse at the edges of the screen, tiles are not revealed as soon as they get into view. This is not a loading time issue, nothing happens until they've been further moved into the center of the screen.

Click to expand...

Are you talking about Viewports here or Graphical paging? The 1st does what you have described, but the 2nd generally does not. The only instance of lagging, as you move over the map, that I have experienced is the initial turn of a game session. After that on my system I don't notice it.

- moving around on the game map a lot can lead to a memory allocation failure ("bad allocation").

Click to expand...

I've have never experienced this on my win 7 Pro x64 OS with nvidia GTi 550 vid card and at least 4 Gb of ram (I have 8GB but even the x64 OS will only use 4GB, there is a 4GB "switch" that can be added to your BtS core that helps make the full 4Gb available for usage during play). If you are running a 32 bit OS and comp system Then some of what you have described here can happen quite easily. Guess that should've been asked at the outset of these discussions.

JosEPh

JosEPh_II said:

@Leoreth,

I wouldn't take Yudishtira's turn times as the avg/mean or even median turn times even in Industrial era.

I play on Huge and sometimes Giant Maps with 12 AI and My turn times are nowhere near as long as Yudishtira.

Now when we generally talk about "turn Times" it's a reference to how long the End of Turn wait is (EoT), and not generally how long it takes to play a turn.

For Yudishtira to take 12 hours to complete 1 Turn of Play ( not EoT) in Industrial there would be many factors to get it to take that long. And again I would say that is Not the norm.

Click to expand...

Oh yeah, that's how I interpreted it, but anything that is worse enough to justify saving every turn and risking a crash seems worse than what I'm dealing with at the moment. I realize that this depends on your game settings, of course.

Are you talking about Viewports here or Graphical paging? The 1st does what you have described, but the 2nd generally does not. The only instance of lagging, as you move over the map, that I have experienced is the initial turn of a game session. After that on my system I don't notice it.

Click to expand...

I'm pretty sure I did not accidentally implement viewports

I was mostly testing it right after initialization, so maybe that's where the problem comes from. If it disappears after having viewed every tile, I suppose it's not a problem.

I've have never experienced this on my win 7 Pro x64 OS with nvidia GTi 550 vid card and at least 4 Gb of ram (I have 8GB but even the x64 OS will only use 4GB, there is a 4GB "switch" that can be added to your BtS core that helps make the full 4Gb available for usage during play). If you are running a 32 bit OS and comp system Then some of what you have described here can happen quite easily. Guess that should've been asked at the outset of these discussions.

JosEPh

Click to expand...

Yeah, I should have been more specific here: I've been using a 64bit OS (Win7) with 8GB RAM, although I'm not aware of that switch you mention, can you link me to more detailed info? That seems very interesting even beyond graphical paging.

I wasn't aware that problems are to be expected with 32 bit OS's, I'll pass the information on to my players.

32bit OS may Need to use Both Viewports (at say 40x60 or smaller) And Graphical paging to avoid MAF's.

I don't remember where here on CFC I got this 4GB "switch/patch", but I did get it from this site. (it may have been from the 3gb switch discussions that RoM/AND2/C2C have had over the years). It's quite small and I've uploaded .zip file. It will reside in the Main BtS folder: C:\Program Files (x86)\Firaxis Games\Sid Meier's Civilization 4\Beyond the Sword . This "4gb patch" is Only for 64 bit OS and Does Not work for 32 bit OS. 32 bit still need the 3gb Switch.

What I don't remember is if it has a read me file.

JosEPh

JosEPh_II said:

32bit OS may Need to use Both Viewports (at say 40x60 or smaller) And Graphical paging to avoid MAF's.

Click to expand...

Alright, from what I've seen the implementation of viewports is quite a bit more complex. I think I'll keep it at graphical paging only and advise the 32bit people to disable it unless they're willing to risk MAFs.

I don't remember where here on CFC I got this 4GB "switch/patch", but I did get it from this site. (it may have been from the 3gb switch discussions that RoM/AND2/C2C have had over the years). It's quite small and I've uploaded .zip file. It will reside in the Main BtS folder: C:\Program Files (x86)\Firaxis Games\Sid Meier's Civilization 4\Beyond the Sword . This "4gb patch" is Only for 64 bit OS and Does Not work for 32 bit OS. 32 bit still need the 3gb Switch.

What I don't remember is if it has a read me file.

JosEPh

Click to expand...

Thanks a ton!

"Does not work for 32bit OS" means that they will remain stuck on 2GB, right? Or will it break compatibility for these OSs?

"Does not work for 32bit OS" means that they will remain stuck on 2GB, right? Or will it break compatibility for these OSs?

Click to expand...

32 bit OS need the 3gb switch so they can utilize up to 3gb of memory. Else Windows hogs it and they only can use 2.

64bit OS already allow the up to 3gb and this small 4gb atch allows it to use up to 4gb.

I think I'll keep it at graphical paging only and advise the 32bit people to disable it unless they're willing to risk MAFs.

Click to expand...

Koshling developed Graphical paging to Help 32bit OS cope with C2C's expanded demands for memory and to reduce the occurence of MAFs in those OS. So not sure where the wires got crossed that 32 bit OS shouldn't use it and risk MAFs. When in fact they should be Encouraged to use it to help reduce the occurrence of MAFs.

Both Viewports and Graphical paging are intended to reduce graphically caused MAFs in 32bit OS. Viewports was the 1st coding project and then Graphical paging was introduced to supersede Viewports. But even then on severely old and graphically ram challenged 32 bit OS koshling made it possible to use Both. That is why C2C still has Both options made available.

Does this help?

JosEPh

JosEPh_II said:

32 bit OS need the 3gb switch so they can utilize up to 3gb of memory. Else Windows hogs it and they only can use 2.

64bit OS already allow the up to 3gb and this small 4gb atch allows it to use up to 4gb.

Click to expand...

Alright, got it.

Koshling developed Graphical paging to Help 32bit OS cope with C2C's expanded demands for memory and to reduce the occurence of MAFs in those OS. So not sure where the wires got crossed that 32 bit OS shouldn't use it and risk MAFs. When in fact they should be Encouraged to use it to help reduce the occurrence of MAFs.

Both Viewports and Graphical paging are intended to reduce graphically caused MAFs in 32bit OS. Viewports was the 1st coding project and then Graphical paging was introduced to supersede Viewports. But even then on severely old and graphically ram challenged 32 bit OS koshling made it possible to use Both. That is why C2C still has Both options made available.

Does this help?

JosEPh

Click to expand...

Okay, I'm not mainly interested in reducing MAFs because I rarely hear about people encountering them while playing DoC (it helps that you can only play scenarios that are equivalent in size to BtS huge maps, and there is not that much additional content compared to the base game). The idea is more to improve on general performance issues related to memory consumption.

With my implementation of the graphical paging feature, some players including myself, who have never had issues with MAFs get them now. So it's a question of whether you would want to risk crashes for an increase in performance.

The way the paging works is to page in on-demand (based on dead reckoning about which tiles should be visible on your screen). The check for which tiles these are occurs every time slice by a routine in cvGame I cannot quite remember the name of (I no longer have the source on my machine I'm afraid) [cvGame::update() ??]. This checks the info the game engine provides about scaling and where the center of the screen is positioned to make the calculation. Possibly the scaling calculation is off on your hardware/resolution for some reason, but if so you should see the same effect in C2C run at the same res (from your posts I guess you don't though right?)

Graphics are only paged OUT when memory usage indicates it's a good idea (memory is getting low), which is why usually once you've looked at an area you don;t take much of a hit looking at it again later (the graphics are still loaded).

First off I suggest you find a reproducible case and try to do the analogous thing in C2C to verify that it at least appear to be a porting issue.

Secondly, as Joseph suggests, maybe you have accidentally grabbed some viewport-related code, though TBH it's reasonably separate to the best of my recollection.

Unfortunately 'bad allocation' is a nightmare symptom to debug, because it usually means 'something has trampled on the heap via a bad pointer', but it could be because you run out of (process) memory. To eliminate the latter (and indeed to stress the whole system more for testing purposes) you could try lowering the threshold at which it decides it needs to page stuff out due to low memory (I'm afraid I can;t remember where that is, but if you breakpoint paging out code it should be obvious from the stack).

Sorry I can't think of anything more precise, but it's been a couple of years since I last looked at this, and it WAS a huge nightmare to debug when I was originally writing it (actually this and viewports were a major factor that caused me to burn out on modding). The game engine isn't really designed to support things quite so radical, so there was lots of trial-and-error trying to find ways to bend it to my will!

Thanks for stopping by, this sheds some light on the inner workings of the feature. Regarding some of the possible explanations:
- I didn't encounter MAFs in C2C, but I didn't make the effort of creating comparable circumstances, I'll try the feature again on larger maps.
- I doubt it's my machine's configuration, players on SVN immediately encountered MAFs as well
- As you say, the parts of the code related to viewports are easy to make out so I'm really sure I didn't accidentally merge in some of that

In any case, I think I'll start by eliminating the possibility that I lack memory by adjusting the page out limit as you suggested.

Thank you very much for your help!

http://forums.civfanatics.com/showpost.php?p=13852520&postcount=36

alberts2 wrote: But what you don't understand is that the bad code was the bigger problem not the number of units. Because those inefficient pieces of code make all turns longer even if there are less units in the game.

Click to expand...

This is Important Info and needs to be saved.

JosEPh

A while back I did up a similar chart for unit and building build costs. At the time I noticed there were some odd spots in the tech research progressions as well and some increase inconsistencies. So over our enforced 'break' here from the forums and the SVN I've done some restructuring and recharting of tech research costs.

This chart should show those who can make heads or tails of it the REASON for the tech research costs to be established as they are here as well as the actual research costs we should now and forever be giving our techs based on the X layer on the tech tree. My impending update will include this as the chart currently stands.

I think DH has a few edits to the chart pending so @DH, please make those edits to the tech infos as they come in from this update before committing if you would.

Anyhow, the master tech cost progression chart is:

Progress Calc Cost is the value that a tech within the referred to X column should have as a Research Cost from here on out.

This would be a good post to link the first post to.

I added two of platyping's tools 'Forgetful Modder' shortcut CTRL-F1 and 'Time Keeper' shortcut CTRL-F2.

• platyping said:

  Forgetful Modder
  
  [*]Features:
  [*]When your mod has 1000 Promotions, 2000 Techs, 3000 Units and 4000 Buildings...
  [*]And for whatever reason, you cannot remember what is the type tag of this particular item you want to modify, fret not.
  
  [*]Just Press <CTRL F1> for help
  
  [*]
  
  [*]Full list of Units, Buildings, Techs, Projects, Promotions blah blah blah.
  [*]Can sort by ID, Type, Name, Graphical Only

  Click to expand...

• platyping said:

  Time Keeper
  
  [*]Features:
  [*]For modders to manage time properly...
  [*]Access via CTRL F2
  
  [*]
  [*]
  
  [*]For each GameSpeed, it shows you clearly for each set of increments, what is the starting year, the duration of it, how many turns etc.
  
  [*]At the end of it, it shows you the final year to see if it matches 2050.
  
  [*]After which, it shows you what is the starting year for each era, so if you see Classical Era starting in 1900 AD, you are screwed.
  
  [*]Notes:
  [*]Similar to Forgetful Modder, this is just a tool to help modders modify their mod, not useful for players, so the only access is via keyboard, rather than giving you a button to press somewhere.
  
  [*]Added to Ultrapack

  Click to expand...

Oh those look very handy. Will they be in the main game?

CTRL-F2 looks VERY nice! The forgetful modder one I usually overcome with wingrep but there may be times when it's super cool to have. Good additions Alberts2!

I commited the wrong version of Timekeeper, i update it with a full C2C version later. C2C can increment the Date by Days not just by Months like normal Civ4.

Pests Modmod now added. City pop sizes changed from 5,6,7, and 13 to 4,6,8,10,12, and 14. 20 pests now in 6 groupings instead of 4. Should greatly reduce City size 6 stagnation/starvation.

JosEPh