1. We have added a Gift Upgrades feature that allows you to gift an account upgrade to another member, just in time for the holiday season. You can see the gift option when going to the Account Upgrades screen, or on any user profile screen.
    Dismiss Notice

Paging System v2

Discussion in 'Civ4 - Caveman 2 Cosmos' started by billw2015, Sep 13, 2019.

  1. AIAndy

    AIAndy Deity

    Joined:
    Jun 8, 2011
    Messages:
    3,420
    These days I program all day at work so I tend to do something else in my free time.

    Good to see that you are making progress.
    So you think the problem is not actually with memory over 2G?
     
  2. billw2015

    billw2015 King

    Joined:
    Jun 22, 2015
    Messages:
    837
    No it isn't. That was just a very preliminary guess based on a quick reading the disassembly and noting the issue seemed to occur as memory approached 2GB usage, and that it didn't happen if everything was just loaded from the start. I guessed that paging in and out combined with other ongoing allocations would push later allocations above 2GB.

    I can look into the memory and see that the entire bit field array that indicates light usage is full at the point where it crashes, so the failure to allocate a new light index is genuine. I'm going to add logging break points in the asm and then try and work out what pattern of usage caused them to fill up only when doing paging, but not when just loading *everything* from the start. My guesses right now are some kind of delayed deallocation of the lights, or a leak.
     
  3. AIAndy

    AIAndy Deity

    Joined:
    Jun 8, 2011
    Messages:
    3,420
    It is possible that if some of the objects that reference the lights are above 2G, some deallocation does not work properly and the lights are leaked.
    So a good solution is still to avoid that the exe gets any of the memory above 2G (e.g. by taking those virtual memory pages from the OS right at the start and only using them in the DLL).
     
    billw2015 likes this.
  4. billw2015

    billw2015 King

    Joined:
    Jun 22, 2015
    Messages:
    837
    It would be a solution looking for a problem at this point, as there isn't any evidence thus far that the address of the allocation is the problem (I was just wrong with my initial guess). I already tested by forcing all paged allocations above 2GB and it made no difference to how much paging I could do before it crashed.

    What *does* make a difference is how aggressive the paging is: If I page out everything outside the view range that I can then it is stable, if I allow lots of paged in data outside the view range this is when it will crash. But not if I just disable the paging entirely and load ALL data. So it is some combination of the high amount of paged in data combined with a constant stream of stuff being paged in and out that can cause it to run out of light indices.
    This is why delayed freeing of the resources is my best current guess. Certainly the callstack where the crash happens on *deallocation* of a light with an invalid index does *not* contain any of our own code. i.e. the deallocation is NOT happening when I am telling the exe to destroy the graphical component. It happens some time later (pretty common behaviour for this kind of system). If that some time later is in fact *a long* time later, then paging in and out more total data than exists on the map could theoretically allocate more light indices than is required for the entire map before it gets around to freeing them again.
    I might be able to find a function that can force a cleanup cycle (or maybe someone here knows of one), thus freeing the unused light indices.
    Regarding replacing the allocator: at some point it would be nice to take more control over the exe allocation system, e.g. so I can add tracking to it (using the knowledge I now have of what objects the exe is working with), but to solve the current crash it isn't relevant as far as I can see.
     
    AIAndy likes this.
  5. alberts2

    alberts2 Emperor

    Joined:
    Aug 16, 2012
    Messages:
    1,947
    Gender:
    Male
    Location:
    Germany
    Have you looked at the properties of the lights which are added but not removed are they all the same?
     
  6. billw2015

    billw2015 King

    Joined:
    Jun 22, 2015
    Messages:
    837
    I didn't confirm that any lights are added but not removed, only that the light allocation table (a 1024 bit array of flags indicating which indices are in use) is full up at the point it crashes. Certainly one possibility is that some aren't being freed at all, in which case I will investigate if there is some problem with them.
     
  7. alberts2

    alberts2 Emperor

    Joined:
    Aug 16, 2012
    Messages:
    1,947
    Gender:
    Male
    Location:
    Germany
    I added some very simple logging to Direct3DDevice9Ex::SetLight and when you move around the map you can see new lights being added up to the index 1023.
    But after reaching 1023 I don't get a crash the engine simply starts again at index 3.
     
  8. Toffer90

    Toffer90 C2C Modder

    Joined:
    Oct 16, 2011
    Messages:
    7,149
    Location:
    Norway
    I'm guessing index 0, 1 and 2 are reserved for global map lighting (and such light sources).
     
  9. billw2015

    billw2015 King

    Joined:
    Jun 22, 2015
    Messages:
    837
    By what method? I am looking for what the best graphics debugger is for dx9, I didn't try any with C2C yet though.

    Yeah because it has freed the ones at the lower end again. The light manager keeps track of the last index used and tries to allocate the next available one, wrapping at 1024. The crash happens when the other ones didn't get freed quickly enough and it can't find a free index. To reproduce you need to increase the max memory setting in the globals XML to around 2000000 or so and pan around on a large and fully revealed map.
     
  10. alberts2

    alberts2 Emperor

    Joined:
    Aug 16, 2012
    Messages:
    1,947
    Gender:
    Male
    Location:
    Germany
    Wrapper dll using the code from https://github.com/elishacloud/DirectX-Wrappers/tree/master/d3d9 with some logging code added.
    Code:
    HRESULT m_IDirect3DDevice9Ex::SetLight(DWORD Index, CONST D3DLIGHT9 *pLight)
    {
        fmt::print(logf, "SetLight:{} x:{} y:{} z:{}\r\n", Index, pLight->Position.x, pLight->Position.y, pLight->Position.z);
    
        return ProxyInterface->SetLight(Index, pLight);
    }
     
  11. billw2015

    billw2015 King

    Joined:
    Jun 22, 2015
    Messages:
    837
    Oh okay same as I did then! As a side note: we can actually use that light pointer to back track through all the game objects :mischief:
     
  12. alberts2

    alberts2 Emperor

    Joined:
    Aug 16, 2012
    Messages:
    1,947
    Gender:
    Male
    Location:
    Germany
    I finally got an error but the ram usage was still below 2GB:
    mem.PNG
     
  13. billw2015

    billw2015 King

    Joined:
    Jun 22, 2015
    Messages:
    837
    Huh okay, will have to look at that. Good to know what version of gamebryo they are using as well.
    edit: There it is video memory it fails to allocate not main RAM, how much VRAM do you have?
    Most newer cards have 3GB+
     
  14. alberts2

    alberts2 Emperor

    Joined:
    Aug 16, 2012
    Messages:
    1,947
    Gender:
    Male
    Location:
    Germany
    My laptop has a 1050Ti with of 4GB VRAM.

    I always limit games like CivIV to 30fps when using the laptop because more fps aren't necessary for these games.
    Your computer is faster than my laptop that and the limited fps could make a difference here.
     
  15. MattCA

    MattCA Prince

    Joined:
    Jan 25, 2019
    Messages:
    322
    Gender:
    Male
    a while ago TB posted that his wife got a new faster computer, and it seemed to MAF a lot more then old cpu
     
    billw2015 likes this.
  16. JosEPh_II

    JosEPh_II TBS WarLord

    Joined:
    Jun 20, 2007
    Messages:
    16,252
    Gender:
    Male
    Location:
    Western IL. cornfields
    A reminder, the BtS game Recommend video ram is 128Mb with DirectX 8 support (pixel and vertex shaders). Straight from the game manual.
    DirectX 9.0c (included in game CDs)
    Recommended OS are: Win 2000 SP1 or higher, XP Home or Pro (plus SP1, or higher), and Vista.

    You have more players with vid cards having less than 3GB of Vram than you have playing with 3+GB Vram.

    I have a GTX 760 Ti OEM card with 2GB DDR5. But I only within the past 2 years upgraded fro the Comp's original CTX 550 Ti with 1GB DDR5.

    When ever ram usage from Task manager hits 2.3-2.7GB I will have crashes.

    Maybe this means nothing to you billw or alberts2, but I thought it expedient to remind you both.
     
    MattCA likes this.
  17. billw2015

    billw2015 King

    Joined:
    Jun 22, 2015
    Messages:
    837
    Sure I was just using 3GB as an example of what to expect on new cards. However I don't think the min specs are relevant to us at all, C2C won't run on them regardless. Video RAM wise I would say 1GB is a fine spec though.

    Yeah seems about right, it will never be much better than this. Although the theoretical limit is 3GB, fragmentation means we can never actually get there. There is definitely still the bug with the light allocation when using paging, so hopefully I can fix that. Then we can be more stable when using heavy paging with high memory limits and usage, so we can look at other projects that need more memory.
     
  18. alberts2

    alberts2 Emperor

    Joined:
    Aug 16, 2012
    Messages:
    1,947
    Gender:
    Male
    Location:
    Germany
    After doing a 1.2GB dummy memory allocation I finally got the game to crash while panning the map.
    But the crash was in the visual c++ runtime dll not inside a d3d dll. However d3d light log showed the same index problem you had.

    To see the real memory usage I think it's necessary to look at the Commit size not the Working set size.
    memusage.PNG
     
  19. billw2015

    billw2015 King

    Joined:
    Jun 22, 2015
    Messages:
    837
    Yeah commit is the total amount of pages you have allocated, working set is those you actually tried to use "recently". I still didn't work out the implications.
    Late last night I did manage to add logging breakpoints to light create and delete functions. I paged around the map until it crashed from light index issue and there were 350 lights deleted and 800 created. So I think it is simply a leak where some aren't deleted rather than caused by delayed deletion. I did notice that light delete happened pretty much as soon as I moved the camera, indicating that the delay is not big (probably on the same frame as the entity destroy is requested).
    Next I am going to test will all different layers of the paging enabled separately to see which one is generating lots of lights and not deleting them.
     

Share This Page