Running out of lights

billw2015 · Oct 8, 2019

Problem: Civ 4 will only support a max of 1024 DirectX lights active at the same time. We exhaust this limit quite easily when not using graphics paging. We hit this limit way before we run out of actual memory in some games.

But it confuses me because we should only need ONE light because there is only the sun (sometimes some effects from mine flares).

This lead me to start investigating what specific models are causing D3D light allocations (some esoteric techniques were required here!).

I started narrowing this down, and found the first offender. Who would have suspected the humble lobster?

Spoiler :

Here is what it looks like in game WITH those two lights:

Spoiler :

And here is what it looks like without:

Spoiler :

Better looking if anything. It is now being only lit by the sun instead of two random extra lights.
I also just found the sneaky shrimp has the same problem.

My plan now is to use the Python NIF tools and scan our entire model library for extra lights. Hopefully I can resolve the running out of lights crash just with model fixes and not more hacky things I was contemplating!

billw2015 · Oct 8, 2019

There are 6 extra lights hiding in here somewhere:

Spoiler :

billw2015 · Oct 8, 2019

Yeah looks like looooad of units and buildings have extra MAX_Default_Light instances, so I will need to write a script to remove them all. Might improve the aesthetics somewhat as well.

Thunderbrd · Oct 8, 2019

Fascinating revelation! I wonder if some of the shadows might give indication on local light settings.

billw2015 · Oct 8, 2019

Here is the full list as far as I can determine:

Spoiler :

Toffer90 · Oct 8, 2019

I was curious about what the issue could be, not all that surprised that there's a lot of unnecessary light sources in some of our models actually.
Many of our models are jury rigged mixtures of different models put together and I know there's a lot of leftover junk data in many of them.

Lobster... lol... Get rid of those light sources where it makes little sense, I (we) can modify the object material and textures if we don't like how it looks as a result of the light removal. No problem.

Edit: {
These probably have no reason to have light sources:
art\units\animals\cobra\cobra.nif
art\units\animals\cow\cow_fx.nif
art\units\animals\dog_1\doggy_2.nif
art\units\animals\dog_1\doggy.nif
art\units\animals\horse\roaminghorse_fx.nif
art\units\animals\horse\roaminghorse.nif

If units hold torches or some modern mechanized units have actual lights like a cars front light we should imo let those be to begin with, same with other models I guess.

billw2015 said:
Yeah looks like looooad of units and buildings have extra MAX_Default_Light instances, so I will need to write a script to remove them all. Might improve the aesthetics somewhat as well.

Probably an ok solution, just don't let the script remove the light sources in:

art\interface\screens\civilopedia\ancientbattlefield\ancientbattlefield.nif
art\interface\screens\civilopedia\civlopediaenvironmentwidget.nif
art\interface\screens\civilopedia\waterenvironment\waterenvironment.nif

These models are showed in the pedia and consist of entire scenes, I don't think the map sunlight reach these scenes.
}

billw2015 · Oct 8, 2019

Toffer90 said:
Probably an ok solution

It should be correct, MAX_Default_Light is ones added by the MAX exporter accidentally (I'm going to assume unless someone knows different). The list above includes every model with a light, I didn't filter for only ones with MAX_Default_Lights.
The lights in ancientbattlefield.nif for instance aren't MAX default ones:

Spoiler :

Toffer90 · Oct 8, 2019

billw2015 said:
It should be correct, MAX_Default_Light is ones added by the MAX exporter accidentally (I'm going to assume unless someone knows different). The list above includes every model with a light, I didn't filter for only ones with MAX_Default_Lights.
The lights in ancientbattlefield.nif for instance aren't MAX default ones:

Spoiler :

View attachment 536274

Ah ok, cool.

I've only used blender to make models for C2C, as it is free, it doesn't add any light sources by default.

JosEPh_II · Oct 8, 2019

Looks like our old friend and former modder Sparth like to use them a lot. lol

Thunderbrd · Oct 8, 2019

JosEPh_II said:
Looks like our old friend and former modder Sparth like to use them a lot. lol

It might have been, but it could also have just been the entire modding community having no idea there was a limit. Sparth did a lot of reskinning, or rather, repainting the texture layer, but I'm not sure this is a part of anything he actually worked on directly.

Dancing Hoskuld · Oct 9, 2019

Thunderbrd said:
It might have been, but it could also have just been the entire modding community having no idea there was a limit. Sparth did a lot of reskinning, or rather, repainting the texture layer, but I'm not sure this is a part of anything he actually worked on directly.

I agree. There is a lot to graphics and most just start by copying what is available and then learning a bit about one bit so that they can improve things. Sparth also got rid of a lot of duplication.

Raven Destroyer · Oct 9, 2019

in any case, it's always very nice Simple solutions can be found to critical problems, especially when it's basically a compounded thing that was done unknowingly, makes one wonder how much room there actually is in comparison to what is being used,

Thunderbrd · Oct 9, 2019

Raven Destroyer said:
in any case, it's always very nice Simple solutions can be found to critical problems, especially when it's basically a compounded thing that was done unknowingly, makes one wonder how much room there actually is in comparison to what is being used,

Could easily be responsible for some late game crashes I've been assuming were MAFs.

billw2015 · Oct 9, 2019

Okay these fixes are in now, I hit max 20-30 lights now while panning around the map, instead of >1000.
If you notice any buildings that look too dark then make sure to report them, I removed *all* lights, and that included non-default ones in 30 or so buildings. The tavern building had 4 lights on its own, and was also moon-lighting as a bank, meaning 8 potential extra lights per city if they are paged in. You can see how this can easily escalate to a crash and its only one building.

We can't control how many instances exist on the map (and thus if it will crash or not), so there needs to be a strong argument for having *any* lights in units or buildings I think, and only if a good effect can't be achieved using texturing, materials and emissivity (which doesn't use a light index).

I can now run stably at about 2500MB memory usage (any higher and I get actual out of memory crashes on EOT in the test save I am using). This allows paging in of most of the map, so I relaxed paging settings to improve how things look.

Toffer90 · Oct 9, 2019

billw2015 said:
Okay these fixes are in now, I hit max 20-30 lights now while panning around the map, instead of >1000.
If you notice any buildings that look too dark then make sure to report them, I removed *all* lights, and that included non-default ones in 30 or so buildings. The tavern building had 4 lights on its own, and was also moon-lighting as a bank, meaning 8 potential extra lights per city if they are paged in. You can see how this can easily escalate to a crash and its only one building.

We can't control how many instances exist on the map (and thus if it will crash or not), so there needs to be a strong argument for having *any* lights in units or buildings I think, and only if a good effect can't be achieved using texturing, materials and emissivity (which doesn't use a light index).

I can now run stably at about 2500MB memory usage (any higher and I get actual out of memory crashes on EOT in the test save I am using). This allows paging in of most of the map, so I relaxed paging settings to improve how things look.

Awesome stuff, hopefully we are now mostly rid of the mysterious CTD that occurs when simply panning the camera around the map of a somewhat advanced game.

raxo2222 · Oct 10, 2019

@billw2015 you are digging even deeper than koshling/AIAndy/alberts

Also I noticed git branch created for EXE hackery

billw2015 · Oct 10, 2019

raxo2222 said:
Also I noticed git branch created for EXE hackery

Yeah I'm keeping it on the dl though :cooool:

Didn't need to take out those guns to fix this problem in the end, so I just put them away for later.

raxo2222 · Oct 10, 2019

billw2015 said:
Yeah I'm keeping it on the dl though
Didn't need to take out those guns to fix this problem in the end, so I just put them away for later.

That Yudisitra save (Nanotech era with 19 AIs, Gigantic map (by area), Deity, Eons) takes around 1500 MB after it loaded.
Even with whole map revealed and with constant panning it climbed to only 1900 MB.

I disabled popups - it goes instantly to next city screen, when filling building queue.

billw2015 · Oct 10, 2019

You have MemSaver on if I remember correctly, I have it off and that save will crash OOM on EOT without paging enabled. The settings I used for default paging are based on making that save not crash!

raxo2222 · Oct 10, 2019

billw2015 said:
You have MemSaver on if I remember correctly, I have it off and that save will crash OOM on EOT without paging enabled. The settings I used for default paging are based on making that save not crash!

So even old Gigantic (24 000 tiles - now is around 13 000 tiles - original C2C_World Huge) and Pits map (bit bigger than Gigantic) should be be playable in Nanotech and later eras

Transhuman era has few units, and galactic and later have effectively no units unlocked - there are few of them but they all are space ones.

Running out of lights

King

King

King

C2C War Dog

King

C2C Modder

King

C2C Modder

TBS WarLord

C2C War Dog

Deity

Prince

C2C War Dog

King

C2C Modder

Time Traveller

King

Time Traveller

King

Time Traveller

Similar threads