-
Civilization 7 has been announced. For more info please check the forum here .
Memory Leak?
- Thread starter jinif
- Start date
snepp
King
oldStatesman said:
It's pretty obvious he didn't read anything. It's a heck of a way to make your first post.
"leaking" address space is about the only memory leak that user-mode applications can do. They allocate their memory inside their virtual address space. If an application forgets to release allocated memory, this address block is not usable to application anymore. But applications don't control where their address space is mapped (to RAM, to swap space on the hard drive or nowhere at all). In some cases OS can help to mask application memory leak by swapping out lost block of mapped application's address space to the swap on the hard drive, thus releasing RAM. But this only works if there few large blocks lost, not when there's a lot of small blocks leaked.oldStatesman said:
So my point is that when programmers talk about memory leak they mean what you call 'soft' leak. (Just wanted to say 'people' instead of 'programmers', but then realised that then 'memory leak' would mean anything remotely related to computers :> )
First of all I want to thank all you guys for your respect. It didn't let me to give up in hardest situations, I would surely stop my efforts, should I know that I write it just for myself Currently I am downloading that hot-fix, probably I'll be able to set it up and see what have they fixed.
I blamed a lot things which didn't actually matter. For example, I blamed Python when it wasn't actually source of problems just because it was the first thing to suspect. For example, when you see a 3-wheel car going slowly, you will suspect 4th wheel absence as the reason of slow-downs, not broken engine. But when you investigate the issue, it actually becomes an engine... Situation here was about the same.
The game does drop back from 650Mb to 128Mb memory usage when I load my hugest test save game and then exit back to the main menu. Well, right by the start it's about 80AMb, so 40Mb are not freed (not necessarily lost), probably due to Python lazy dellocation; I don't know. But if I load same savegame again, it's again 650Mb, not 690 or 700 - so it's ok.
Firaxis didn't concact me yet. Probably just because no exe/dll still stands behind my words, just words... Neither I contacted them. Well, just because my memory manager which probably makes allocations better, actually makes no sence, since the problem originates in Direct3D allocations size, not the way they are aligned.
My solution would optimize memory usage, so that after allocation 200Mb of pure data you would get, say 250Mb of actualy used memory, not 400 or 500. But it wasn't the problem... There is no need to optimize placement of a pile of boxes if the reason is just too big amount of them.
I said that entire game engine is written in Python. Even if this true, this doesn't make sense either. What would help this game is some 256Mb or 512Mb of video memory - that'll make it, but again only with a fix.
I have spotted that Direct3D objects are not placed in video memory, though they are placed in D3DPOOL_MANAGED. Probably they had their dirty hands on wrong priorities management, I don't know... Placing everything to D3DPOOL_DEFAULT sped it up a little, and less resources were kept in main RAM, but it needs more adjustments, so that textures don't go black after alt-tabbing and stuff. That's my current solution of the problem.
I saw about 250Mb of GEOMETRY DATA with only ~45Mb of textures!!! John Carmack would rather code Quake1 soft renderer in visual basic, rather then allow such a number... Also this seems to be backed by civ4 internal memory state, so we have about 500Mb of waste... I can fight off D3D waste of 250mb - that's what I am currently doing.
Currently I am downloading that hot-fix, probably I'll be able to set it up and see what have they fixed.I blamed a lot things which didn't actually matter. For example, I blamed Python when it wasn't actually source of problems just because it was the first thing to suspect. For example, when you see a 3-wheel car going slowly, you will suspect 4th wheel absence as the reason of slow-downs, not broken engine. But when you investigate the issue, it actually becomes an engine... Situation here was about the same.
The game does drop back from 650Mb to 128Mb memory usage when I load my hugest test save game and then exit back to the main menu. Well, right by the start it's about 80AMb, so 40Mb are not freed (not necessarily lost), probably due to Python lazy dellocation; I don't know. But if I load same savegame again, it's again 650Mb, not 690 or 700 - so it's ok.
Firaxis didn't concact me yet. Probably just because no exe/dll still stands behind my words, just words...
Neither I contacted them. Well, just because my memory manager which probably makes allocations better, actually makes no sence, since the problem originates in Direct3D allocations size, not the way they are aligned.My solution would optimize memory usage, so that after allocation 200Mb of pure data you would get, say 250Mb of actualy used memory, not 400 or 500. But it wasn't the problem... There is no need to optimize placement of a pile of boxes if the reason is just too big amount of them.
I said that entire game engine is written in Python. Even if this true, this doesn't make sense either. What would help this game is some 256Mb or 512Mb of video memory - that'll make it, but again only with a fix.
I have spotted that Direct3D objects are not placed in video memory, though they are placed in D3DPOOL_MANAGED. Probably they had their dirty hands on wrong priorities management, I don't know... Placing everything to D3DPOOL_DEFAULT sped it up a little, and less resources were kept in main RAM, but it needs more adjustments, so that textures don't go black after alt-tabbing and stuff. That's my current solution of the problem.
I saw about 250Mb of GEOMETRY DATA with only ~45Mb of textures!!! John Carmack would rather code Quake1 soft renderer in visual basic, rather then allow such a number... Also this seems to be backed by civ4 internal memory state, so we have about 500Mb of waste... I can fight off D3D waste of 250mb - that's what I am currently doing.
I think they know nothing about vertex blending and stuff, so they just allocate a separate vertex buffer for every unit, and probably even generate per-frame vertex buffers for each visible unit beforehand, based on amount of physical memory you have - that might explain why it gets to 1.2Gb with 1Gb when with 512Mb it goes only 650-700Mb.
Harkonnen: I like your analysis.
I have worked with many programming languages but not Python. I took a quick look and found out that it is an object oriented language. It also looks like it is interpreted. From what little I have read in these forums, it looks like it is interpreted in Civ 4.
Twenty years ago someone proved to me that P-code (the object code compiled from Pascal and then interpreted) was ten times faster than Basic which was completely interpreted. Assembler was ten times faster than P-code. I was employed to write in Pascal and assembler for real time systems where Basic was too slow.
Object oriented languages like C++ use extra memory to simplify object concepts and to allow late binding. Late binding itself is slow. But it gives much more flexibility in conceptualizing an idea in a program. I remember when C++ first became popular. I read a lot of arguments between C++ and C programmers about speed. C at the time was the fasted compiled language. C++ gave up a little in speed and memory usage to add object oriented concepts.
Python likely cut down on the development time and made modification easier. It looks like the cost was higher memory needs and less speed. So far my few problems appear to occur when other applications like a scheduled virus scan grabs the foreground. But I have 2GB memory and about 3.8 GHz of CPU speed.
I have seen over the years that computer games keep requiring more and more resources. This also has been occurring with other applications as well. But games seem to have been a big performance driver. Civ 4 is state-of-the-art. It uses a lot of advanced hardware to provide its spectacular graphics and sound.
In all, I am happy to see the game now instead of a year from now. I am happy that I have up-to-date hardware to play it on. In the past I have purchased games that looked like fun but wouldn’t run on my second-hand hardware. It didn’t bother me then and it doesn’t bother me now. If something doesn’t work on my current hardware I save it until I next update my systems. And it looks like software advances are making systems obsolete in two years instead the three years it took ten years ago. Where I work we have problems running non-game software on equipment that was new two years ago. I kind of expect the same or more in the gaming world.
I have worked with many programming languages but not Python. I took a quick look and found out that it is an object oriented language. It also looks like it is interpreted. From what little I have read in these forums, it looks like it is interpreted in Civ 4.
Twenty years ago someone proved to me that P-code (the object code compiled from Pascal and then interpreted) was ten times faster than Basic which was completely interpreted. Assembler was ten times faster than P-code. I was employed to write in Pascal and assembler for real time systems where Basic was too slow.
Object oriented languages like C++ use extra memory to simplify object concepts and to allow late binding. Late binding itself is slow. But it gives much more flexibility in conceptualizing an idea in a program. I remember when C++ first became popular. I read a lot of arguments between C++ and C programmers about speed. C at the time was the fasted compiled language. C++ gave up a little in speed and memory usage to add object oriented concepts.
Python likely cut down on the development time and made modification easier. It looks like the cost was higher memory needs and less speed. So far my few problems appear to occur when other applications like a scheduled virus scan grabs the foreground. But I have 2GB memory and about 3.8 GHz of CPU speed.
I have seen over the years that computer games keep requiring more and more resources. This also has been occurring with other applications as well. But games seem to have been a big performance driver. Civ 4 is state-of-the-art. It uses a lot of advanced hardware to provide its spectacular graphics and sound.
In all, I am happy to see the game now instead of a year from now. I am happy that I have up-to-date hardware to play it on. In the past I have purchased games that looked like fun but wouldn’t run on my second-hand hardware. It didn’t bother me then and it doesn’t bother me now. If something doesn’t work on my current hardware I save it until I next update my systems. And it looks like software advances are making systems obsolete in two years instead the three years it took ten years ago. Where I work we have problems running non-game software on equipment that was new two years ago. I kind of expect the same or more in the gaming world.
Fallblau
Chieftain
- Joined
- Sep 16, 2005
- Messages
- 91
>I saw about 250Mb of GEOMETRY DATA
Lord... that's immense... You can keep the poly-count low on each object, but when you cram as many objects into a persistent state as they seem to be doing, it really stacks up. Still, it makes one wonder... Are they fully rendering objects that are currently hidden from view and just not displaying them, rather than determining first whether the object can be "seen" and then making the render/no-render decision? That's what it is starting to sound like from what you're describing...
It may be a "three-layered" problem, though, as you could have three states: One, object hidden by "fog-of-war"; Two, not in "fog" but not on-screen; Three, fully within field of view.
The primary symptom described (i.e., game begins to chug when world maps are traded) would suggest that a render/no-render check exists between state one and state two only, so that all objects "exist" to the renderer (and therefore occupy "space" (i.e. VRAM)) in both states two and three, giving a wastage at the ratio of Known World/Visible World....
All theoretical b*llsh*t and just thinking out loud... Was that 250Meg of geometry constant, or a peak state in the game?
Lord... that's immense... You can keep the poly-count low on each object, but when you cram as many objects into a persistent state as they seem to be doing, it really stacks up. Still, it makes one wonder... Are they fully rendering objects that are currently hidden from view and just not displaying them, rather than determining first whether the object can be "seen" and then making the render/no-render decision? That's what it is starting to sound like from what you're describing...
It may be a "three-layered" problem, though, as you could have three states: One, object hidden by "fog-of-war"; Two, not in "fog" but not on-screen; Three, fully within field of view.
The primary symptom described (i.e., game begins to chug when world maps are traded) would suggest that a render/no-render check exists between state one and state two only, so that all objects "exist" to the renderer (and therefore occupy "space" (i.e. VRAM)) in both states two and three, giving a wastage at the ratio of Known World/Visible World....
All theoretical b*llsh*t and just thinking out loud... Was that 250Meg of geometry constant, or a peak state in the game?
Downloaded that patch... Yes, it didn't install - it told I have nothing. I investigated it a little, extracted CABs... some notices:
1) File version tells it comes from MacroVision (provider of that ~e0005.exe
stuff with SafeDisk), so this still might be their issue.
2) That damn 100Mb I downloaded (and I pay for traffic) are all about June2005 DirectX run-time. Only ~1-2Mb is left to installer core and probably Firaxis fixed executables which I was unable to extract by hands yet. Gonna try some tool now...
1) File version tells it comes from MacroVision (provider of that ~e0005.exe
stuff with SafeDisk), so this still might be their issue.
2) That damn 100Mb I downloaded (and I pay for traffic) are all about June2005 DirectX run-time. Only ~1-2Mb is left to installer core and probably Firaxis fixed executables which I was unable to extract by hands yet. Gonna try some tool now...
alexti2
Thanks for the notice, it really helps. I saw such string in some resource files, but I though it's some internal Microsoft stuff. This will help me to learn their code better. And it says that rendering is not completely python-based
Fallblau
Actually you are right in everything. D3DPOOL_MANAGED tells DirectX to keep memory copy of data and commit it to video-memory upon request. That memory copy allows automatic restoration when video memory resources are lost (e.g. after alt-tabbing). Placing them into D3DPOOL_DEFAULT utilizes video memory, but alt-tabbing issues must be handled by game engine.
I am going to write a wrapper layer betwene Direct3D and the game, and manage pool by myself, placing local copies in memory-mapped files. This should reduce number of committed virtual memroy pages (virtual memory size), and split memory pool between D3D data (rarely accessed all over the place) and game data (frequently accessed all over the place).
Also I suspect that among those 250Mb burden a ot of duplicates occur. Since we have lock/unlock interface for accessing D3D data, I can CRC vertex buffers data and eliminate dupes with copy-on-write mechanics to track differences. This might impact speed though... but I must write it first to test.
I am happy you're betting on this Just a happy week out from main work is over, so now I can spend only 3-5 hours with this (mostly during the night), that week it was ~12. Damn, I had to start with D3D, not with memory 
Thanks for the notice, it really helps. I saw such string in some resource files, but I though it's some internal Microsoft stuff. This will help me to learn their code better. And it says that rendering is not completely python-based
Fallblau
Actually you are right in everything. D3DPOOL_MANAGED tells DirectX to keep memory copy of data and commit it to video-memory upon request. That memory copy allows automatic restoration when video memory resources are lost (e.g. after alt-tabbing). Placing them into D3DPOOL_DEFAULT utilizes video memory, but alt-tabbing issues must be handled by game engine.
I am going to write a wrapper layer betwene Direct3D and the game, and manage pool by myself, placing local copies in memory-mapped files. This should reduce number of committed virtual memroy pages (virtual memory size), and split memory pool between D3D data (rarely accessed all over the place) and game data (frequently accessed all over the place).
Also I suspect that among those 250Mb burden a ot of duplicates occur. Since we have lock/unlock interface for accessing D3D data, I can CRC vertex buffers data and eliminate dupes with copy-on-write mechanics to track differences. This might impact speed though... but I must write it first to test.
I am happy you're betting on this
Just a happy week out from main work is over, so now I can spend only 3-5 hours with this (mostly during the night), that week it was ~12. Damn, I had to start with D3D, not with memory Fallblau
That was constant (not peak) memory usage by d3d9.dll using LocalAlloc API function. Sum of LocalAlloc/LocalFree/LocalReAlloc calls equaled to amount of vertex data.
This is also not a leak with vertex buffers, since index buffers get to about 60-70Mb in size, an expected proportion to 250Mb of vertex buffers data.
That was constant (not peak) memory usage by d3d9.dll using LocalAlloc API function. Sum of LocalAlloc/LocalFree/LocalReAlloc calls equaled to amount of vertex data.
This is also not a leak with vertex buffers, since index buffers get to about 60-70Mb in size, an expected proportion to 250Mb of vertex buffers data.
elderath2000
Chieftain
- Joined
- Nov 16, 2005
- Messages
- 1
Hi, hope im not just spinning a broken record here but im not quite clear if the problem im having is the same as the one you guys seem to be suffering from.
I start the game and it looks great, runs great, and just seems like the most incredible game ive ever played! then... right about when i hit gunpowder, things slow down a tad...no biggie... but when i save, then load my game at a latter time...... ICREDIBLE LAG! i mean, its nasty, i cant scroll side to side and combat is pretty much done without my units actually moving. this just seems odd to me because i can play some intense battlefield 2 with no real chop, and yet wee little civ4 seems to pretty much freeze up. Is this a memory leak? my comp is very well maintained and im running a p4 2.0 gig with 1 gig of ram and a 128meg ati vid card (9600 turbo, or something like that) again, sorry if im just adding to the spam, but if any one can clear up my little problem it would be appreciated
I start the game and it looks great, runs great, and just seems like the most incredible game ive ever played! then... right about when i hit gunpowder, things slow down a tad...no biggie... but when i save, then load my game at a latter time...... ICREDIBLE LAG! i mean, its nasty, i cant scroll side to side and combat is pretty much done without my units actually moving. this just seems odd to me because i can play some intense battlefield 2 with no real chop, and yet wee little civ4 seems to pretty much freeze up. Is this a memory leak? my comp is very well maintained and im running a p4 2.0 gig with 1 gig of ram and a 128meg ati vid card (9600 turbo, or something like that) again, sorry if im just adding to the spam, but if any one can clear up my little problem it would be appreciated
alexti2 said:You're doing great job! The bets on who will issue the patch first is still on
In case it helps, Firaxis is using Emergent's Gamebryo engine in Civ4.
Eeep! That's what Pirates used (similar problems) and what ES Oblivion is going to use. I'll have to rethink Oblivion then.
frank
Fallblau
Chieftain
- Joined
- Sep 16, 2005
- Messages
- 91
Well, maybe we'll see their (Firaxis') solution soon...
http://www.2kgames.com/civ4/firaxis_note_01.htm
They mention "memory" and "caching" and "performance"... So hopefully they got a chance to tighten things up a bit...
http://www.2kgames.com/civ4/firaxis_note_01.htm
They mention "memory" and "caching" and "performance"... So hopefully they got a chance to tighten things up a bit...
Ok, I managed to install that D2D hotfix wih my DVD version. Nothing changed at all, at least with respect to memory consumption. I don't know if it affects other aspects of the game like peace treaty exploit and alike.
If you want to try it, follow these steps:
1) Run 'regedit.exe' (registy editor).
2) Go to HKEY_LOCAL_MACHINE\Software\Microsoft\Windows\CurrentVersion\Uninstall
3) Export sub-key named {4377F918-E6C9-4ECA-A7F5-754B310B7ED8} to, say, c:\civ4.reg
4) Open c:\civ4.reg with notepad
5) Replace every occurence of "{4377F918-E6C9-4ECA-A7F5-754B310B7ED8}" with "{1CF028E5-705D-4B62-AC1D-A59593B7C0BB}" (I had two occurences to replace)
6) Run c:\civ4.reg as if it was an executable
7) Run civ4_d2d_hotfix.exe - it should work now
If you want to try it, follow these steps:
1) Run 'regedit.exe' (registy editor).
2) Go to HKEY_LOCAL_MACHINE\Software\Microsoft\Windows\CurrentVersion\Uninstall
3) Export sub-key named {4377F918-E6C9-4ECA-A7F5-754B310B7ED8} to, say, c:\civ4.reg
4) Open c:\civ4.reg with notepad
5) Replace every occurence of "{4377F918-E6C9-4ECA-A7F5-754B310B7ED8}" with "{1CF028E5-705D-4B62-AC1D-A59593B7C0BB}" (I had two occurences to replace)
6) Run c:\civ4.reg as if it was an executable
7) Run civ4_d2d_hotfix.exe - it should work now
I joined this website just so I could respond to DaEezt. (Actually, I was probably going to join anyway, but DaEezt was the catalyst).
DaEezt, your posts are VERY condesending, even more so since you can't seem to even see it for yourself.
Semester 7? That's weaksauce to me. I have Mast Degree in Finance. 90%+ of what I hear most people say about Finance (and economics) I could correct, expand upon, laugh at, or other display a sense of superiority about.
No, I'm not going to explain anything about Finance to you or other prove/display my knowledge-base. (now, doesn't that sound arrogant...? maybe just a little bit?)
Good for you that you that you are going to college. Not everybody does or can. Good for you that might might actually know something that could be helpful to somebody else. Maybe in addition to improving your knowledge of computers you might improve your wisdom regarding what you will do with this knowledge....hopefully that is covered in semester 8...
As for my posts....don't bother troubling yourself with them. I don't want your computer knowledge and don't need you to berate my computer science ignorance.
Regards,
PS: I happen to be an executive at a video game company and if you worked for me, that attitude would change in a hurry or you'd be gone....
DaEezt, your posts are VERY condesending, even more so since you can't seem to even see it for yourself.
Semester 7? That's weaksauce to me. I have Mast Degree in Finance. 90%+ of what I hear most people say about Finance (and economics) I could correct, expand upon, laugh at, or other display a sense of superiority about.
No, I'm not going to explain anything about Finance to you or other prove/display my knowledge-base. (now, doesn't that sound arrogant...? maybe just a little bit?)
Good for you that you that you are going to college. Not everybody does or can. Good for you that might might actually know something that could be helpful to somebody else. Maybe in addition to improving your knowledge of computers you might improve your wisdom regarding what you will do with this knowledge....hopefully that is covered in semester 8...
As for my posts....don't bother troubling yourself with them. I don't want your computer knowledge and don't need you to berate my computer science ignorance.
Regards,
PS: I happen to be an executive at a video game company and if you worked for me, that attitude would change in a hurry or you'd be gone....
warpstorm
Yumbo? Yumbo!
Harkonnen said:alexti2
Thanks for the notice, it really helps. I saw such string in some resource files, but I though it's some internal Microsoft stuff. This will help me to learn their code better. And it says that rendering is not completely python-based
I would think that all of the rendering is done by the Gamebryo rendering engine they licensed. It is a D3D based library written in C++.
Similar threads
