Massive Humans vs Humans Game/Lets Player Tracker

[Thunderbrd]

It's been my presumption that when a unit dies, its index gets listed as available for the next unit initialized for that player to reuse. Yes... by the remove functions. The ORIGINAL bug we ran smack into should never have been possible. The iNextFreeIndex referred to an index that was null. So what I attempted to do, and it worked at the time, was sever the secondary list chain, destroy the rest of those indexes by putting them in inaccessible quarantine, and repairing the m_iFreeListCount, then letting the game initiate a new unit index entirely for the unit being initialized.

Something went very wrong after this succeeded. It worked just fine for that round but then the next round, this happens: Massive Humans vs Humans Game/Lets Player Tracker

 

[Koshling]

Correction:
01010
01001 &
01000 = 8

It doesn't yield 0, so it enters the 'if' and is corrected to 16 (10000 in binary)

Doh! Sorry - I must be blind!

The purpose of this baroque id allocation structure (as I recall) is to support all of the following:

1) Very fast id->entity resolution (mask and then deref through an array)
2) Modest memory usage for lookup indexes
3) Never to reuse the same id twice (strictly it only has to guarantee to not re-use an id that might still be in use, but 'in use' includes as a reference from another entity such as a follow order, or a build order tagged as support for an existing unit, or a try to follow-and-attack order, so death of a unit does not guarantee immediate non-use of its id)
4) Not moving through the id space too fast, or we run out before the end of the game

Easily solved with 64-bit ids, but the game would need a very significant rewrite to accommodate a switch from 32 to 64 bit ids.

 

[Thunderbrd]

Although it's a mystical system that's VERY difficult to follow the steps on, I guess I understand that if you were to have a much simpler vector to track unitIDs and just enumerate them, and track ones that can be reused, it would also be much slower to reference a particular index. The amount of times getAt() is called is horrifyingly enormous so I guess it needed the fastest possible method.

 

[Thunderbrd]

Although sometimes my methods might be reminiscent of the engineering techniques of Krynn Gnomes (point to you if you understand the reference), I believe I have worked out a solution. My last 'fix' did indeed set us up for the problem this round. It was what was going to happen after the free list (of indexes being recycled) ran down to the last one. The way I had fixed the count was correct with the exception that it left an unashamed invalid index where the last index in the free list was concerned. This caused problems but it didn't manifest until the list ran out and the validation counts, which apparently are very important because those counts say how many units (and any other items for that matter) should be written and read during the save/load sequence, were correct with the exception that one bad allocation still remained once it was put into play. So when that happens, instead we now assign the last index value and assign a new variable that tallies into the final count to track the one corrupted value that is ultimately being written off.

I know my ability to explain this is in no way a programmers and it's probably been extremely confusing to read. It would make more sense if you just look at the SVN changelog. I believe this will also be important to include in the main mod just in case a game hits the same corruption issue this whole sideshow of 'fixes' can correct.

Anyhow, once the dlls are compiled, Praetyre has given me full instructions on what he wanted done so I'm just going to have Whispy take her turn and then we can commit the fixed DLLs and her turn. At that point, I think Koshling will be next. I THINK we might be able to get this done tonight.

EDIT: Yep, all is good and we may proceed!

UPDATE YOUR DLLs again!

Koshling is up!

 

[Koshling]

With the latest DLL it bombs out when I try to load the game (after I select the save file to load but before asking e for my password [in fact very quickly after the loading progress bar appears]).

As a sanity check I tried loading the OTHER PBM game - it gets to the password stage fine, so it's specific to the main game. Since it happens BEFORE asking for my password it should be reproducible by anyone from the current save, so if someone could verify...

 

[MagnusIlluminus]

Semi-verified. It crashes just as the theme song gets to "What if Civ...", or in other words, just after the background appears. NOTE I have not updated to the newest DLL so the error is not necessarily with the new DLL.

The MP game loads just fine.

 

[Spirictum]

I've updated the dll and the crash also happened here, in the Initialization process

 

[Thunderbrd]

*sigh* ok... if you have a password, please send it so I can simply make sure it can get loaded once I'm finished with it. There's apparently another landmine to path around here.

EDIT: nope won't need that. I've got this solved already but It'll take replaying those last two turns. I overlooked the fact that I would need to save and load the new variable. This allows the next turn to load and save but then the turn after that the math falls back out of synch. Fixing now.

EDIT: looks like it's going to be a little more complicated than I'd thought. I think I know what I'm up against but unfortunately I'm really out of time more than anything at the moment. And I'm out of commission tomorrow entirely so It won't be till Saturday before I can address this. Apologies... was hoping to have it completed and back on track tonight.

EDIT: As I was posting the last edit, I had a sudden insight and just had to press my time limits here and test the thought... and it looks like I was right. I had simply failed to give a default initial value for the new tracking variable since the processing stream never hits the standard initialization routine, and that's throwing off a host of other things later. Ok, easy enough fixed. Just compiling and getting Whisperr to take her turn again and we'll be back on track again. I'll confirm by attempting to load Koshling's save as far as I can (which would trigger the crash if it's still an issue once this is done.)

Should be able to proceed shortly here.

EDIT: Apparently we're out of time to have whisperr take her turn so we'll do that Saturday.

 

[JosEPh_II]

I've updated the dll and the crash also happened here, in the Initialization process

Late to the party but confirmed, CTD during Intialization.

JosEPh

 

[Thunderbrd]

Late to the party but confirmed, CTD during Intialization.

JosEPh

lol. Yes. Any attempt will do that at the moment. The fix is in place on my system already, provided whispy can take her turn again without issue and I can then load Koshling's, which I'll have to test tomorrow.

 

[Spirictum]

I was wondering after Koshling mentioned:

3) Never to reuse the same id twice (strictly it only has to guarantee to not re-use an id that might still be in use, but 'in use' includes as a reference from another entity such as a follow order, or a build order tagged as support for an existing unit, or a try to follow-and-attack order, so death of a unit does not guarantee immediate non-use of its id)

What exactly this list we've been talking about stores as data? Is it only for units? I suppose that template T stuff has something to do with being able to use these lists for several different types of information in the game

 

[Koshling]

I was wondering after Koshling mentioned:


What exactly this list we've been talking about stores as data? Is it only for units? I suppose that template T stuff has something to do with being able to use these lists for several different types of information in the game

At least units and groups. Can't remember if its anything else. Each usage has its own id-space though so they are independent.

 

[Spirictum]

Presuming the 8192 slots are filled by units (I don't think this is impossible in a Gigantic map), would the game call for a second list to keep track of units above the limit of one list?

If anyone can point me to calls to these functions in other files, I would be glad to read them. I just don't know where to look now

 

[Thunderbrd]

At least units and groups. Can't remember if its anything else. Each usage has its own id-space though so they are independent.

It's everything that has its own independent ID in the game. Players, Teams, Units, Plots, Maps, Even Agreements etc... It is NOT used for Techs, Promotions, Buildings, Civics. These are modifiers to the things that have an ID, not things that need their own outside of their 'type' enumeration. You can't have two different Combat promotions, for example, in the game. A combat promotion is a combat promotion no matter what it's attached to. Nothing can attach to it.

If anyone can point me to calls to these functions in other files, I would be glad to read them. I just don't know where to look now

getAt(int iIndex) is universally used for accessing all the different kinds of T types.

Presuming the 8192 slots are filled by units (I don't think this is impossible in a Gigantic map), would the game call for a second list to keep track of units above the limit of one list?

No, I believe it crashes at that point due to hopeless data corruption. However, I believe that the slots for units are tracked by player, not by the game. However, and this is where the issue we're working with is here, the various counts of used indexes must be accurate or it's inaccuracy spills into the next ID tracking counts and is so horrifically bad when it does this that the programming is setup to shortcircuit with a very assertive assert when this happens.

 

[Thunderbrd]

Hopefully 3rd time is a charm.

UPDATE YOUR DLLs again!

Koshling is up!

 

[Koshling]

Turn taken successfully! Magnus is up...

 

[Spirictum]

No, I believe it crashes at that point due to hopeless data corruption. However, I believe that the slots for units are tracked by player, not by the game. However, and this is where the issue we're working with is here, the various counts of used indexes must be accurate or it's inaccuracy spills into the next ID tracking counts and is so horrifically bad when it does this that the programming is setup to shortcircuit with a very assertive assert when this happens.

So that explains so many redundant and normally impossible asserts. But the issue you're mentioning is the one about the animal barbarian, or the original issue?

 

[Thunderbrd]

So that explains so many redundant and normally impossible asserts. But the issue you're mentioning is the one about the animal barbarian, or the original issue?

Was all the same issue. It was just a matter of not knowing how many landmines I would be stepping in and when they would go off. AKA, I kept being shown another layer to learn with each fix. The tracking on this stuff is intense and is not forgiving if it goes off track and its that way by design to protect the integrity of the game. So I had to basically invent a way to wipe out a corrupt spot of data without setting off all the alarm systems. Was kinda like mathematicaly playing 'operation'. Or figuring out how to pull the statue off the stone without triggering the trap in the opening scene of the Raiders of the Lost Ark. Get the sand that replaces it to weigh out just right or else.

 

[Spirictum]

But do you believe what corrupted the data in the first place was something not likely to do it again? Or do you think it's something that can be recurring?


Also, if 8192 slots may not be enough for certain T data, and could be causing an overwritting somewhere else (likely in other lists of different T data types), why not use a larger index structure? Because now these lists use 13-bit IDs

 

[Thunderbrd]

But do you believe what corrupted the data in the first place was something not likely to do it again? Or do you think it's something that can be recurring?

I think it happened as a result of someone not updating when they were supposed to. Something rather unique to the PBEM environment. Hopefully won't happen again. I've seen something similar to this I think in the main but it happens with extreme rarity at best. This would not be the first time that sometimes 2+2 equaled 3 somewhere way back in the labyrinth of coding and you'd have to have been there monitoring that exact spot in the code to find out how it happened. I tend to chalk it up to the thought that corruption happens sometimes.

Also, if 8192 slots may not be enough for certain T data, and could be causing an overwritting somewhere else (likely in other lists of different T data types), why not use a larger index structure? Because now these lists use 13-bit IDs

If someone were fully qualified to make this upgrade, I'd be pleased if they joined the team and did so. The usual problem with these sorts of things, and I really think would be the main issue here, is that some things in the DLL cannot be changed because they report back to the EXE we cannot see. That core beast tends to have its own ideas as to how things are supposed to appear and if you violate these hidden rules, all hell breaks loose. I strongly suspect that trying to change the index structure in any major way like this would cause the game to crash in the .exe somewhere.