C2C - UEM - Ultimate-Earth-Map 100% MOD and SVN update compatible by Pit2015

[Thunderbrd]

TB never wants the code to change.

It's just hard to keep up with its many changes as it is so if there's no good REASON for it to change, why change it? Obviously changes for the sake of truly improving things are great. I don't want to change the number of players because if we DO have enough data room for that now, we still need that room for more important things to come.

@Thunderbrd give me more civs, if more civs are possible there is no one forced to use them but it is possible. I say build it into the C2C mod.

It is not nearly as straightforward as it sounds and requires a shton of reprogramming through numerous places in the xml as well - it took me about a month to get it down from 100 to 50 and adding some room for NPCs and in the years that have passed since then I have forgotten all the many barbed complexities of bugs that doing this without care to hit on some 50 other checkpoints in the code to adjust to accommodate + all the python and xml files that must be then altered and by doing this I'm just making a dll that then won't allow a player to finish the tech tree and as stated above, even if they could, we have plans for major expansion still that we didn't just clean up the inefficient memory usage just to fill with this kind of excess. I can't begin to imagine how much Multimaps and Nomadic Starts and volumetric resources and the ideas project will demand in terms of memory expense.

I haven't gone out of the way to explain because I've hardly had time to explain it, but there are a LOT of bugs that will go wrong when this gets adjusted. Learning how to alter the player count is simple - tracking down the months of niggly bug reports that follow and solving them is not.

 

[Somebody613]

I haven't gone out of the way to explain because I've hardly had time to explain it, but there are a LOT of bugs that will go wrong when this gets adjusted. Learning how to alter the player count is simple - tracking down the months of niggly bug reports that follow and solving them is not.

Which rather confuses me a lot.
Sure, the most obvious problem would be the same as the one that was fixed recently - that certain callbacks refer to NPCs by their actual #, not their name_tags.
But this should be easy to fish out and fix - why are you expecting it to be so much massive, really?
And besides that, I can't even think of any serious bug that would appear directly due to changing the number of max_civs, like at all.
Can you maybe elaborate, please?

 

[Pit2015]

It's just hard to keep up with its many changes as it is so if there's no good REASON for it to change, why change it? Obviously changes for the sake of truly improving things are great. I don't want to change the number of players because if we DO have enough data room for that now, we still need that room for more important things to come.


It is not nearly as straightforward as it sounds and requires a shton of reprogramming through numerous places in the xml as well - it took me about a month to get it down from 100 to 50 and adding some room for NPCs and in the years that have passed since then I have forgotten all the many barbed complexities of bugs that doing this without care to hit on some 50 other checkpoints in the code to adjust to accommodate + all the python and xml files that must be then altered and by doing this I'm just making a dll that then won't allow a player to finish the tech tree and as stated above, even if they could, we have plans for major expansion still that we didn't just clean up the inefficient memory usage just to fill with this kind of excess. I can't begin to imagine how much Multimaps and Nomadic Starts and volumetric resources and the ideas project will demand in terms of memory expense.

I haven't gone out of the way to explain because I've hardly had time to explain it, but there are a LOT of bugs that will go wrong when this gets adjusted. Learning how to alter the player count is simple - tracking down the months of niggly bug reports that follow and solving them is not.

Adding more civs will be cool, also if memory may still a problem for some months, you can recommend 50 civs and no one is forced to use more then 50 civs but it will be possible, on my current long game memory usage is low and has improved alot. I say try to raise it and if bugs come up lets see if they can be fixed. I agree that there may come up some bugs, but well it is C2C lets boldly go where no man has gone before.

 

[Somebody613]

Adding more civs will be cool, also if memory may still a problem for some months, you can recommend 50 civs and no one is forced to use more then 50 civs but it will be possible, on my current long game memory usage is low and has improved alot. I say try to raise it and if bugs come up lets see if they can be fixed. I agree that there may come up some bugs, but well it is C2C lets boldly go where no man has gone before.

There are several automatic problems there:
1. I was told that the max_civs number is activated regardless of how many actual civs you use on a certain map.
Though why would non-existing civs taking up much memory, I don't really know, lol.
2. The number of units and cities (but mostly units) is usually proportional to the number of civs in play.
But I already suggested using the "only 1 unit per tile" BUG rule, which should defuse that problem drastically.
Again, this might lead to other problems (like making units unable to Merge, maybe, didn't check yet), but it would quite nicely prevent the "overcrowding" memory problem to begin with.
Heck, I get MAFs now even without changing anything (with just 40 civs), so I'm thinking of maybe trying that "1 unit" rule already on a normal map - and see how it works specifically.
3. On the other hand, if the above issues are actually solved (or not real), then there's no need to go only "a few" civs up - there'd be no difference between 50 civs and 200 civs, if it's fixed.
After all, the map is still the same - so the number of available cities and units hardly depends on the number of separate civs producing them.
Which is why I mentioned another mod that has HUNDREDS of cities (but definitely not more than 40 civs, if even that) on an Earth map - and is supposed to be playable anyways.
So all of this IS possible, indeed.

 

[Thunderbrd]

But I already suggested using the "only 1 unit per tile" BUG rule, which should defuse that problem drastically.

Which would destroy AI function unless one were to completely rewrite how the AI works. And that's assuming you AREN'T, as I am, playing a CivIV mod already entirely because you can't stand how 5 and 6 went to 1UPT and that's your main beef.

Adding more civs will be cool, also if memory may still a problem for some months, you can recommend 50 civs and no one is forced to use more then 50 civs but it will be possible, on my current long game memory usage is low and has improved alot. I say try to raise it and if bugs come up lets see if they can be fixed. I agree that there may come up some bugs, but well it is C2C lets boldly go where no man has gone before.

To me this is just playing rubber band. Running from one side of a court to another every time the situation changes to require it, where it takes a lamentable amount of effort to do it. I'm not doing these laps but if someone else who REALLY knows what they are doing wants to constantly have to take the some hundred or two hundred hours to mod this every time we realize it needs to change against when it CAN change, then whatever. My problem is that person that expands it out probably won't be there to bother when it needs to be retracted again. Most of the memory for a civ is set aside whether its in a game or not which is why the dll hardcodes it.

 

[Somebody613]

Which would destroy AI function unless one were to completely rewrite how the AI works. And that's assuming you AREN'T, as I am, playing a CivIV mod already entirely because you can't stand how 5 and 6 went to 1UPT and that's your main beef.

To me this is just playing rubber band. Running from one side of a court to another every time the situation changes to require it, where it takes a lamentable amount of effort to do it. I'm not doing these laps but if someone else who REALLY knows what they are doing wants to constantly have to take the some hundred or two hundred hours to mod this every time we realize it needs to change against when it CAN change, then whatever. My problem is that person that expands it out probably won't be there to bother when it needs to be retracted again. Most of the memory for a civ is set aside whether its in a game or not which is why the dll hardcodes it.

Can you be more specific about how changing the mere number of max civs affects anything besides the number of spots in the memory for those civs?
I honestly don't see any potential effect this change should have on the files, except for the case where the NPCs are referenced by their # and not their tags, which is easily editable (and was done).
Seriously, what are you complaining about here - I utterly fail to understand THAT, lol?

And as of 1-unit-per-tile.
This was mentioned merely "in case memory goes nuts, but we still want to try out 200 civs".
I kinda doubt we'd need to go there in the first place, though, cue me mentioning the mod that put hundreds of cities on the Earth map - and still had it playable.
Which again comes back to the question of "what effect this number of civs even has to begin with".

 

[Thunderbrd]

Can you be more specific about how changing the mere number of max civs affects anything besides the number of spots in the memory for those civs?
I honestly don't see any potential effect this change should have on the files, except for the case where the NPCs are referenced by their # and not their tags, which is easily editable (and was done).
Seriously, what are you complaining about here - I utterly fail to understand THAT, lol?

You might want to ask Toffer as he decided to try to make some changes recently in this area and began to see the laundry list of small bugs it introduces here and there, potentially. As I said, it's about the endless cleanup of little bugs it creates that it takes sorting through with patience to see how and why its a bug that was introduced by this change. It may be possible to do that and then figure out how to not let it be a problem again in the future but I don't have the patience to keep breaking what has been fixed over and over. Nor do I have the personal memory to recall all the fixes I had to make.

"what effect this number of civs even has to begin with".

If this question were asked in proper English it would be easier to answer. What kind of effect are you asking about? Are you asking "What numeric impact is there on memory limits to have more Civs allocated?" I'm not honestly sure but I know that the unit limits in the game are 'per player'. There are game options, particularly Rev and Barb Civs, that allow for every player slot to keep getting another allocated active player. Every player slot is fully tracked for the sake of the game playback at the end, whether that slot was ever activated or not. I'm not sure how much memory is allocated for every possible player slot, but the fact is that whether they are used or not, there is some memory allocated to each one and cannot be used for the rest of the game, whether those player slots are even active or not, thus squeezing the memory usage in all departments due to the extreme memory limits we have to work within here.

You said you were getting MAFs before finishing a game without this change? That's why I would never consider it. We reduced the max player count so as to help to avoid those to begin with. This particular map is one of the most expensive and least likely to get to the end of a game as it is in particular due to the massive size of the map and thus tons of plots to track and even more cities and units across it all.

More units were injected into the game recently to help the AI manage itself better by making units a bit cheaper to make and it has helped for now until a more strategically efficient war managing AI can be programmed. Speaking for myself, I struggle to find ANY time to mod any kind of effective effort directly rather than just talk about it a bit here and there so if I DO get some time for it, reversing a measure I already took so as to help fix things is not high on my agenda.

 

[Somebody613]

,,,

I'll try explaining my question again.
Let's assume that the game can support exactly 1000 units and exactly 200 cities at the same time.
If you have max 40 civs on the map at any given moment, then you allow for 25 units and 5 cities per civ to exist at the same time.
And if you raise the max number to 200, then you allow for 5 units and 1 city per civ to exist at the same time.
But so long as you don't reach that limit - there shouldn't even be a memory problem in the first place, since the actual limit is not reached yet.
Meaning that having 999 units (aka still below the limit that is 1000 units) for 40 civs OR for 200 civs is... the same in regards to memory consumption.
Or similar enough not to matter significantly.
On the other hand, even 40 civs can easily grow to exceed that limit of 1000 units just as well - and then have a crash despite still being only 40 civs.
So it's not the number of civs that would cause such a crash, but rather the number of units or cities on the map.
On the other hand, if we go with my suggestion and lower the number of potential units on the map (by making it 1 per tile, solely for this reason), then we simply dodge this problem all along.
Basically, if the memory can support 2000 units, but the map can only host 1500 (regardless of belonging to how many civs) - then we would dodge the MAF altogether.
Regardless of how many civs would be sharing those 1500 units - since it's the total number of units on the map that is the issue, not the number of units per each civ.
Unless it is the other way around - which is what I asked you about:
What is directly affected by the number of on-map civs (or max-civs)?
And as of "bugs" - once again, can you name me at least a few such "bugs", because I really can't think of any "bugs" that wouldn't be related to NPCs Player_Number, and that one is easy to fix.
Thanks.

 

[Thunderbrd]

then you allow for 5 units and 1 city per civ to exist at the same time.

These aren't given hard coded limits but rather also have limitations based on memory and indexing limitations and when they are hit, it's not a memory limitation that makes the game error out at that point but rather the fact that a player has been allowed to be so excessive by the cheapness of units or the expanse of an overly large map perhaps. In sum, they do all add to the overall memory count that sums up to a potential MAF, which is the point at which total game memory possible to use has been exceeded.

Just allocating a player takes a portion of that total memory and puts it aside whether the player is in the game or on the map at all. Therefore, hardcoding too many potential players will impact a game with 2 players as much as it will impact a game with 150. You might assume from there that the number of players IN play might make a difference, and it does, but relatively it's more just that they were made possible and thus the shell to track them was established when the dll was compiled.

On the other hand, if we go with my suggestion and lower the number of potential units on the map (by making it 1 per tile, solely for this reason), then we simply dodge this problem all along.

Yes, and at that point you're rewriting the entire foundation of the way the game plays, which is heavily based on an endless stack potential on a tile. This is both in the expectations of how armies must be comprised to defend, attack, and the AI itself requires escorts on some units, like settlers for example, for those units to even be willing to move, stacks must have an established amount of power to be willing to mobilize, have enough healing and siege and some other key elements, or you'll never see the AI invade. Criminals require being able to slip into cities and LE units must also be able to be on that tile as well, sharing the space with military to protect the city and so on.

You might as well try writing a different game. Even limiting a tile to a number of units cripples the way the AI is programmed to attempt to win cold wars by having larger stronger forces than their foes.

Basically, if the memory can support 2000 units, but the map can only host 1500 (regardless of belonging to how many civs) - then we would dodge the MAF altogether.

The map can host an endless amount until the total maf is reached from all memory storage use. Each player is limited to some number of thousands of units but that's just because of how the unit index list is established and I do think there's a similar limit on a given plot but it's just again a limit due to the limit of the size of an Integer I believe. It's not usually a problem but when barbs can pump out hundreds of units in a city in a round and are left much of the game to just hoard as many as they can, they WILL exceed and crash the game for that reason, probably long before a MAF on the total memory usage becomes a problem. There is no hard limit on how many units can be in the game at all... but it's certainly diminished by quite a margin if you're putting aside a bunch of memory for players that are never even likely to be in the game.

And as of "bugs" - once again, can you name me at least a few such "bugs", because I really can't think of any "bugs" that wouldn't be related to NPCs Player_Number, and that one is easy to fix.

Well, for one thing, this is a change that disrupts game saves every time its made. If a few things aren't done properly in the player save and load sequence, you'll get all kinds of odd behaviors where a value for a player's variables get written as if it should be a different variable - stupid things like that - I don't know how much was just the establishment of the allocations for NPC slots but the fact that those have to be dedicated and often remembered by their ID numbers means wherever they are referenced as such needs to then be adjusted to match them properly, which is in hundreds of places.

 

[Somebody613]

These aren't given hard coded limits but rather also have limitations based on memory and indexing limitations and when they are hit, it's not a memory limitation that makes the game error out at that point but rather the fact that a player has been allowed to be so excessive by the cheapness of units or the expanse of an overly large map perhaps. In sum, they do all add to the overall memory count that sums up to a potential MAF, which is the point at which total game memory possible to use has been exceeded.

Just allocating a player takes a portion of that total memory and puts it aside whether the player is in the game or on the map at all. Therefore, hardcoding too many potential players will impact a game with 2 players as much as it will impact a game with 150. You might assume from there that the number of players IN play might make a difference, and it does, but relatively it's more just that they were made possible and thus the shell to track them was established when the dll was compiled.


Yes, and at that point you're rewriting the entire foundation of the way the game plays, which is heavily based on an endless stack potential on a tile. This is both in the expectations of how armies must be comprised to defend, attack, and the AI itself requires escorts on some units, like settlers for example, for those units to even be willing to move, stacks must have an established amount of power to be willing to mobilize, have enough healing and siege and some other key elements, or you'll never see the AI invade. Criminals require being able to slip into cities and LE units must also be able to be on that tile as well, sharing the space with military to protect the city and so on.

You might as well try writing a different game. Even limiting a tile to a number of units cripples the way the AI is programmed to attempt to win cold wars by having larger stronger forces than their foes.


The map can host an endless amount until the total maf is reached from all memory storage use. Each player is limited to some number of thousands of units but that's just because of how the unit index list is established and I do think there's a similar limit on a given plot but it's just again a limit due to the limit of the size of an Integer I believe. It's not usually a problem but when barbs can pump out hundreds of units in a city in a round and are left much of the game to just hoard as many as they can, they WILL exceed and crash the game for that reason, probably long before a MAF on the total memory usage becomes a problem. There is no hard limit on how many units can be in the game at all... but it's certainly diminished by quite a margin if you're putting aside a bunch of memory for players that are never even likely to be in the game.

Well, for one thing, this is a change that disrupts game saves every time its made. If a few things aren't done properly in the player save and load sequence, you'll get all kinds of odd behaviors where a value for a player's variables get written as if it should be a different variable - stupid things like that - I don't know how much was just the establishment of the allocations for NPC slots but the fact that those have to be dedicated and often remembered by their ID numbers means wherever they are referenced as such needs to then be adjusted to match them properly, which is in hundreds of places.

Dude, raxxo fixed the NPC_# issue like last week or so.
And it's definitely not in "dozens of places", given how his fix removed the problem in my "compilation for 200 civs" altogether.
If anything, it's probably already entirely fixed to begin with.
Or at least it clearly didn't give me any trouble at all in my test game.
And at the very worst, you'd need to fish for the words like PlayerType or whatever the problematic # issue is.
It will take you maybe 5 minutes to locate all relevant files in Git SOURCES - and then 5 more minutes to switch all PlayerTypes to Animal_Player style, instead of Player(40) style or whatever.
It's literally that easy to do.

As of MAFs, let's wait until I finally produce that 200-civ map edition and try playing it for a while, lol.
I'm pretty sure it WON'T be much of a trouble, but it's easy to just play-and-see, ya know.

 

[raxo2222]

Dude, raxxo fixed the NPC_# issue like last week or so.
And it's definitely not in "dozens of places", given how his fix removed the problem in my "compilation for 200 civs" altogether.
If anything, it's probably already entirely fixed to begin with.
Or at least it clearly didn't give me any trouble at all in my test game.
And at the very worst, you'd need to fish for the words like PlayerType or whatever the problematic # issue is.
It will take you maybe 5 minutes to locate all relevant files in Git SOURCES - and then 5 more minutes to switch all PlayerTypes to Animal_Player style, instead of Player(40) style or whatever.
It's literally that easy to do.

As of MAFs, let's wait until I finally produce that 200-civ map edition and try playing it for a while, lol.
I'm pretty sure it WON'T be much of a trouble, but it's easy to just play-and-see, ya know.

Mattca changed SpawnInfos so player types (like player_beast) can be used instead of player enums (like 45), I did XML edit there.
Also there were few hardcoded places in python, but I guess it was only places where did this happen.

 

[Somebody613]

Mattca changed SpawnInfos so player types (like player_beast) can be used instead of player enums (like 45), I did XML edit there.
Also there were few hardcoded places in python, but I guess it was only places where did this happen.

I didn't remember exactly.
Anyways, it seems like this is already fixed, lol.
So what is TB complaining about, what do you think?

 

[Toffer90]

So what is TB complaining about, what do you think?

It will break saves, and it will increase memory usage, and it will increase the amount of iterations performed each time all players are iterated, which is a couple hundred times per end turn in a normal game (meaning 150 multiplied with a couple hundred iterations more per turn on average if we upped the limit to 200).

The hard thing is to set it up with an xml define, where changing it won't break saves, and supporting two dll's like many other mods do (like RoM's 100 civ dll) is not something anyone on the team wants, and we simply don't see the need for changing the current value considering it will have a performance impact.

We like the current limit and see little reason to change it the next time we decide to break saves.
Maybe we'll increase the max limit down the line, after further comprehensive code optimizations have been implemented.

 

[Somebody613]

It will break saves, and it will increase memory usage, and it will increase the amount of iterations performed each time all players are iterated, which is a couple hundred times per end turn in a normal game (meaning 150 multiplied with a couple hundred iterations more per turn on average if we upped the limit to 200).

The hard thing is to set it up with an xml define, where changing it won't break saves, and supporting two dll's like many other mods do (like RoM's 100 civ dll) is not something anyone on the team wants, and we simply don't see the need for changing the current value considering it will have a performance impact.

We like the current limit and see little reason to change it the next time we decide to break saves.
Maybe we'll increase the max limit down the line, after further comprehensive code optimizations have been implemented.

I understand the points about save-breaking and DLL-replacement.
But it sounded more like "there gonna be dozens of bugs due to this", which isn't merely what I wrote here.
So I still don't see where all those "dozens of bugs" would come from.
Also, it's literally changing a few lines in one file - so how does compilation work that it isn't simply a change in one resulting file that could be replaced as a modmod?
Or maybe it is, then what's the big deal with switching one file around for different saves?
Just make two RARs of "40 civs DLL" and "200 civs DLL" - and unRAR the one you currently need, big deal.
I'm really confused by how much this is posited as a problem, while it doesn't look to me that way whatsoever, really.

 

[Toffer90]

So I still don't see where all those "dozens of bugs" would come from.

I wouldn't know, TB is probably exaggerating, changing the max civ number has never been hard and bundling an alternative 100 civ dll's have been commonplace in big mods for many years.
However there is always a potential for unexpected bugs to pop up in some python/dll code, or even worse perhaps even in exe due to some exe code assumption regarding top player slots.
We in the C2C team have never thoroughly investigated/confirmed that the exe is 100% happy with e.g. a max player slot of 100, or 200 as you want.

Or maybe it is, then what's the big deal with switching one file around for different saves?

Mostly this, would be annoying to debug saves only to realize we compiled the wrong dll, because the player posting the save didn't specify which dll he/she used.
Automatically compiling a second final release dll at each SVN revision would also add 30-45 minutes to the time it takes appveyor to finish a new SVN revision.
Also, we would have to have two versions of at least one of the cpp source files, which has the potential for a modder to only mod one of them and forget about the other, we would also have to rewrite some appveyor scripting and mess with our file structure and build tools and such for an alternate source file to be easy to accommodate, would just be a practical mess for us modders in certain ways is all I'm saying..

 

[Toffer90]

In other words, the answer to all your questions is:

• We are lazy, deal with it.

;p

 

[Somebody613]

In other words, the answer to all your questions is:

• We are lazy, deal with it.

;p

You still could do it as a modmod at each release and/or big SVN save-breaks.
Like, once a month or something.
Or really just once per release, lol.
And why is it taking you 30+ minutes, if it takes me merely 5 minutes to compile everything?
Anyways, I might eventually be not-lazy enough to test it out myself one day.
Maybe.
I'm even more lazy than you, loool.

 

[Thunderbrd]

You still could do it as a modmod at each release and/or big SVN save-breaks.
Like, once a month or something.
Or really just once per release, lol.
And why is it taking you 30+ minutes, if it takes me merely 5 minutes to compile everything?
Anyways, I might eventually be not-lazy enough to test it out myself one day.
Maybe.
I'm even more lazy than you, loool.

As I said, if you want to keep up on trying to make sure that say, the SVN or a modmod thread always has the latest compiled version of a 200 player dll, go for it - we change the code so rapidly that would be annoying af to keep up with for any of us. And if and when bugs come up because of those adjustments, maybe you can find the ways to make it easier to adapt things. Many of the NPC players, for example, are hardcoded in the XML by their Player ID #, which is different when you have a different count of potential players. Spawns are coded by their player ID # and how many spawn XML entries are there? Maybe some of this has already been made easier somehow - there's probably ways to make it all link up better and MattCA likes implementing those kinds of things. I just strongly suspect you'll find more bugs exist than you expected. I'm not really wanting to research to tell you what, specifically, they would be. But you already encountered a few and maybe in so doing have lubricated the mechanism to enable this to be easier to maintain as an alt dll. Then try to explain to people how to USE an alt dll... ugh.

 

[Pit2015]

The gamer wants more civs, will also improve the mod and make more detailed scenarios possible and will make the mod even more interesting. Maybe something can be done that the engine only use the civs that are added on a map, so like only reserve memory for civ slots that are used. Otherwise maybe as option or something, i am sure @Toffer90 will develop some idears. And i think it will not increase the memory usage to much, as said before someone can think about how to displace units in citys or something from memory by saving them to the harddrive.

Only one unit per tile sucks, its no option. But dont load units into RAM that are not displayed in viewport may work.

Then this can be combined with @Joij21 civ megapack and so on and will push the mod into a new universe. 4 GB limit will be enouth for endless maps if viewport is improved and only stuff in viewport loaded into RAM and then cleared correctly again when moving the viewport or something like this.

But as said, memory usage is improved already, so more civs should not increase it to much, maybe toffer can do a test with 200 civs and check memory usage in taskmgr and if unused empty civs will increase memory usage at all or how low.

Change "int" variables to "long" variables if i remember correctly from my coding days to close memory leaks and prevent MAFs or overflowing of variables.

https://www.arduino.cc/reference/en/language/variables/data-types/long/

 

[MattCA]

Anyone can build the dll.
Developer Guide · caveman2cosmos/Caveman2Cosmos Wiki (github.com)