C2C - UEM - Ultimate-Earth-Map 100% MOD and SVN update compatible by Pit2015

Thunderbrd said:
I'm not saying there's not some things I believe I can do, just saying that what you are suggesting, basically a relocating of unit data - a sort of caching - is light years beyond my understanding and I cannot imagine it would not create massive slowdowns as the game stores and restores units with great regularity, even if just to check the potential threats around a given unit, or calculate what unit would be best to attack with in a 200 unit stack.
Click to expand...
You are right that as there is currently no consideration about when a unit is accessed, this would likely cause huge slowdowns and that despite being a huge effort in implementing it.

To me, it seems the only real solution would be to improve the method of storing some variables and doing all we can to cut down on how many variables are in use and try to identify some variables that never need to be initialized for some units by their type. All that alone is a mega project in itself. And the crappy thing is, work like this is completely unnoticeable because it adds nothing to the game but a little more wiggle room for it to continue past where resources are pushed to the limit.
Click to expand...
The reduction of the amount of memory used up by units is definitely the low hanging fruit here. As a first step you could check if a lot of the ints in there actually need to be ints or if they can be shorter (might need pragma pack to get an actual result). Or you could pack bools into bits. Especially if they are rarely accessed so the potentially slower access matters less.
Then you could split off more of the data that tends to be the default value for most units into one or more separate structs similar to how the keyed infos are done (e.g. info that is only used for a specific type of unit).
Another optimization possibility is that a lot of those structs that can be split off have the same value for a lot of units (as the value comes from the unit type and maybe a promotion or two). It is possible to store that value in a map as key with a reference count as value (so you can delete it if it is no more referenced) as then a simple pointer to that map entry can be stored in the unit itself. The difficulty about that is that you cannot simply change the value in there then but instead each time you change something in that struct for one unit you have to create a new separate entry (potentially deleting the old one). This would not be that great currently as there are a lot of changes to single values while the unit is created.
 
AIAndy said:
You are right that as there is currently no consideration about when a unit is accessed, this would likely cause huge slowdowns and that despite being a huge effort in implementing it.


The reduction of the amount of memory used up by units is definitely the low hanging fruit here. As a first step you could check if a lot of the ints in there actually need to be ints or if they can be shorter (might need pragma pack to get an actual result). Or you could pack bools into bits. Especially if they are rarely accessed so the potentially slower access matters less.
Then you could split off more of the data that tends to be the default value for most units into one or more separate structs similar to how the keyed infos are done (e.g. info that is only used for a specific type of unit).
Another optimization possibility is that a lot of those structs that can be split off have the same value for a lot of units (as the value comes from the unit type and maybe a promotion or two). It is possible to store that value in a map as key with a reference count as value (so you can delete it if it is no more referenced) as then a simple pointer to that map entry can be stored in the unit itself. The difficulty about that is that you cannot simply change the value in there then but instead each time you change something in that struct for one unit you have to create a new separate entry (potentially deleting the old one). This would not be that great currently as there are a lot of changes to single values while the unit is created.
Click to expand...
I won't pretend I completely understand your suggestions but I think I get an inkling about what you're suggesting and I might be able to eventually research my way through to a deeper grasp of it.
 
Thunderbrd said:
I won't pretend I completely understand your suggestions but I think I get an inkling about what you're suggesting and I might be able to eventually research my way through to a deeper grasp of it.
Click to expand...
The first suggestion is the easiest. Consider if you really need 32 bit integers for each of the entries variables in a unit. If the value never gets larger than 32k, then a 16 bit integer (short, but I prefer typedefs like int16_t) is enough to store it. If the value is always smaller than 255, then an 8 bit integer (signed char, or better a typedef like int8_t) is sufficient. Of course that change can introduce subtle overflow buffs and integer promotion rules can be weird.
Bools only store true/false, so they can be stored in 1 bit instead of 1 byte. The easiest way to get that effect is to group them into a bit field inside the class. The disadvantage is that access to that kind of compressed bool is slower.

The second suggestion is still rather easy. Move some of the data to a separate struct, then only store the pointer to that struct within the unit class. If the values in the struct are all default, then you don't allocate the struct and just point to NULL. That way you save memory if most units don't have that separate struct.

The third is more complicated so should only be considered as a second step.
 
Thunderbrd said:
Yeah it's rough on processing but I love it when I can get a small stack to hold it's ground against this sort of force, or when the AI realizes a couple of nukes doesn't care how many units you have when they're all on the same plot.
Click to expand...

That behavior happens when you gamed too much Nuclear War in your earlier days. :goodjob::goodjob::goodjob: I bombed there alot too. :king:

 
AIAndy said:
The first suggestion is the easiest. Consider if you really need 32 bit integers for each of the entries variables in a unit. If the value never gets larger than 32k, then a 16 bit integer (short, but I prefer typedefs like int16_t) is enough to store it. If the value is always smaller than 255, then an 8 bit integer (signed char, or better a typedef like int8_t) is sufficient. Of course that change can introduce subtle overflow buffs and integer promotion rules can be weird.
Bools only store true/false, so they can be stored in 1 bit instead of 1 byte. The easiest way to get that effect is to group them into a bit field inside the class. The disadvantage is that access to that kind of compressed bool is slower.
Click to expand...
I think ill start with idea #1. Sounds simple.
Idea #2 about the bool, I think I understand the concept but I would need code to copy/paste from. :)
 
MattCA said:
Idea #2 about the bool, I think I understand the concept but I would need code to copy/paste from. :)
Click to expand...
I can't help you with where it is in the code, but this method is used for the setting that determines what city buildings are graphically represented on the map.
(Something along the lines of:
if bit 0 only is set, the integer has the value 1, and this means show only military buildings
if bit 1 only is set, integer has value 2, and only religious buildings are displayed
at integer value 3, both these 'bools' are 'true', so only military AND religious buildings are represented.
Etc.)
These are probably not the actual uses of these bits, 8 bits in all are used although valid values only go up to about 131 (ie. 10000011 binary)
 
Yudishtira said:
I can't help you with where it is in the code, but this method is used for the setting that determines what city buildings are graphically represented on the map.
(Something along the lines of:
if bit 0 only is set, the integer has the value 1, and this means show only military buildings
if bit 1 only is set, integer has value 2, and only religious buildings are displayed
at integer value 3, both these 'bools' are 'true', so only military AND religious buildings are represented.
Etc.)
These are probably not the actual uses of these bits, 8 bits in all are used although valid values only go up to about 131 (ie. 10000011 binary)
Click to expand...
Yes, that is how you do it manually. Very commonly used while programming on microcontrollers where RAM is at a premium (on Cortex-M4 you usually have something like 128KB) and hardware peripherals are accessed with registers that pack a lot of different info or settings into 32 bits.
The easier way to get that without masking and shifting yourself is to use bit fields. In bit fields you specify for each variable how many bits it should occupy. That way you can also have integers with uncommon sizes. You access them like normal variables but the compiler will generate different code, which is usually slower. The main difference is that you can't take the address of these variables (they don't occupy a full byte after all). So for the save/load code you need to change it to load into a normal local variable before storing it into the bit field variable.
Bit fields are declared like normal variables except with the size in bits of the variable added at the end:
Code: 
struct Foo
{
  bool b1 : 1;
  bool b2 : 1;
  int veryshort : 3;
}
You should probably group all variables that get the bit field treatment within the class.
One of the disadvantages of bit fields is that a lot of it is implementation defined so the compiler is not under an obligation to really pack the variables and might sometimes align them for faster access but more space usage.
This means that you should check the real size before and after with sizeof.
 
Much of what you said there, Bit field treatment, grouping variables, Implementation defined, pack the variables, sizeof, is Greek, but I think I get the idea of the struct somewhat. The operator ':' apparently is there to define the bit space that a variable takes up? Why wouldn't bools already only be 1?

Does : only work within the span of a struct definition? Or does that work anywhere where a variable is declared?
 
I think these should work for the first idea. If anyone sees a mistake let me know.
Spoiler :

//!===================================
//! typedef int16_t (32,000)
//!===================================
int m_iX;
int m_iY;
int m_iReconX;
int m_iReconY;
int m_iAttackPlotX;
int m_iAttackPlotY;
int m_iGameTurnCreated;
int m_iLastMoveTurn;
int m_iLevel;
int m_iControlPointsLeft;
int m_iAttackFromPlotX;
int m_iAttackFromPlotY;
int m_iMADTargetPlotX;
int m_iMADTargetPlotY;
int m_iExtraWorkPercent;
int m_iExtraCityAttackPercent;
int m_iExtraCityDefensePercent;
int m_iExtraHillsAttackPercent;
int m_iExtraHillsDefensePercent;
int m_iExtraHillsWorkPercent;
int m_iExtraPeaksWorkPercent;

//!===================================
//! typedef int8_t (255)
//!===================================
int m_iCanMovePeaksCount;
int m_iCanLeadThroughPeaksCount;
int m_iKamikazePercent;
int m_iMoves;
int m_iExtraMoves;
int m_iExtraMoveDiscount;
int m_iExtraAirRange;
int m_iAmphibCount;
int m_iZoneOfControlCount;
int m_iCelebrityHappy;
int m_iOffensiveVictoryMoveCount;
int m_iOneUpCount;
int m_iBlitzCount;
int m_iRiverCount;
int m_iIgnoreZoneofControlCount;
int m_iPillageCultureCount;
int m_iPillageEspionageCount;
int m_iPillageMarauderCount;
int m_iPillageOnMoveCount;
int m_iPillageOnVictoryCount;
int m_iPillageResearchCount;
int m_iHillsDoubleMoveCount;
int m_iBaseDCMBombRange;

bit field thing sounds interesting. Ill make an attempt at some point.
 
Thunderbrd said:
Much of what you said there, Bit field treatment, grouping variables, Implementation defined, pack the variables, sizeof, is Greek, but I think I get the idea of the struct somewhat. The operator ':' apparently is there to define the bit space that a variable takes up? Why wouldn't bools already only be 1?

Does : only work within the span of a struct definition? Or does that work anywhere where a variable is declared?
Click to expand...
Any normal variable has to be addressable and you can only address bytes, not bits. That and speed is the reason that bools are at least 1 byte and not 1 bit in size. There is std::vector<bool> which actually packs the bools into bits but that is generally considered a mistake nowadays (more so because it makes it so different from the other vectors, not because it is a bad idea to pack bools into bits in large vectors).
But yes, the number after ':' specifies the number of bits assigned to that variable within the bit field. You can only have bit fields within structs or classes.
 
Snofru1 said:
Please have this discussion in its own thread! Here it will never be found anymore in a few months or years.
Click to expand...
Was thinking similar thoughts about that. I'd appreciate it if someone could take the time to quote this tech discussion into a new thread. I also hope AIAndy would be kind enough to continue the discussion there because I have lots more questions about this stuff and am currently reluctant to get too deep into this here. Thank you very much for your guidance so far @AIAndy - I'm surprised by how much this is making sense actually. Very interesting stuff to know. I'm also very wary of trying to implement any of it for fear of stepping into complex debugging territory where I'd want to just rage quit the whole project over it. lol. I'm not budgeting as much time and dedication to the project these days so small investments of effort that can make impacts here and there without being likely to cause a huge problem are certainly of interest but sweeping restructures are... intimidating.
 
Hunting currently mammoths in my current game, they found DNA on the best conserved mammoth and one guy eats the 40.000 years old flesh of it. :crazyeye: Possible to clone them now.

German doku:


English doku:

 
I want to do a small mod mod, well i can do it manually... but any idears how to make it automatic by a small mod mod?

What i will do is that if the tech seafaring is researched all wodden ships should be able to travel sea and ocean plots. (Except rafts)

I can manually edit the ships by hand if i researched seafaring, but is there another way to remove all the movement restrictions of some units by only editing the tech seafaring? (Will be cooler if the AI researchs the tech seafaring earlyer than the gamer and so the ships of the AI can go earlyer to sea and ocean plots)
 
Last edited:
Pit2015 said:
I want to do a small mod mod, well i can do it manually... but any idears how to make it automatic by a small mod mod?

What i will do is that if the tech seafaring is researched all wodden ships should be able to travel sea and ocean plots. (Except rafts)

I can manually edit the ships by hand if i researched seafaring, but is there another way to remove all the movement restrictions of some units by only editing the tech seafaring? (Will be cooler if the AI researchs the tech seafaring earlyer than the gamer and so the ships of the AI can go earlyer to sea and ocean plots)
Click to expand...
There's a tag on Seafaring, and the other tech that enables sea travel, that can be moved to being used on earlier techs if you wish.
 
Thunderbrd said:
There's a tag on Seafaring, and the other tech that enables sea travel, that can be moved to being used on earlier techs if you wish.
Click to expand...

We had that discussion before, and there was no tag, is there a tag now or where is that tag? Maybe changing the units manually still the only way? I looked into CIV4Techinfos.xml still so:

<TechInfo>
<Type>TECH_ASTRONOMY</Type>
<Description>TXT_KEY_TECH_ASTRONOMY</Description>
<Civilopedia>TXT_KEY_TECH_ASTRONOMY_PEDIA</Civilopedia>
<Strategy>TXT_KEY_TECH_ASTRONOMY_STRATEGY</Strategy>
<Advisor>ADVISOR_GROWTH</Advisor>
<iCost>2528</iCost>
<Era>ERA_RENAISSANCE</Era>
<iAsset>32</iAsset>
<iPower>4</iPower>
<bTrade>1</bTrade>
<iGridX>52</iGridX>
<iGridY>15</iGridY>
<Flavors>
<Flavor>
<FlavorType>FLAVOR_SCIENCE</FlavorType>
<iFlavor>7</iFlavor>
</Flavor>
<Flavor>
<FlavorType>FLAVOR_CULTURE</FlavorType>
<iFlavor>3</iFlavor>
</Flavor>
<Flavor>
<FlavorType>FLAVOR_GROWTH</FlavorType>
<iFlavor>5</iFlavor>
</Flavor>
<Flavor>
<FlavorType>FLAVOR_RELIGION</FlavorType>
<iFlavor>1</iFlavor>
</Flavor>
</Flavors>
<OrPreReqs>
<PrereqTech>TECH_ASTROLABE</PrereqTech>
</OrPreReqs>
<Quote>TXT_KEY_TECH_ASTRONOMY_QUOTE</Quote>
<Sound>AS2D_TECH_ASTRONOMY</Sound>
<SoundMP>AS2D_TECH_MP_ASTRONOMY</SoundMP>
<Button>,Art/Interface/Buttons/TechTree/Astronomy.dds,Art/Interface/Buttons/TechTree_Atlas.dds,7,1</Button>
</TechInfo>

No tag there. So maybe in another file or its bound to units only.

Look here:

https://forums.civfanatics.com/thre...n-update-compatible-by-pit2015.552901/page-63
 
Last edited:
@Pit2015 That's because it's not Astronomy, it's Navigation. Assets\XML\Technologies\CIV4TechInfos.xml, line 12418 - 12443:

<TerrainTrades>
<TerrainTrade>
<TerrainType>TERRAIN_OCEAN</TerrainType>
<bTerrainTrade>1</bTerrainTrade>
</TerrainTrade>
<TerrainTrade>
<TerrainType>TERRAIN_OCEAN_POLAR</TerrainType>
<bTerrainTrade>1</bTerrainTrade>
</TerrainTrade>
<TerrainTrade>
<TerrainType>TERRAIN_OCEAN_TROPICAL</TerrainType>
<bTerrainTrade>1</bTerrainTrade>
</TerrainTrade>
<TerrainTrade>
<TerrainType>TERRAIN_TRENCH</TerrainType>
<bTerrainTrade>1</bTerrainTrade>
</TerrainTrade>
<TerrainTrade>
<TerrainType>TERRAIN_TRENCH_POLAR</TerrainType>
<bTerrainTrade>1</bTerrainTrade>
</TerrainTrade>
<TerrainTrade>
<TerrainType>TERRAIN_TRENCH_TROPICAL</TerrainType>
<bTerrainTrade>1</bTerrainTrade>
</TerrainTrade>
</TerrainTrades>
 
tmv said:
@Pit2015 That's because it's not Astronomy, it's Navigation. Assets\XML\Technologies\CIV4TechInfos.xml, line 12418 - 12443:

<TerrainTrades>
<TerrainTrade>
<TerrainType>TERRAIN_OCEAN</TerrainType>
<bTerrainTrade>1</bTerrainTrade>
</TerrainTrade>
<TerrainTrade>
<TerrainType>TERRAIN_OCEAN_POLAR</TerrainType>
<bTerrainTrade>1</bTerrainTrade>
</TerrainTrade>
<TerrainTrade>
<TerrainType>TERRAIN_OCEAN_TROPICAL</TerrainType>
<bTerrainTrade>1</bTerrainTrade>
</TerrainTrade>
<TerrainTrade>
<TerrainType>TERRAIN_TRENCH</TerrainType>
<bTerrainTrade>1</bTerrainTrade>
</TerrainTrade>
<TerrainTrade>
<TerrainType>TERRAIN_TRENCH_POLAR</TerrainType>
<bTerrainTrade>1</bTerrainTrade>
</TerrainTrade>
<TerrainTrade>
<TerrainType>TERRAIN_TRENCH_TROPICAL</TerrainType>
<bTerrainTrade>1</bTerrainTrade>
</TerrainTrade>
</TerrainTrades>
Click to expand...
That's only for trade, it doesn't affect when specific ships can use those terrains.
To change when a ship can move certain terrains one would have to modify each specific unit in CIV4UnitInfos.xml or other xmls where they may be defined.
 
Toffer90 said:
That's only for trade, it doesn't affect when specific ships can use those terrains.
To change when a ship can move certain terrains one would have to modify each specific unit in CIV4UnitInfos.xml or other xmls where they may be defined.
Click to expand...
Indeed. One example is Civ4UnitInfos.xml, line 31134 - 31167:
Spoiler Terrain modification for the Galleass :

<TerrainImpassableTypes>
<TerrainType>TERRAIN_OCEAN</TerrainType>
<TerrainType>TERRAIN_OCEAN_POLAR</TerrainType>
<TerrainType>TERRAIN_OCEAN_TROPICAL</TerrainType>
<TerrainType>TERRAIN_TRENCH</TerrainType>
<TerrainType>TERRAIN_TRENCH_TROPICAL</TerrainType>
<TerrainType>TERRAIN_TRENCH_POLAR</TerrainType>
</TerrainImpassableTypes>
<TerrainPassableTechs>
<TerrainPassableTech>
<TerrainType>TERRAIN_OCEAN</TerrainType>
<PassableTech>TECH_ASTRONOMY</PassableTech>
</TerrainPassableTech>
<TerrainPassableTech>
<TerrainType>TERRAIN_OCEAN_POLAR</TerrainType>
<PassableTech>TECH_ASTRONOMY</PassableTech>
</TerrainPassableTech>
<TerrainPassableTech>
<TerrainType>TERRAIN_OCEAN_TROPICAL</TerrainType>
<PassableTech>TECH_ASTRONOMY</PassableTech>
</TerrainPassableTech>
<TerrainPassableTech>
<TerrainType>TERRAIN_TRENCH</TerrainType>
<PassableTech>TECH_ASTRONOMY</PassableTech>
</TerrainPassableTech>
<TerrainPassableTech>
<TerrainType>TERRAIN_TRENCH_TROPICAL</TerrainType>
<PassableTech>TECH_ASTRONOMY</PassableTech>
</TerrainPassableTech>
<TerrainPassableTech>
<TerrainType>TERRAIN_TRENCH_POLAR</TerrainType>
<PassableTech>TECH_ASTRONOMY</PassableTech>
</TerrainPassableTech>
</TerrainPassableTechs>

Edit: If you want to change all of these tags, grepWin is your friend. Just look for the string "<PassableTech>TECH_ASTRONOMY</PassableTech>", and for your own version replace it with "<PassableTech>TECH_SEAFARING</PassableTech>".

This tag is only in Civ4UnitInfos.xml (25 times), so this should be an easy replacement.

Spoiler German version of grepWin :
AstronomyReplace.png
 
Last edited:
You must log in or register to reply here.

Similar threads

The_J
  • Sticky
Old World update #123
Replies
2
Views
346
Dale
Dale
Pit2015
[MODCOMP] RELEASED: C2C - Ultimate-Earth-Map V5.6 100% MOD and SVN update compatible.
Replies
4
Views
4K
Pit2015
Pit2015
The_J
Old World update #122
Replies
0
Views
249
The_J
The_J
The_J
Old World update #121
Replies
3
Views
638
Solver
Solver
The_J
Old World update #120
Replies
8
Views
1K
The_J
The_J
Top Bottom