[MOD] Colonization: 2071

orlanth · Sep 28, 2013

Ok, good to know. I guess I haven't figured out the right way to update Master branch DLL on github separately from the 2071 branch. My local folder using github's windows application to sync contains the 2071 branch & I can't find a way to have two local folders for branches of the same repo, so I edited the master branch directly via the web which I guess isn't ideal since that results in one commit per file edit.

It loads to the main map without further asserts, but then shows the bar of yields along the bottom like in the cityscreen (perhaps since it got hung up trying to set prices for these?)
M:C does the same thing. I haven't found the cause yet, but once the game is started it works just fine. It's a minor graphical glitch, which will not cause any problems later on.

Ok. In the 2071 branch currently, the map screen's yield bar never goes away so you can't access anything from the usual interface. On ending turns, it again asserts for price setting; then gives the following assert which I think we first saw with the initial "skeleton" version

Assert Failed
File: CvPlayerAI.cpp
Line: 6163
Expression: pBestYieldCity != NULL

(I'm thinking this may be related to having few/no Terrain yields in the base version, if you agree I'll plan on adding those next).

I want to make it ignore uninvented professions, which isn't that hard to do. It should be done at the top and using a cache to make it really fast and neither are tricky.

However skipping as many professions as possible would be the best option and then I started wondering. Natives never use a bunch of "Eurpoean" professions and vice versa. If we tell the cache that those never used professions aren't invented, then we would save some time. In fact I think we could save quite a lot of time. It would also be easier to code if we have a function canUseProfession() for each player and it tells if it can be used, but not why it can't be used right now.

The problem with this approach is that it should work in ProfessionInfos and CivilizationInfos (possibly UnitInfos as well if it can't be avoided). Where does it say which civ can use which profession?

That's handled by <Professions> in civilizationinfos AFAIK (I think all profs are enabled by default, so the civ lacks access to only those with <bvalid>0 .
Professioninfos also has tags:
<bNativesInvalid>1</bNativesInvalid>
<bEuropeInvalid>1</bEuropeInvalid>
Profs can also be enabled by civicinfos as you know, and specific units can have <ProfessionsNotAllowed/>, but if you're looking to make a civ-specific check I think the civilizationinfos and professioninfos tags should be it.

About the Shrine and Trading Post, they currently add +3 of every yield (you can see all the building stats now in the 2071 Pedia, though the yield icons are wonky).

orlanth · Sep 28, 2013

Ok, I've tried to update the cpp and h files to make yieldgroups consistent for everything and include all the yields. But after building the DLL, on loading it still gives a bunch of asserts like the following:

File: Yields.cpp
Line: 45
Expression: !this->function(eYield)
Message: Yield YIELD_ACTINIDES is a false positive in YieldGroup_AI_Raw_Material

even though on looking at the cpp and header, it seems like Raw Material yieldgroup should match appropriately. Could you take a look at these and tell how I went wrong? :confused:

Nightinggale · Sep 28, 2013

orlanth said:
Ok, good to know. I guess I haven't figured out the right way to update Master branch DLL on github separately from the 2071 branch. My local folder using github's windows application to sync contains the 2071 branch & I can't find a way to have two local folders for branches of the same repo, so I edited the master branch directly via the web which I guess isn't ideal since that results in one commit per file edit.

You just clone the repo twice (once for each branch) and place them in two different locations. They will work independently, except that it copies the DLL from one to the other using YOURMOD path.

It would be best if you compile your edits before committing. Otherwise you risk committing code, which can't be compiled.

orlanth said:
Ok. In the 2071 branch currently, the map screen's yield bar never goes away so you can't access anything from the usual interface. On ending turns, it again asserts for price setting; then gives the following assert which I think we first saw with the initial "skeleton" version

They are supposed to go away when the python error appears. Fixing that error will make them go away (presumably).

orlanth said:
(I'm thinking this may be related to having few/no Terrain yields in the base version, if you agree I'll plan on adding those next).

That is not unlikely. It claims to be unable to figure out what the yields to produce in the city plot.

orlanth said:
That's handled by <Professions> in civilizationinfos AFAIK (I think all profs are enabled by default, so the civ lacks access to only those with <bvalid>0 .
Professioninfos also has tags:
<bNativesInvalid>1</bNativesInvalid>
<bEuropeInvalid>1</bEuropeInvalid>
Profs can also be enabled by civicinfos as you know, and specific units can have <ProfessionsNotAllowed/>, but if you're looking to make a civ-specific check I think the civilizationinfos and professioninfos tags should be it.

Ahh there. That shouldn't be hard to implement. It would be nice to check civ restrictions, native restrictions and "if invented" with just one function call, specially since this function is virtually instant.

orlanth said:
About the Shrine and Trading Post, they currently add +3 of every yield (you can see all the building stats now in the 2071 Pedia, though the yield icons are wonky).

I get a lot of smilies in the pedia as well as a bunch of clones :lol:

I'm not sure why it behaves this way, but I have an idea of where to look. Besides the pedia page on the shrine itself is broken.

Nightinggale · Sep 28, 2013

orlanth said:
even though on looking at the cpp and header, it seems like Raw Material yieldgroup should match appropriately. Could you take a look at these and tell how I went wrong?

It's on github as well.

cpp said:
void Check_YieldGroup_AI_Sell_To_Europe::build()
{
YieldVector.push_back(YIELD_SILICATES);
}

h said:
static inline bool YieldGroup_AI_Sell_To_Europe(YieldTypes eYield)
{
return eYield == YIELD_STATE_SECRETS;
}

The game assumes you want the AI to sell YIELD_SILICATES to Europe. It tries all yields with YieldGroup_AI_Sell_To_Europe() and find that it returns false for YIELD_SILICATES. On the other hand it returns true for YIELD_STATE_SECRETS, which isn't mentioned in the cpp file and as such is a false positive.

correct h said:
static inline bool YieldGroup_AI_Sell_To_Europe(YieldTypes eYield)
{
return eYield == YIELD_SILICATES;
}

The idea with this split (as I think I mentioned earlier) is that the header can be optimized to be almost unreadable to humans while the cpp file always list all the included yields. Exploiting that YIELD_* are just a bunch of numbers (from a compiler point of view) we can write code like

return (eYield >= YIELD_CRYSTALLOIDS && eYield <= YIELD_CORE_SAMPLES) || eYield == YIELD_ALIEN_RELICS

This is true for the 3 yields mentioned and every yield between YIELD_CRYSTALLOIDS and YIELD_CORE_SAMPLES in the enum. This mean there are 16 positive yields in this group, but at runtime we only need 2 or 3 comparisons to figure out if a random yield is in the group. You can write

return eYield == YIELD_CRYSTALLOIDS || eYield == YIELD_NUCLEIC_ACIDS || ...

It will be slower at runtime, but it demands less coding skill and it will work. We can always optimize later once all groups are done. In other words don't think too much about how to write this as fast as possible right now.

orlanth · Sep 28, 2013

I see that, but in message #342 above I had worked further on correcting them using your previous comments and was posting about the revised DLL sources attached to that message (didn't sync those to github yet since they're not yet working correctly). In these, it looks to me like the yieldgroups should now match appropriately yet I'm still getting asserts after compiling and am unsure of why. :confused:

Spoiler :

Nightinggale · Sep 28, 2013

I think I will try to write a perl script to generate the correct (and somewhat optimized) header check for each group. I have an idea on how it might work.

orlanth · Sep 28, 2013

I think I will try to write a perl script to generate the correct (and somewhat optimized) header check for each group. I have an idea on how it might work.

I just saw in the log that you already made a script creating EmphasizeInfos xml - I shouldve asked before adding my own for that just now. :crazyeye:

They are actually not too hard to make for xml files once you get the hang of it. What I do is paste the desired XML block into Textpad, then use regexp find/replace to replace leading tabs with a filehandle print statement with the appropriate number of leading tabs; you can then adjust the individual xml tags to prepopulate with variables however you want :cool:

.

I synced a commit generating Terrains xml to provide a variety of yields, and finishing dds art for Yields; but after turn end there are 2 asserts followed by CTD:

Spoiler :

Nightinggale · Sep 28, 2013

I committed a perl script (scripts/yield.pl) to generate the return line. It's in master branch as that's where we handle C++ changes anyway and it has hardcoded relative paths to find C++ files from where it is located now.

It's fairly simple to use. It takes arguments in any order and right now listens to

2071: use input/output for Colonization 2071 instead of Medieval Conquest
YieldGroup* or Check_YieldGroup*: The return line is generated based on the group in the cpp file

I didn't actually change any source file, but while testing it I copied a bit from your previous post and ran

Code:

./yield.pl 2071 Check_YieldGroup_AI_Sell_To_Europe
./yield.pl Check_YieldGroup_AI_Sell_To_Europe 2071
./yield.pl 2071 YieldGroup_AI_Sell_To_Europe
./yield.pl YieldGroup_AI_Sell_To_Europe 2071

(Any of those lines will do as the output is the same)
The output from that is

Code:

return (eYield <= YIELD_CORE_SAMPLES && eYield >= YIELD_ACTINIDES) || (eYield >= YIELD_PHOTONICS && eYield <= YIELD_HARD_CURRENCY);

I tried a number of different combos and while I can make up scenarios where I can optimize a bit better by hand I say 70% of the time it does as good as human spending time on it and 20% it spots additional (but minor) optimization the human missed. I think we can live with the remaining 10% and save our minds for other tasks

I think we should add new scripts to this new directory as it's a clean location to keep them and it allows us to share scripts. Also this script can get more features easily as it reads the command line arguments. It starts by reading the header file and create a hash and an array which allows moving between index and string (both directions). Something which is extremely useful for tasks like the one I just finished. We could reuse it for other tasks in the future.

As for the new asserts.... I don't know without investigating those. They aren't in any code I wrote.

EDIT: looks like assert 2 is a result of assert 1. The problem appears that it reads yields off the land and it may very well be using AI code to value land to find the best plot. A yield, which aren't in a yieldgroup is valued to 0 and as a result the whole plot can end up having the value 0. The asserts shows up because the value of the best found plot is 0. However it isn't clear to me precisely where the code for assigning the land value is located. It might even be in python somewhere as it calls gDLL->getPythonIFace()->callFunction(PYGameModule, "getCityFoundValue", argsList.makeFunctionArgs(), &lResult);

Nightinggale · Sep 28, 2013

The perl script now supports two new commands

python
checkXML

Using either will tell where in the C++ code the output should be pasted into. This should be done whenever the enum is changed.

Again this works for both M:C and 2071.

Nightinggale · Sep 29, 2013

Now I think I'm done with perl for now. I added two more arguments:

makegroups: loops though all groups to generate return code.
newyield: calls both checkXML and python.

Both these commands writes the output directly to the source files instead of the screen. I can't think of an easier to use solution than this. In fact these two new arguments renders the other obsolete, though they are still there if you need them. Also they share code meaning the output should be the same.

orlanth · Sep 29, 2013

cool

I used your scripts to generate updated DLL, now there are no remaining asserts on loading (except for the one about Duralloy being Armor but not Light or Heavy Armor; and I'm guessing those groups may no longer be needed anyway once the combat system is revised). On starting a game, there are still price setting asserts though.

I updated the scripts to run from the /scripts dir as you suggested and extended the xmlgen script to generate xml for Features. We're getting close to the point where most of the core xml framework is generated and successfully loaded by the game. :king:

On the downside, it looks like the current AiWeights assert and crash wasn't fixed by making more yields available through Terrains and Features. I also added nominal aiweights to the standard units on the off chance that that was involved, but it doesn't look like that matters. I'm also currently unable to build a city in the current version; this may just be because the interface isn't present, though the B hotkey also doesn't do it.

I don't think I would've thought of using Perl to actually write the c++ code, but it seems a natural use of it to create other programs. It's like this book Higher Order Perl that I was too intimidated to buy

and is now available free online. (I think I had best focus on learning Lower Order Perl for the time being..) :crazyeye:

Nightinggale · Sep 29, 2013

orlanth said:
cool I used your scripts to generate updated DLL, now there are no remaining asserts on loading (except for the one about Duralloy being Armor but not Light or Heavy Armor; and I'm guessing those groups may no longer be needed anyway once the combat system is revised). On starting a game, there are still price setting asserts though.

You need to set up the yield groups in Yields_Colonization_2071.cpp as the game uses the groups to estimate plot value (I think. They are used for yield value at some places). This will also remove the armor assert. In fact at this point it might be best to leave all 3 armor groups empty.

I see you moved the source code into the 2071 branch. This is precisely what I DIDN'T want as it means maintaining two source codes. As said way back you check out the master branch somewhere and edit source files in that one and you set it to copy the compiled DLL into 2071. I thought you already had this running, but it looks like it's giving you problems now :confused:

orlanth said:
I don't think I would've thought of using Perl to actually write the c++ code, but it seems a natural use of it to create other programs. It's like this book Higher Order Perl that I was too intimidated to buy and is now available free online. (I think I had best focus on learning Lower Order Perl for the time being..)

Generating the same text over and over is ideal for scripting. In this case the target happened to be C++ code. I never edited individual lines in a file, but it's actually fairly simple. It would be pure hell in bash though. In fact I think I'm dropping bash. I figured out some more about perl and now there are less reasons to use bash. Also I would say perl is like 20 times faster. The bash script to make a wiki page takes as long as a complete recompile of M:C.

I never read any perl books and this is like my 4th script or something like that. Maybe I should read that book.

orlanth · Sep 29, 2013

Ok, I committed updated DLL with 2071 yields to the master branch. I was a little afraid of messing with the master branch DLL and breaking it, but that shouldnt be a concern since it's all using ifdef for 2071. Do you think I should delete the DLL_Sources folder from the 2071 branch?

I took yields out of Armor groups for now as you suggested, and edited the cpp and reran your script so that all the yields should now belong to groups. It now loads without any asserts :cool:

, but there are still the price setting asserts on starting a game and the aiweight assert and crash was again totally unaffected. It looks to me like all yields are assigned to groups in cpp and h files; I'm not sure what's causing those asserts but wonder if it may be something unrelated to these groups. :confused:

Looking at the DLL sources now, do they look correct to you?

I never read any perl books and this is like my 4th script or something like that. Maybe I should read that book.

It looks like useful stuff if definitely way over my head atm and intended for hard-core CS people. I've been almost totally away from Perl & any programming for years; when I was first learning around 2000 or so it was allegedly going to be an increasingly major language that everyone would use for CGI scripts etc, as well as parsing genomic data which is what I was using it for (it's still good for that since the data is essentially huge swaths of text.) Now, it seems its generally being used a lot less widely, and Larry Wall is focused on an essentially separate language in Perl 6 which I've really never encountered. I wonder if I should still try to learn and use Perl 5 as I had been or move on to something else.

Kailric · Sep 29, 2013

Nightinggale said:
I looked into how armor is made and found a number of issues.

All armors have their own UnitCombatInfo. I don't think that was the intension of that class. It's more like a class giving details about say mounted, which in turn is used to give bonus vs mounted. It's just basic info meaning they really don't have much to them at all.

CIV4UnitCombatInfos fails to add Type into the storage, which getDefineINT() reads from (vanilla bug or intended?). This is both good and bad. Good because the less defines we have, the faster getDefineINT() is. Bad because it mean lines like GC.getDefineINT("UNITARMOR_LEATHER") returns 0 instead of the expected 9. Some of the C++ code is broken because of this. Luckily it is just the Pedia setup code, no real ingame bugs.

I propose that we get rid of a most of the UnitCombatInfos. Instead we add the info to ProfessionInfo.
UNITTACTIC_PARRY, UNITARMOR_SHIELD, UNITWEAPON_BLUNT
Those appears to be just bools, which we can add to professions.

Crushing Blow and Glancing Blow are also bools, which could be added to profession.

However I came up with another idea. We could add such bools to yields. That way we could make the game give those features to any unit, which requires YIELD_PLATE_ARMOR. This mean it can automatically give those bonuses to units for alt equipment without setting up anything than just yield requirements.

Both solutions mean we can get rid of any mentioning of specific armors in the DLL. We can also allow units to use multiple armors at once if we like, say using YIELD_TARGETING_COMPUTER and YIELD_EXPLOSIVE_AMMO at the same time or whatever. It gives a high degree of modding freedom without changing the DLL.

Alternatively we can move all the code to free promotions. This would be the easiest solution from a coding point of view. However I'm not sure I like it as we will end up with a lot of promotions even units with 0 xp and units with experience will be harder to spot.

What do you guys say to this?

Also I realized that while yield requirements for professions are cached, yield requirements for alt equipment is uncached. Luckily that shouldn't be too difficult to fix and with a noteworthy speed boost to follow.

I noticed one potential bug/limitation in the yield equipment cache. If the profession needs no yields for the base, then the game will assume the alternative also uses no yields for performance reasons. The easy solution is not to make a profession with that combo. I might fix this as I make a cache for the alt equipment anyway.

It will work for yields such as Plate Armor but for things like UNITARMOR_SHIELD, UNITWEAPON_BLUNT or any new attribute not yield related there will need to be a Boolean in the Professions XML, that could be a slew of new booleans when we revisit this part of the game to expound on it. I did CombatGearTypes with CombatTypes because it was simple to do and you can easily add new options. This code is only called during combat or in the pedia. So, I don't see performance being much of an issue either way you do it. If you make new booleans to the Professions then we would need to create a new boolean with any new option we wanted to create. That means editing the XML and CvInfos.cpp and all related files to add the new boolean. With the CombatGearTypes we just add a new CombatType and then add code to DLL for that CombatType, which like I said will only be called during combat.

orlanth · Sep 29, 2013

I agree it's good to expose these combat features to boolean tags rather than hardcoded inside DLL (e.g. <bDeflect> allowing you to set armor deflection on or off, or even instead use <iDeflect> with a percent chance of deflection), but if these tags need to be made anyway, the best place to put them would definitely be Promotions rather than Yields or Professions. You could then set Professions to inherit these promotions however desired, including the existing way or any other potential way. In 2071, there would be lots of places where I'd want to assign a combat feature not necessarily firmly linked to a certain Profession or Yield. For example, promotions having combat effects could be assigned to ships and vehicles depending on techs or unitclass, without them having to adopt specific professions to do so (which doesn't make much sense anyway :lol:

).

Nightinggale · Sep 29, 2013

orlanth said:
Ok, I committed updated DLL with 2071 yields to the master branch. I was a little afraid of messing with the master branch DLL and breaking it, but that shouldnt be a concern since it's all using ifdef for 2071. Do you think I should delete the DLL_Sources folder from the 2071 branch?

I was thinking the same thing. There is no point in having unused source lying around. It's a temptation for future mistakes.

orlanth said:
Looking at the DLL sources now, do they look correct to you?

I think I better go though finding the start plot in the debugger to see what goes on here. A brief look at the code didn't reveal anything.

orlanth said:
I wonder if I should still try to learn and use Perl 5 as I had been or move on to something else.

Java was big and the future around year 2000. So much that I learned it because it became mandatory at the university. I never really used it and now Oracle is try make money out of companies stuck with Java. My guess is that I will never use java. C/C++ on the other hand was outdated in 2000 (some people claimed that), yet I wanted to learn it anyway and C++ is still the language of choice in the coding industry if performance is an issue, though firmware and to some extend microcontrollers mainly uses C.
It's hard to figure out which language to learn, but perl isn't a bad one. The thing is that learning the first is the hardest. Learning the next is way easier, because a lot of it is the same even though you need some other keywords and stuff. An array is still an array etc. If I were to pick just one language, then I would likely pick perl as you learn how to code at the same time you get the ability to write some rather good scripts. Perl supports some rather advanced stuff like TCP/IP connections. Doing that in another scripting language like bash would demand writing a C application to handle that and then execute that app from the script.

Perl is claimed to be around 3 times faster than python when doing the same thing.

Kailric said:
It will work for yields such as Plate Armor but for things like UNITARMOR_SHIELD, UNITWEAPON_BLUNT or any new attribute not yield related there will need to be a Boolean in the Professions XML, that could be a slew of new booleans when we revisit this part of the game to expound on it. I did CombatGearTypes with CombatTypes because it was simple to do and you can easily add new options. This code is only called during combat or in the pedia. So, I don't see performance being much of an issue either way you do it. If you make new booleans to the Professions then we would need to create a new boolean with any new option we wanted to create. That means editing the XML and CvInfos.cpp and all related files to add the new boolean. With the CombatGearTypes we just add a new CombatType and then add code to DLL for that CombatType, which like I said will only be called during combat.

The point is to get rid of the hardcoded types in the DLL as it messes up with 2071. However I noticed something else. UnitCombatInfo lacks it's own type in C++ and is instead of type CvInfoBase. This mean it actually has Civilopedia meaning the hardcoded text in C++ Pedia code could just read CvInfoBase from XML.

However I wonder about another thing. We could give it a type and move the file into XML\Units and use the schema file there. That way we can add bools and other data to the combat types themselves.

Nightinggale · Sep 29, 2013

orlanth said:
the best place to put them would definitely be Promotions rather than Yields or Professions.

The only thing I have against that approach is that free promotions shows up as promotions. If we have too many free ones, the earned ones though experience will disappear in the crowd. It's a visual rather than technical issue.

orlanth · Sep 29, 2013

True, but it also has the advantage of making it more clear what bonuses each unit has. Also it's likely possible to limit what promotions are displayed using python if that's needed, so that only non-free ones show up as icons. (IIRC some Civ4 mods such as FfH and Master of Mana use very many promotions per unit, they may have used python to address that.) Also in 2071 at least, some of the combat features could actually work best as regular earnable promotions rather than free ones, so that wouldn't add to the amount displayed.

Nightinggale · Sep 29, 2013

I updated the yield and set the AI to sell all "unknown" yields to Europe. That way all asserts regarding yields are gone.

I added a new assert "iMaxStored != 0". The problem here is that it calculates a yield modifier and divides with iMaxStored. To get around that crash I made the code skip the modifier code and added this assert as a reminder. It can be ignored for now.

I'm not sure it helped a lot ingame though. It complains about out of index in python. Now if we could fix the python issue, then we would get main screen up and running and the same goes for city screen.

EDIT: found the cause... well sort of. CvMainInterface.py line 1185. It reads BUILDING_GRID[iSpecial][1] and iSpecial is set to getNumSpecialBuildingInfos() while BUILDING_GRID has a python hardcoded size. My guess is that 2071 has more special buildings than M:C and as a result it reads something outside of the array. Now how to fix this nicely....

Line 145 to 178 is the setup, which needs some 2071 duplicate. We can just add it and called it something else and that way have two setups. The array is populated at line 430. I added the functions GC.isColonization2071() and GC.isMedievalConquest() as I predicted something like this would happen.

EDIT2: yes fixing this problem removes the python bug and I get the correct main screen. I can also visit the city screen. Sadly this doesn't help on the asserts. Now I need to figure out how to write this in a way which will not break M:C. I think I have the solution, but I ran out of time now. I will have to test it later.

As for promotions. How about having two icons for each promotion? One for xp and one for free. If they are the same except for the background color (say blue and yellow), then we can detect experience easier.

orlanth · Sep 29, 2013

wow that was fast :cool:

I should've known it might be the number of buildings.. the current 2071 has about the same number of yields and specialbuildings as RaR, so hopefully performance and screen space won't be too bad once everything's working. But I definitely shouldn't add any more yields (that's another reason having combat effects not be inextricably linked to yields will be good).

As for promotions. How about having two icons for each promotion? One for xp and one for free. If they are the same except for the background color (say blue and yellow), then we can detect experience easier.

Sure that would be possible to do; with a batch macro in photoshop it would be fast to make alternate color versions of each button. OTOH the number of free promotions will likely be very small compared to the regular ones and will already look recognizably different, so the regular ones wouldn't need any changing. E.g for M:C, units have at most one Armor type which could use the existing Armor icon for that yield, and have at most 1-2 Weapon or Shield effects which can use buttons that also look obviously distinct from the blue experience promotion badges.

[MOD] Colonization: 2071

Storm God. Yarr!

Storm God. Yarr!

Deity

Deity

Storm God. Yarr!

Deity

Storm God. Yarr!

Deity

Deity

Deity

Storm God. Yarr!

Deity

Storm God. Yarr!

Jack of All Trades

Storm God. Yarr!

Deity

Deity

Storm God. Yarr!

Deity

Storm God. Yarr!

Similar threads