Advanced Civ

[keldath]

Rawwwrr
make sure you tagged all the logs on in both advc GlobalDefines_devel.xml and the civ ini file.​

 

[Rawwwrr]

Rawwwrr​

make sure you tagged all the logs on in both advc GlobalDefines_devel.xml and the civ ini file.​

Everything is included in civ ini, I didn't know what else could be included in GlobalDefines_devel.xml , I will make and post a fresh magazine. By the way, can you tell me what exactly to include in GlobalDefines_devel.xml ? Or is it better to replace 0 with 1 on all parameters?

 

[Rawwwrr]

Logs with enabled GlobalDefines_devel.xml (In any case, I changed all 0 to 1 there). Using catapults (removing the percentage of protection) very often leads to a message about desynchronization. Moreover, an error occurs if it is used by even a human, even an AI (Not 100% sure)

 

[keldath]

hey,
you need also the logs from the other player, sorry, i forgot to mention it.the logs can be compared.

not sure i get what you mean here:
Using catapults (removing the percentage of protection) .
you mean bombard?

--
maybe something in this code:

m_iSearchRangeRandPercent = syncRand().get(101, "SearchRangeRand",
getX() * 1000 + getY(), getID()); // </advc.128>
if (getDomainType() == DOMAIN_LAND)

---
do you play on a different language than English?
theres some weird codecs errors. try to play in english translation.

 

[Rawwwrr]

hey,
you need also the logs from the other player, sorry, i forgot to mention it.the logs can be compared.

ok, next time I'll do it

you mean bombard?

Yes

do you play on a different language than English?

Yes, we play in Russian. It's problematic for us to play in English, but maybe I'll try

 

[keldath]

try in english,
just try to recrate the actions.
i have a hunch on this.

 

[f1rpo]

@Rawwwrr:

I have applied the fix you recommended for OOSLogger.py There is no encoding error, but the desynchronization warning continues.

Did you add the import codecs to OOSLogger.py? PythonErr.log says "global name 'codecs' is not defined" and that exception prevents most of OOSLog from getting printed. (Or maybe the codecs module doesn't exist in Python 2.4; but the answer to this StackOverflow question suggests that it does ...) Although, as keldath wrote, the full OOSLog from only a single player is probably not going to help either. Same for MPLog.txt. The last messages in the copy you've uploaded deal with the randomization of AI search ranges, i.e. have been printed during AI unit moves. But game states may well have diverged earlier than that. The second upload also ends with AI units moves. And in both cases, AI spies are moved near the end of the log. And it's curious that the OOS issue sometimes self-corrects. Still too little for me to go on ...

Do you have a savegame from which the issue can be reproduced with some trial and error? What you write about bombardment sounds like it should be somewhat reproducible. That would really be my best bet for diagnosing the problem. If I find the time, I might run a network game on a single machine on AI Auto Play for some 250 turns and then try bombarding some cities; maybe I'll stumble upon the issue that way.

I don't think changing anything in GlobalDefines_devel.xml on your end will help. Doesn't sound like the UWAI component is part of the problem, so I wouldn't recommend enabling the log files for that. And ENABLE_DEBUG_TOOLS_MULTIPLAYER and PER_PLAYER_MESSAGE_CONTROL_LOG are settings that I'll want to use if I can reproduce the issue on a single machine, but you don't need to have them enabled.

maybe something in this code:
m_iSearchRangeRandPercent = syncRand().get(101, "SearchRangeRand",
getX() * 1000 + getY(), getID()); // </advc.128>
if (getDomainType() == DOMAIN_LAND)

The search range gets randomized whenever an AI unit moves, so these rolls show up in the MPLog a lot. I don't think it indicates a problem.

 

[f1rpo]

so yes, i used KB_Z, i cant see in all the files usage for it , so its valid i guess,

So you were pressing X, but the game saw Z and you weren't able to fix that and therefore changed it to Z in Python? As a German using the QWERTZ keyboard layout I'm very much used to Z and Y getting mixed up, but Z and X is puzzling.

btw remember when i said my load takes time -> not with this version. another oddity.

It seemed that XML loading specifically had become a little slow iirc. Is that still your impression, i.e. slower XML loading with the Steam version?

also,
after almost 20 years of civ4, i finally bother to find how to print stuff from the python to the debug file

*smh* I often try to nudge users toward doing that when they ask why their Python code "does nothing," but somehow people have a real aversion to tracing their Python code. They usually just keep trying things blindly. Edit:

im trying to see log prints for unit stuck in a loop.
i turned on all the logs in the config ini.
i used crtl +z and also removed the _debug check for the loop print.
but i dont see the log / log enrty with that name.

suggestions?

MessageLog should be needed (CivilizationIV.ini) and it should write to MPLog.txt. Ah, and a Debug build, you're right. Are you sure that the iMaxAttempts is reached? I guess you could always set a breakpoint in CvDLLLogger::logUnitStuck. Or maybe you're avoiding the debugger now because of the Steam limitations? Is Steamless difficult to install?

 

[keldath]

hey,
so, yeah, works for me with kb_z in the python , so ctrl + shift + z.

yes xml load is faster with none steam.

--
yes , until now my code is ninja, no error on a single write
(kidding aside, i code in python for a few years now in my work, just in civ didnt bother to print stuff to the log).

--
hope you/we manage to help with the mp play oos.

 

[f1rpo]

yes , until now my code is ninja, no error on a single write
(kidding aside, i code in python for a few years now in my work, just in civ didnt bother to print stuff to the log).

Well, I hope a Python debugger is also available at work. I generally don't know what I'm doing with Python, but inserting print statements does feel archaic.

yes xml load is faster with none steam.

Ah, who knew. Maybe more Steam security holding things up.

hope you/we manage to help with the mp play oos.

Yes, I feel bad that Rawwwrr and his friends keep trying and I can't do anything with his logs.

 

[keldath]

yup,
Python debugger.
using classic pycharm.

"Yes, I feel bad that Rawwwrr and his friends keep trying and I can't do anything with his logs."
im here to help if i can

 

[f1rpo]

Ah I made a mistake, I was playing on Epic. I still think however that it was overall feeling too short and that less chapters overall would probably be better to allow for the best impact.

I see. I've been meaning to write a report about an R&F game for some time, but I never get very far with it. My last attempt was actually on Epic speed, and I concluded that I should try Normal, just to speed things up. The 8 chapters do seem to align nicely with the eras though (attaching screenshots of the time table). And short chapters make it easier to take setbacks in stride. I find that I can often start a minor war already at the start of a chapter, putting my supposedly defensive units to use. At least in the first half of the game, a major war usually needs to target a weak neighbor so that preparations can get underway after researching/ bulbing just a couple of techs and making some trades. Just getting the war started before the end of the chapter can be good enough; the AI may be able to see it through. Not sure if adding 15 turns would change this situation much. I do think Marathon has too many chapters

And regarding Currency, even if they don't use build wealth as much as players do, by the +1 extra currency alone I would rate it as one of the most important classical tech.

Typically something like +8 commerce per turn (an extra foreign trade route in each of 4 cities), to which buildings may add 25% research on average (assuming an Academy in the capital). That could mean 60 turns to amortize if it costs 600 research. Less if additional cities get founded. Calendar can match that, Feudalism could come close too, just for the Farm boost. Construction is attractive for the Catapults. Maybe the AI is generally rather optimistic when evaluating techs, e.g. when predicting how much commerce will be gained from Machinery; and Currency, due to the simplicity of its effect, doesn't benefit much from such optimism – and should therefore perhaps simply have its trade route value multiplied by 1-point-something. I'll try to keep an eye out for certain leaders researching it especially late and whether the – by then – low cost is sufficiently taken into account.

 

[keldath]

hey ,

i got a unit stuck in a loop error, so finally sat down to debug.
i found a situation where,
a civ's worker, that was on unowned area after a city conquer,
surrounded by other cultural borders not of his own without a way to its own city.

it tried to carry out - a_reatreattocity.
and the order was to move_retreat.

but it couldnt and a loop was cycling through.

trying to think of a way out.

 

[f1rpo]

So the final AI_retreatToCity call in CvUnitAI::AI_workerMove returns true although there is no path? It should return false so that AI_handleStranded and AI_safety get run. In the end, the worker is probably just going to sit there (SKIP), which at least doesn't get the game stuck in a loop. But getting AI_retreatToCity to recognize that there is no path might be somehow difficult. I would set a breakpoint at pBestPlot = &getPathEndTurnPlot(); in the FOR_EACH_CITYAI loop and check which pLoopCity corresponds to the pBestPlot that gets chosen in the end. One may also have to look at the path that the AI intends to take toward that city. Sounds like the parameters or preconditions used for the pathfinder at that point are inconsistent with those used when the worker actually tries to move. Maybe there is a path through hostile borders (can enter those at war), but the worker is too afraid to use it?

On the OOS issue, by the way, there has been progress. I've received copies of the OOS log and, to my surprise, that was actually helpful. It showed that the only difference in game states was that one player knew of one more CvSelectionGroup than the other. Putting that together with the hint about bombardment being a trigger, it turned out that, after group bombardment, AdvCiv code had been splitting the selected group of siege units – and that was happening only on one machine; here. I had not realized that checking IsSelected() means that the code will no longer be executed for both players. May have been a self-correcting issue, but the OOS warnings were disruptive in any case. I'm hopeful that this was the only frequently occurring OOS error. Edit: Perhaps the same one as reported here last fall.

 

[keldath]

oh nive on the OOS! awesome he sent the logs, very deterministic.

ill update my end of advciv as well with the fix once you push it.
maybe other places this is at risk of reoccuring?

--
this is where hte ai retreat doesnt do anything:

if (pBestPlot != NULL)
    {
        if (at(*pBestPlot))
        {
            getGroup()->pushMission(MISSION_SKIP, -1, -1, NO_MOVEMENT_FLAGS,
                    false, false, MISSIONAI_RETREAT);
        }
        else
        {
            pushGroupMoveTo(*pBestPlot,
                    /*    was iPass >= 3
                        advc (caveat): Flags here need to be consistent with those in the loop */
                    iPass >= 2 ? MOVE_IGNORE_DANGER : NO_MOVEMENT_FLAGS,
                    false, false, MISSIONAI_RETREAT);
        }
        return true;
    }

i break pointer the entire functions flow (was long ) but i couldn't point on something,
i though about the getPathEndTurnPlot,
but i dont know much how it works -> i guess it find some path or doesnt to the target (as said, there is no real action the worker that is stuck there can do.
so only option is to skip , or kill the unit?


i had thought about adding:
if (canMove())
return false;

i figured if the unit cat move, it means it had performed, some mission and its safe to return a false.

--
if you wanna take alook i can get you my save game and my git so you can see, i used no random seed an all so its reproducible every time.
i guess this is one of those rare situation that bts/advciv couldnt catch beside the saftey net of "unit stuck in a loop" max tries.
let me know.

ill continue on.

--
edit,
looks like that
if (canMove())
return false;
did the trick, the worker was able to go retreat to a city. i guess form another ,method down the line
(i was wrong in the description, it does has acces with the path to the cultural borders and cities.
gonna try now:
if (canMove())
pushGroupMoveTo(*pBestPlot,
MOVE_IGNORE_DANGER,
false, false, MISSIONAI_RETREAT);
to see if its the pass> 2 problem.

 

[f1rpo]

ill update my end of advciv as well with the fix once you push it.
maybe other places this is at risk of reoccuring?

I've checked the other IsSelected calls in CvSelectionGroup. Mostly just K-Mod unit cycling changes, which, I guess, should only affect the active player.

this is where hte ai retreat doesnt do anything:

Right, it shouldn't enter that branch if the worker can't move at all. I.e. it shouldn't find a city to move to and no pBestPlot to enter next on the path to that city.

so only option is to skip , or kill the unit?

AI_workerMove already has routines for handling this case (handleStranded, safety, skip); the retreat routine just needs to get its pathfinding right and move out of the way.

(i was wrong in the description, it does has acces with the path to the cultural borders and cities.

In that case, it's hard to say whether the error is in AI_retreatToCity or in the processing of the move mission.

if you wanna take alook i can get you my save game and my git so you can see, i used no random seed an all so its reproducible every time.

Sounds like something I could also reproduce in WorldBuilder. A screenshot maybe?

i figured if the unit cat move, it means it had performed, some mission and its safe to return a false. [...]
looks like that
if (canMove())
return false;
did the trick, the worker was able to go retreat to a city.

canMove should always be true in those AI movement subroutines; units should have moves left. So this looks to me like it just disables AI_retreatToCity entirely.

gonna try now:
if (canMove())
pushGroupMoveTo(*pBestPlot,
MOVE_IGNORE_DANGER,
false, false, MISSIONAI_RETREAT);
to see if its the pass> 2 problem.

Could well be something about those flags. Although it seems that I did at least manage to keep them consistent within AI_retreatToCity. If there's an inconsistency, then perhaps rather between AI_retreatToCity and CvSelectionGroup (start/continueMission functions).

 

[keldath]

IsSelected

cool, so whats the suggested fix then?

 pathfinding right and move out of the way.

that part of the code is too rough for me to understand with all the path class and what not.

n AI_retreatToCity or in the processing of the move mission.

my uneducated guess would mean that its a path issue. form whatever step into i did in the functions.
cause, with the check i added, it does find the path i expected to the nearest city.

disables AI_retreatToCity entirely.

humm, shouldnt a successful pushmoveinto mission, with the mission_retreat already set the movement of a unit to maxmoves/finshed moves?
thats why i used that, expected a succesfull moveinto. the wrapper returns true either way. so if the mission didnt set the moves, i would figure the loop refires.

in the screenshot,
Shanghai, is being taken by aituroplay,
the red circle, is where the worker gets stuck right after with the loop.
with the canmove false i did, it manages to retreat to Nanjing City.
while the a_retreat_city, failes.
that none culture area is engulfed with the blue (self) player and china which is as said is at war with blue.

--edit:
looks like this worked well:
if (canMove())
pushGroupMoveTo(*pBestPlot,
MOVE_IGNORE_DANGER,
false, false, MISSIONAI_RETREAT);

maybe ill try this flag? MOVE_AVOID_DANGER

BTW- this flag: MISSIONAI_RETREAT
is only managed in one place in the code.
maybe some part should address it more, dunno.

night be straight forward to mimic with Wb.

 

[Oxidized]

Hi, I love the mod and how it aims to improve on BTS gameplay and balance in a minimalist manner . I stumbled upon your draft list of changes for Civ 4 and I used some of it as an inspiration for easing myself into modifying the DLL when I found something that looks like a bug. In CvUnit.h, there's this code for returning the collateral damage of a unit.

int collateralDamage() const
{
    return std::max(0, m_pUnitInfo->getCollateralDamage());
}

(link to version 1.11 Github branch here)

It looks normal and returns the collateral damage for the unit info, but it's missing the existing function CvUnit::getExtraCollateralDamage() that accounts for collateral damage bonuses from promotions (which is the entire Barrage promotion line). You can see a few lines above it how the CvUnit::evasionProbability() function calls both evasion probabilitity function from the unit info class as well as the extra evasion probability which is also modified by promotions like Ace. I haven't investigated how this impacts gameplay, but this function is used for AI and gameplay calculations so it might be fairly significant.

 

[f1rpo]

@Oxidized: Thanks. There seems to be a bit of history behind that odd treatment of extra collateral damage. Apparently, Barrage was considered useless by players of BtS 3.13: Is Barrage broken?
And then update 3.17 made a deliberate change, some side-effect of which is discussed here: Collateral Damage and no end
I don't know if these are worth reading (I haven't really done so), but interesting to note that it was a point of discussion.
The 3.17 list of changes only says "Fixed Barrage promotion bug". And, related: "Flanking damage now depends on strength of defending unit, not on strength of siege weapon". Well, I guess this second change doesn't really affect extra collateral damage.
The code changes can be reviewed here: 3.17 as a Git commit
These appear to be the relevant changes:

Spoiler :

Then, in the 3.19 list of changes:
"Collateral damage modifiers, collateral damage immunity, and air defense modifiers, multiply instead of add"
The corresponding diff in the Unofficial Patch, of which patch 3.19 adopted ... parts, I suppose (full diff):

Spoiler :

I assume that this deplorable mess behaves mostly as intended rules-wise. Checking the call locations in AI code is a good thought. getExtraCollateralDamage having been removed from collateralDamage in 3.17, apparently without any accompanying AI changes (nor in 3.19) does not bode well. I guess the collateralDamage() == 0 checks are OK, i.e. no need to check for extra collateral damage as those units can't get Barrage promotions. That leaves AI_airStrikeValue, AI_defendBaseAirStrike [edit one week later: I don't think Bombers can get Barrage, so these two don't really matter], AI_bestUnitForMission (MISSION_BOMBARD case), AI_currEffectiveStr (rough K-Mod estimate of coll. damage), AI_getBestGroupAttacker, AI_sacrificeValue (sacrificing an attacker for coll. damage), LFBgetBetterAttacker (AI attack order, also for human group attack).

Should probably add a function AI_collateralDamageFactor to replace collateralDamage in all those location. Maybe in pretty much all AI code, i.e. also the zero checks, to support Barrage also on units that don't deal collateral damage out of the box.

Simply this?

int CvUnitAI::AI_collateralDamageFactor() const
{
    return (collateralDamage() * (100 + getExtraCollateralDamage()) / 100);
}

Seeing that iCollateralDamage in Civ4UnitInfos.xml is 100 for all siege units, just adding them up would have the same result – except for Battleship, Missile Cruiser which deal only 50 coll. damage (and Cho-Ku-No, which can't receive Barrage). Could get called a lot of times, kind of annoying to divide by 100 every time (not entirely cheap computationally). Thinking of some mod-mod perhaps wanting to re-enable Barrage for Tanks (which still have a collateral damage limit and target limit set in XML), in a quick test, a Tank with Barrage1 dealt 4 points of damage to a defending Archer, putting it at a displayed strength of 2.9/3. So multiplying by ExtraCollateralDamage is not going future-proof things well. Although the damage is almost negligible ...

I may need to take a closer look at CvUnit::collateralCombat:

Spoiler :

int iCollateralStrength = (getDomainType() == DOMAIN_AIR ?
            airBaseCombatStr() : baseCombatStr()) * collateralDamage() / 100;
if (iCollateralStrength <= 0 &&
    getExtraCollateralDamage() <= 0) // UNOFFICIAL_PATCH
{
    return;
}
// ...
int iTheirStrength = kTargetUnit.baseCombatStr();
int iStrengthFactor = ((iCollateralStrength + iTheirStrength + 1) / 2);
/*    advc (note): This makes the damage proportional to (3*r + 1) / (3 + r)
    where r is the ratio of the attacker's to the defender's base combat str.
    This ratio is used again a few lines below in the iMaxDamage formula. */
int iCollateralDamage = (GC.getDefineINT(CvGlobals::COLLATERAL_COMBAT_DAMAGE) *
        (iCollateralStrength + iStrengthFactor)) /
        (iTheirStrength + iStrengthFactor);
iCollateralDamage *= 100 + getExtraCollateralDamage();
iCollateralDamage *= std::max(0, 100 - kTargetUnit.getCollateralDamageProtection());
iCollateralDamage /= 100;
if (pCity != NULL)
{
    iCollateralDamage *= 100 + pCity->getAirModifier();
    iCollateralDamage /= 100;
}
iCollateralDamage = std::max(0, iCollateralDamage/100);

Maybe I should just add the two values up in AI_collateralDamageFactor. That's fast, will work somewhat for Barrage Tanks and will only really distort things for ships.

I stumbled upon your draft list of changes for Civ 4 and I used some of it as an inspiration for easing myself into modifying the DLL [...].

Those PDFs of balance changes? I had wanted to give those another overhaul, but got overwhelmed by the complexity of the whole puzzle - reconciling closeness to BtS with game balance, ease of implementation and some notion of historicity, and, on that last point, kept becoming aware of further details of the original game that are nonsensical, as well as new rationales for justifying some of the nonsense I used to consider unacceptable ... Too much work for a draft I wasn't even going to execute. But I'm glad you're finding some ideas in there.

 

[f1rpo]

cool, so whats the suggested fix then?

I've got a bunch of changes yet to be untangled into individual commits, so maybe best to just drop a screenshot here:

Spoiler :

humm, shouldnt a successful pushmoveinto mission, with the mission_retreat already set the movement of a unit to maxmoves/finshed moves? thats why i used that, expected a succesfull moveinto. the wrapper returns true either way. so if the mission didnt set the moves, i would figure the loop refires.

You're right, a successful move mission should consume at least one movement point. So you're detecting the unsuccessful move attempt in CvUnitAI and then try again with the IGNORE_DANGER flag. But I guess the failed move will still strigger the stuck-in-a-loop warning? At any rate, CvUnitAI ought to be able to check upfront whether the move will succeed. After all, CvUnitAI and CvSelectionGroup have access to the same pathfinder. But it's good to know that there is apparently a pathfinding issue and that it has to do with danger from enemy units. I'll try to reproduce your screenshot; thanks. Worker with enemy unit to its left, coast to its right, safe city NW. (Although it could also be some rare under-the-hood issue with the pathfinder that'll only come up under very specific circumstances.) Edit: It moves directly into the city, as it probably should. Doesn't get stuck.

Spoiler :

Probably tries to move SW first in your savegame. That's where it (successfully) went in my game too until I asked Spain to close its borders.

BTW- this flag: MISSIONAI_RETREAT
is only managed in one place in the code.
maybe some part should address it more, dunno.

I see, only used by AI_retreatToCity and basically doesn't get any special treatment (treated the same as NO_MISSIONAI in one place). I guess the mission AI types generally don't have much impact. Hm, if it's enough to just go to a city, maybe that's fine.