Espionage target weighting

[jdog5000]

Here is the current method all AIs use for determining the 0 to 99 weighting they apply to all other teams in the game, from CvPlayerAI::AI_doCommerce:

First, zero things out:

	if (isCommerceFlexible(COMMERCE_ESPIONAGE))
	{
		if (getCommercePercent(COMMERCE_ESPIONAGE) > 0)
		{
			setCommercePercent(COMMERCE_ESPIONAGE, 0);

			for (int iTeam = 0; iTeam < MAX_CIV_TEAMS; ++iTeam)
			{
				setEspionageSpendingWeightAgainstTeam((TeamTypes)iTeam, 0);
			}

			bReset = true;
		}
	}

Then compute new values:

	for (int iTeam = 0; iTeam < MAX_CIV_TEAMS; ++iTeam)
	{
		CvTeam& kLoopTeam = GET_TEAM((TeamTypes)iTeam);
		if (kLoopTeam.isAlive() && iTeam != getTeam() && !kLoopTeam.isVassal(getTeam()) && !GET_TEAM(getTeam()).isVassal((TeamTypes)iTeam))
		{
			int iTarget = (kLoopTeam.getEspionagePointsAgainstTeam(getTeam()) - GET_TEAM(getTeam()).getEspionagePointsAgainstTeam((TeamTypes)iTeam)) / 8;

			iTarget -= GET_TEAM(getTeam()).AI_getAttitudeVal((TeamTypes)iTeam);

			if (iTarget > 0)
			{
				changeEspionageSpendingWeightAgainstTeam((TeamTypes)iTeam, iTarget);
			}
		}
	}

There is no other place in the code where the AI changes these weightings aside from initializing the values when creating a player. Only when it actually goes to set the weighting are the limits of 0 to 99 enforced.

In the first piece of code I think there is an obvious programming bug ... the weights to other teams will only be reset if the Espionage commerce slider is above 0, this doesn't make any sense. This is especially suspect given that in the second piece of code weights are not changed for vassals or masters, its assuming that the weights were set to zero in the first piece.

The logic for picking weights also seems like it could use some more well logic. Right now there are two terms:

(kLoopTeam.getEspionagePointsAgainstTeam(getTeam()) - GET_TEAM(getTeam()).getEspionagePointsAgainstTeam((TeamTypes)iTeam)) / 8

This is a proportional term, (their esp against us - our esp against them)/8. This causes the AI to try to catch up with other teams who are spending against it while also lowering weighting for a team who gets a big lead in espionage points.

-GET_TEAM(getTeam()).AI_getAttitudeVal((TeamTypes)iTeam);

The AIs attitude towards the other team is simply subtracted from the proportional term. Attitude values are usually between -15 and +15, though they can be higher.

One example:

Team A hates team B with attitude -15. Team A also has a >= 120 point lead in espionage points. The resulting weighting: 0.

The proportional terms is very strong right now, causing AIs who are behind in espionage spending to another player to weight them very highly regardless while causing players with large leads to reduce their weighting even towards enemies. Certainly the proportional term is important, but right now it's really dominant.

So what can we do to have the AI weight enemies more intelligently? Some ideas:

1) Proportional term is currently absolute, could be made relative to AIs espionage point generation rate or current number of espionage points against the opponent.
2) War plans! If the AI is in or planning war, weighting should be increased drammitically.
3) Target espionage levels. Against enemies, for example, AI should desire to have city visibility level of points and have a new proportional term to help it reach that level.
4) Reduce occurrence of saturation. Current strength of proportional term, particularly when AI has esp point lead, causes AI to set most values to either 0 or 99. Since this is a weighting and not a spending level, hitting saturation simply obscures priorities.
5) Non-zero neutral value. Again because this is a weighting, having 10 or 25 be "neutral" instead of 0 would allow differentiation between known friends at 0 and players the AI is cautious and has small delta esp points with at neutral.
6) Spy mission likelihood. If the AI would consider launching spy missions against the player, weighting could be increased. I don't know what factors which aren't already covered could be added here, need to investigate how the AI picks spy mission targets.

Opinions? Other ideas? Let's brainstorm some improved behavior!

 

[Yakk]

I agree. This value sucks:

(kLoopTeam.getEspionagePointsAgainstTeam(getTeam()) - GET_TEAM(getTeam()).getEspionagePointsAgainstTeam((TeamTypes)iTeam)) / 8

A first pass:

int their_points_to_us = kLoopTeam.getEspionagePointsAgainstTeam(getTeam());
int our_points_to_them = GET_TEAM(getTeam()).getEspionagePointsAgainstTeam((TeamTypes)iTeam);

int percent_ratio = (their_points_to_us+1)*100 / (their_points_to_us + our_points_to_them+2);

Ie, ratio = x/(x+y).

(The +1 and +2 terms prevent division by 0, and the *100 term makes it a percent).

This approaches 1 as x >> y, and 0 as y >> x. If x=y, then this approaches 0.5.

Then, we can apply this:
GET_TEAM(getTeam()).AI_getAttitudeVal((TeamTypes)iTeam)
in a more intelligent way. . .

How about +10 meaning "we are happy with half as many esp. points", and -10 meaning "we are happy with twice as many esp. points", with each +/- 1 being a 10% shift?

That gives us a second pass:

int our_attitude_to_them = GET_TEAM(getTeam()).AI_getAttitudeVal((TeamTypes)iTeam);

int their_points_to_us = kLoopTeam.getEspionagePointsAgainstTeam(getTeam());
int virtual_points_to_us = their_points_to_us;

if (our_attitude_to_them < 0) {
  // virtual points *= 100%+(negative our attitude * 10%)
  virtual_points_to_us = (virtual_points_to_us*(10+(-our_attitude_to_them)) + 5)/10;
}
int our_points_to_them = GET_TEAM(getTeam()).getEspionagePointsAgainstTeam((TeamTypes)iTeam);
int virtual_points_to_them = our_points_to_them;
if (our_attitude_to_them > 0) {
  // virtual points *= 100%+(our attitude * 10%)
  virtual_points_to_them = (virtual_points_to_them*(10+(our_attitude_to_them)) + 5)/10;
}

int percent_ratio = (virtual_points_to_us+1)*100 / (virtual_points_to_us + virtual_points_to_them+2);

which generates the described effects.

And if both players have 10 times as many esp points, the values don't really change, instead of getting all crazy.

Given that this is a percent value, doing more transformations on it can be easy. You can also fudge with the virtual values for a "nicer" transformation that won't hit the hard caps of 0% and 100% except in the extreme limit.

 

[jdog5000]

Yeah, I like the way you've made it relative using percentage ahead/behind and changed the role of the attitude effect so it's more integrated and relevant. I think the -10 attitude leading to a goal of 200% of the opponents espionage points may be a little aggressive, maybe something more like 150%.

 

[Nor Me]

I think the current AI epsionage spending is actually quite a solid defensive strategy. In particular, it only uses the espionage slider to catch up. If 2 AI players are both trying to get a big enough ratio with each other to , say, see each other cities then I doubt it would be worth both of them spending 10% of their commerce to do it.

I too thought that not zeroing the weights when at 0% espionage was an obvious bug at first. But it might well be intentional. If an AI player were ahead of everyone in espionage, without it they would have all the weights zero. If the other AI are playing catch-up, that's the wrong thing to do. They'll all be able to keep up so spending any points on missions would still require you to catch up. And spending points on your friends risks starting an espionage cold war and makes them spend less on your real rivals. The defensive thing to do is to spend the points on players who are more likely to overtake you. The current strategy guesses that those are the players who last had a higher total, hence not zeroing. Only when multiple weights hit 99 does it start to produce less than sensible results.

With the catching-up behaviour of the AI, when I want to use spies aggressively, I concentrate on just one player. I think it might be a good offensive strategy for an AI on 0% espionage who is producing more points than are necessary to catch up to do spend those spare points like that.

I would do that by making it always zero the weights and putting something like this after the existing weight setting loop:

if(iEspionageTargetRate < getCommerceRate(COMMERCE_ESPIONAGE))
{
	int iBestValue = -42;
	int iValue;
	TeamTypes eBestTeam = NO_TEAM;
	
	for (int iTeam = 0; iTeam < MAX_CIV_TEAMS; ++iTeam)
	{
		CvTeam& kLoopTeam = GET_TEAM((TeamTypes)iTeam);
		if (kLoopTeam.isAlive() && iTeam != getTeam() && !kLoopTeam.isVassal(getTeam()) && !GET_TEAM(getTeam()).isVassal((TeamTypes)iTeam))
		{
			iValue = -GET_TEAM(getTeam()).AI_getAttitudeVal((TeamTypes)iTeam);
			if(::atWar(getTeam(),(TeamTypes) iTeam))
				iValue += kLoopTeam.getPower(false);
			if(iValue > iBestValue)
			{
				iBestValue = iValue;
				eBestTeam = (TeamTypes) iTeam;
			}
		}
	}
	if(eBestTeam != NO_TEAM)
	{
		changeEspionageSpendingWeightAgainstTeam(eBestTeam, getCommerceRate(COMMERCE_ESPIONAGE) - iEspionageTargetRate);
	}
}

jdog5000 editted iEspionageTargetRate out of the code he posted. It's just the sum of the weights given to the teams so far. After this (and after the code I'm proposing), it's modified a bit and the espionage slider is increased until it reaches that rate. As the espionage rate is set to 0% earlier, getCommerceRate(COMMERCE_ESPIONAGE) is the 0% spending.So the inequality in the above code means it shouldn't cause the AI to run at 10% espionage rather than 0% any more than it does now. It should reduce tit-for-tat spending, as that will now be closer to iTarget for each team, which is what AIs without spare points are aiming for.

Then it's just a question of making spies target players with whom you have a good ratio so if targetting one civ gives the AI a lead, it would exploit it. The existing spy movement code is not perfect. For one, as far as I can tell the AI builds ships to put spies on, but spies never board ships.

 

[jdog5000]

I see your reasoning that it could be a backhanded way of giving espionage spending some kind of history, but that doesn't seem very likely and is certainly no replacement for logic which actually determines whether continuing weighting another player is desirable. The fact that unless the Espionage slider is above 0 it will never reset its weighting on players that have capitulated to it makes it clear to me this is not some ingenious strategy.

Imagine the following scenario (assuming a non team game):

Player A has a significant esp point deficit with Player B but not with any other player. Then Player B runs an expensive espionage mission against A, causing the deficit to switch. There are then two possible ways this plays out:

1) If this deficit was very large or A's espionage production is not good, A would have set their espionage slider to 10 or 20% to try to speed up recovery from the deficit. On their next turn after B's mission A's slider is elevated so A would reset spending against B and then given the reversal of the deficit would not weight B above 0.

2) If instead A's deficit to B is less than their non-commerce slider espionage production, A's slider would be at 0. After B's mission A would not reset their weighting of B because the slider was at 0 and then the reversal of the deficit would cause them not to change their weighting from the prior 99.

The drastic differences in behavior between these scenarios depending simply on whether the deficit has caused A to bump up their espionage slider makes no sense to me.

I do agree though that given the interconnectedness of weighting and spending simply fixing the resetting bug/quirky behavior is not a viable solution. The current latching behavior produces interesting results but I wouldn't exactly call it intelligent. There's an opportunity cost to allocating more points where you already have a lead, particularly if you're friendly towards the other player and may never use the points (or they're now your vassal).

We need a new approach which maintains the current emphasis on playing catchup with others' espionage levels while also adding some logic so that the AI can determine when to intentionally keep allocating points to another team when it has a lead.

 

[Yakk]

Ok, that code is on serious crack.

iTarget and iEspionageTargetRate are values of true type "spy points".

Adding attitude values, which are an utterly different dimension, seems like a bad idea.

Using the spy point target values as weights works decently, but they are then capped at 99... which is no good.

We can do a two-pass algorithm: the first sums up the iTarget values, and the second generates weights. Either with a subprocedure, or a copy/paste job.

Once we are past that, the rest of the code is suprisingly sane -- at first glance, I thought they where really stupid, but they compare the target amount of spy spending...

Hmm.

Each country wants to have spend (minus attitude) spy points for a given player (this does NOT scale with era or economy -- which would hurt low-level AIs).

They also want to catch up by 1/8th the difference between your spy points. (which, naturally, never results in convergence).

If Gold spending is > 50%, they won't spend %commerce on it.
They won't double their research spending.
They won't pass 20% spy spending (I suspect this is here to prevent the AI from being dumb).
They won't spend on spies when they are in money trouble.

...

I think that isn't that good. But it appears that they are less on crack than I thought.

If we renormalize the iTarget values to be between 0 and 99, by dividing them all by the iEspionageTargetRate (with some fiddly bits), things will be less insane.

But really, this code is seriously tit-for-tat stuff. Is it really that important to have spy points near your opponent?