Tech rate and wonders

[karadoc]

I've recently played a few games with Better AI, and my impression is that the AI builds fewer wonders and techs slower than without Better AI.

I'd be interested to know if other people agree with me, or if you think I've just gotten 'lucky' in the few games I've played. I'd also be interested in watching an auto-play game in which some of the players are using Better AI and some aren't — has this been tried? (I imagine that it would be a bit of work to set it up — a lot of code copy-pasting and renaming of functions...)

 

[Fimbulvetr]

Might be relaeted to the too-early warplans. Not sure what's the best way to fix it, Afforess alraedy has an idea though.

 

[karadoc]

Maybe it's that, I'm not sure. I just a quick look at what the AI does. Here are the two main problems I noticed:
- The AI takes too long to improve tiles. It often gets a second city and sometimes even a third city before building its first worker.
- The AI really doesn't like working plains cottages. It prefers to work forested grassland over plains cottages. If it has more food than it can use, it should just work the cottage. If it doesn't have enough food to support the plain cottage, then it should probably improve the tiles that it actually intends to work first.

There are a bunch of other issues such as a questionable tech order, lack of spawn-busting, always escorting settlers (which is slow, and can be avoided by using units to scout in advance instead of directly escorting). But these other things relate to more advanced techniques which are probably quite difficult to program. I think shortage of workers in the early game is a problem that can be easily addressed. Since Jdog seem to be missing in action, I might try to improve it myself.

 

[karadoc]

I've looked at the code a bit and I think I have some understand of what the problem is, but fixing it will not be easy.

There is already some special code which is meant to favour building an early worker. It looks like this:

if( isCapital() && (GC.getGame().getElapsedGameTurns() < ((30 * GC.getGameSpeedInfo(GC.getGameINLINE().getGameSpeedType()).getTrainPercent()) / 100)))
{
	//...
	if( iExistingWorkers == 0 && AI_totalBestBuildValue(area()) > 10)
	{
		if (!bChooseWorker && AI_chooseUnit(UNITAI_WORKER))
		{
			if( gCityLogLevel >= 2 ) logBBAI("      City %S uses choose worker 1a", getName().GetCString());
			return;
		}
		bChooseWorker = true;
	}
}

So that basically says "if it's early in the game and we don't have any workers yet, and if a worker would be helpful... then build a worker."

The catch is the last part, about being helpful, AI_totalBestBuildValue(area()) > 10. That function adds up the value of all the improvements that could be made in the area. The problem is that it doesn't take into account tiles that can't be improved right now but soon will be, such as tiles in the outer part of the fat-cross and improvement which require a tech that the AI doesn't have yet.

A human player would build a worker right away, confident that his borders will have popped and the research will be finished by the time the worker is ready, and thus the worker will be useful... but the AI doesn't look into the future like that. It only sees what it could improve right now.

So, I tried just disabling that last part of the check. Like this:

if( iExistingWorkers == 0 /* && AI_totalBestBuildValue(area()) > 10*/)

Doing that helped some of the AIs, but made it worse for some others. Apparently the AI doesn't mind letting their starting city wait awhile before getting the techs to improve the nearby food resources. So forcing it to build a worker first doesn't help it.. the worker just sits around doing nothing while the AI researches mysticism etc.

The best thing would be to have a coherent strategy, where the AI builds the worker first and specifically chooses tech that will give the worker something to do by the time the worker is built — preferably tech which will speed up growth, such as agriculture and animal husbandry (if there is livestock nearby).

The coherent stuff might be a bit tricky... so tomorrow I'll try a simpler solution: I'll force the worker first like I did already, and I'll try to work out how to make the AI value early improvement-enabling techs more highly so that the AI is more likely to research stuff that gives its worker something to do. I expect that there will still be some problems. I know there are some cases where it is a really dumb idea to build a worker first. But I'll just try it and see how it goes.

 

[Yakk]

One thing you could do is work out the N-turn yield increase from having a worker+improvement technology.

Ie, I think all of the below can be calculated quickly by an AI, and if we restrict it to the case where we have 0 workers it wouldn't slow down the game much:
Latency(Start) = max(turns to make worker, turns to research tech)
Tiles that can be improved = (Latency of improvement increased fat cross area)
Latency(Yield) = (turns for worker to build improvement)
Increased yield = (delta between best setup without improvement and best with)
Cost = (??? for picking tech early, (food + hammers) to build worker)
Benefit = (N - Latency(Yield) - Latency(Start)) * Increased yield
Balance = Benefit-Cost

Instead of "summing over N turns", we could instead code in a decay factor for "things in the future are worth less than things now".

The effect of the above is that if we ran into a tile that had massive improvements from working, we'd really want a worker -- and if there was no tiles or improvements possible, we'd skip the improvement techs and not build workers.

(As an aside, we probably also want to make sure we can build a worker, before we do the above -- or add in the cost to research the worker-building tech).

 

[Afforess]

I've recently played a few games with Better AI, and my impression is that the AI builds fewer wonders and techs slower than without Better AI.

Are you using Better AI in your mod, or just plain Better AI?

 

[karadoc]

I just looked at what the normal AI (without Better AI) does. On noble it tends to build a warrior and then a worker; on emperor (which is what I usually play on) it tends to go straight for the worker. Unfortunately for it, it does so even if the worker will have nothing to do — just like when I forced Better AI to do it. Strangely enough, in the map I've been using as a primary test neither Better AI or normal AI builds a worker first in the city that I think would benefit the most from it. Silly old Mansa Musa is too keen to build a skirmisher and thus squanders a massive early game advantage. (I've attached the file for anyone or is interested.)

Yakk, I like that sound of your suggestion. That would be much better than hard-coding a strategy in — which is effectively what I was talking about testing (force it to build a worker first, then force it to research a tech to help that worker).
In economics, people sometimes talk about a 'discount rate' — which describes an exponential decay of value over time, because "things in the future are worth less than things now." I think it would be a good idea to use that kind of reasoning all through the AI, and different AIs could even use different discount rates if they wanted to... but I'm not sure it would work if it only affected a special few decisions rather than all of them.

But I'm not entirely sure how to approach the coding of it. I'm pretty new to this whole modding business, so I'm not very familiar with most of the code. My impression so far is that the AI's choice of research, and city production, and what their units are doing, are all independent of one another. Your suggestion seems to rely on a more coherent decision making process. I'm still browsing through the code to see how things are done to see where it is best to change it... My thinking so far is that AI should highly value techs which would allow it get to food improvements if it doesn't already have plenty of food; and that it needs to take the outer part of the fat-cross into account for any city that has positive culture-per-turn, and also take into account the improvement options that will open up when the current tech has finished researching... Having written all of that, it's just pretty much what you said. I guess I'm not making any progress yet.

Maybe I'll try something different before tackling all of this. I could just tweak some stuff to make it more likely to build workers in general, and less likely to build a settler before a worker (which I think is almost aways a mistake.) It would be good to get the first few AI decisions to be perfect, but I think the AI's problems are much more broad than that anyway, so I'm going to look for some other stuff to improve while I learn more about the code and think about how to implement what we've talked about.

Are you using Better AI in your mod, or just plain Better AI?

It's my own mod, but it is a very minimalist mod; almost nothing has been changed. I've just tweaked how maintenance is calculated and made some minor changes to a couple of civics.

[edit]
There is already a way for AIs to choose a worker based on what they are currently researching... I wonder why it doesn't happen as often as I expect.

[edit again]
Combinations of factors can conspire to make the AI play poorly... for example, in the game I posted, the Germans get off to a really slow start. They _should_ start by researching animal husbandry and building a worker... but they choose fishing because they have more seafood nearby than land food. They then don't consider building a worker because they won't be able to improve the pig yet, so they choose a settler for lack of something more appealing. The settler takes a long time to build... when it is finally done, they already have animal husbandry but they decide to start building a workboat before a worker. The workboat is also painfully slow to build. Later, when they have settled their second city, they really need to research mysticism so that they can pop their borders to get the rice, but they don't think of that. They research all sorts of order junk first while their second city does basically nothing.

From my point of view, the key issue that caused the Germans to have a really slow start was that they started building a settler without good reason, and then felt obligated to finish building it before starting something more useful. The production of the settler is far too slow in this case to be worth while. Ideally they would have realised that researching animal husbandry first would give their city something useful to do (ie. build a worker), whereas fishing just makes their city waste time waiting for the research to finish. The AI is apparently capable of choosing to build a worker as a result of the research it has chosen, but what it needs this time is the other way around: choosing to research animal husbandry to suit the worker it is (should be) building!

But even after that initial mistake with the choice of fishing, they should realise that they need to interrupt their settler to build a workboat or two, so that they have 2 citizens working 2 high-food tiles instead of 1 citizen working 1 low food tile.

The mysticism thing is a separate issue.

To fix this stuff, I think the tech evaluation system needs to be changed. Having more seafood around the city doesn't mean fishing is better, because it takes a long time to build all the workboats to get the seafood, and you probably don't need all that food at once anyway. Initially, the city is only at size 1. So it only really needs 1 good tile to work until it grows, and the pig would be the best tile. So instead of just counting all the improvements it could make with a tech, the AI should only consider the improvements that it will actually be able to use in the near future — and getting one good tile early should be worth more than getting a few good tiles later.

Let me know if you have any thoughts about what I've said or any ideas for implementing this stuff. Like I said, I'm still a bit of a newbie when it comes to editing the code, so I'm still not confident enough to really start ripping up old code on my own.

 

[Yakk]

There is not much in the way of coherent strategy for the Civ4 AI, as you have noticed.

So honestly? Cheat.

Write the code that solves the "if I have 0 workers, should I start working on a worker"?

Working on a worker includes grabbing the tech needed to build workers, grabbing the tech needed to possibly do the improvement, and building a worker near the place where the improvement should be.

Only run this code when:
1: We have 0 workers, and
2: We can build a worker, or we are adjacent to a tech that can build a worker.

In the standard game, 2 always qualifies. So you end up doing extra work at the start of the game when you have 0 workers (or when an enemy has managed to steal every one of your workers just as your empire is falling -- which could result in a funny AI bug if it isn't caught!).

The code would be run on both a city production and a tech production decision point.

It scans the neighborhood of each city. For each square, it calculates:
how long before it is culturally controlled (rough estimation)
how long it takes to build the various improvements there.
how long it takes to get the technology to build the various improvements there (so improvement tech plus possibly worker-building tech)
how long it would take to locally build a worker.
how much it would "cost" to locally built the worker.
what would be the change in city yield from improving that tile.

Calculate either a time threshold based on game speed, or a discount factor based on game speed and (either estimated, or fixed) game duration.

(the discount factor math ends up being pretty neat.

If you are going to get extra yield Y from turn T through T+K with discount factor D, the present value of it is Y * D^T * (1-D^(K+1))/(1-D).

If it is from turn T until infinity, it is Y * D^T / (1-D), as with K->infinity D^(K+1) -> 0.

We should do the same for the worker-build investment. Spending X resources for T turns, starting with now, has a present value of X * (1-D^(T))/(1-D).

Discount factor, as used above, is a number usually just under 1. 1.00 means "no discount", while 0.00 means "ignore the future". 0.99 means "next turn matters 1% less than this turn".

Discount factors should be closer to 1 for long games, further from 1 for short games. If there is a crisis (war with significant threat, economic collapse, etc), discount factors should move away from 1 so short-term wins out over long-term. But that is beyond the scope of this fix -- using a game-speed dependent discount factor should work well.)

The hard parts in the above include (A) actually writing it, (B) finding the plots and the improvements, (C) finding "local" worker builders (we could make it half-assed by building the worker anywhere?), (D) working out all of the costs to build the improvements, (E) working out values for yields (is 50 hammers worth 3 coin per turn?).

---

Note that the AI goes for mysticism so it can found a religion. If everyone is building workers first, then the player gets the ancient religions, which are a huge advantage.

 

[karadoc]

I don't suppose you have any interest in coding that idea you just posted... ?

The problem I mentioned involving the AI and mysticism is that sometimes they just don't research it. I think the problem is this: if the ancient religions all get founded while a particular AI is researching other stuff, like agriculture, bronze working, etc. then that AI will no longer feel the incentive to research mysticism. This AI will have no means of producing culture whatsoever, and they do not realize that mysticism is the cheapest way gain access to a cultural building. Having access to some form of culture is vital; so I think it is important that the AI learns to value mysticism for monuments.


Now. We've been talking mostly about the early game, but my original point was that the AI seems to tech slower with Better AI, which may or may not be due to early game problems. Maybe the most significant problems come later on.

In the game I'm currently playing, the two top AI players are doing well. Their tech seem to be fine (although their tech choices are sometimes highly questionable, and they seem to research the same stuff as each other). However, there are two other AI players who are far far behind for no apparent reason. There is one in particular who's economy has completely fallen off the rails. (Saladin) He was doing fine for a long time. He and I and most of the other AIs were all pretty close in terms of tech, then he declared war on me because I didn't give in to one of his demands. He was pretty strong in the war, but I managed to hold him off and then capture one of his cities, and then pay him for peace. This happened in the era of knights, pikemen, etc. After that war, his research completely ground to a halt. He spent about 80 turns researching democracy! His empire is still bigger than mine, and I see no significant reason why his economy should be so weak. My guess is that he has built too many military units and not enough cottages, and that he is coming in and out of 'financial trouble' without ever really growing the economy back up to full strength. I think he still has war ambitions, but it is completely out of the question for him. I now have industrialism and artillery.. and he doesn't have steel or rifling. Meanwhile, the two top AIs are very strong and have defensive pacts with each other...

The reason I'm posting all this is that I suspect I need to look more broadly for problems rather than only focusing on the early game.

 

[Fimbulvetr]

The reason I'm posting all this is that I suspect I need to look more broadly for problems rather than only focusing on the early game.

Early game is simpler to analyse, and as it's almost impossible to come back from a bad start, it's a good thing to look at that very closely.

As for your forced worker: don't. Just replace the "> 10" with "> 0", then the AI will build the worker as soon as it has something to do. It's what I use in Better BUG AI.
Making the AI build a worker while the tech needed for improvements/builds is still being researched is not that simple. You'd have to check if the research would actually be finished before the worker, or at least with very few turns delay. Then you have to update best plot builds with that additional tech you don't have yet (dangerous, only do this if you really have zero workers) to see if something pops up - if some tile becomes improveable, that totalBestBuildValue will the > 0 and worker 1a will be chosen. Of course you should also somehow make sure the tech selection stays that way .. it's really more tricky than it seems.
Taking into account tiles that aren't yet within cultural borders but soon will be, might be a bit simpler but still..

Anyway, if the tech currently being researched enables a build to improve a bonus tile, then this is already being accounted for, same goes for tiles that aren't yet within cultural borders (worker 1), sea workers have higher priority though (worker sea 1a is considered before worker 1).
Maybe that should be changed. I suggest moving the whole block including worker 1, worker sea 1 and worker 2 to before the early game worker logic block.

 

[karadoc]

So far I've just made some minor tweaks to how things are done. For example, that > 10 thing you mentioned, I changed to "(AI_totalBestBuildValue(area())+5*iLandBonuses) > 10". And a few places where the AI would build a worker when pop > 3, I changed to pop >=3... and there was one or two spots where it had "if (number of workers != 0)" before deciding whether to build a worker; that didn't make sense to me so I removed that if.

So basically the AI will be more likely to build workers sooner than before. I also changed something in the evaluation of techs which enable bonuses. Like this:

/* old code
if (iNumBonuses > 0)
{
	iBonusValue *= (iNumBonuses + 2);
	iBonusValue /= kImprovement.isWater() ? 4 : 3;	// water resources are worth less
	iImprovementValue += iBonusValue;
}
*/
if (kImprovement.isWater())
{
	iBonusValue *= (iNumBonuses + 2);
	iBonusValue /= 4;
}
else
{
	iBonusValue *= (2*iNumBonuses + 2);
	iBonusValue /= 3;
}

That old code seems to have mistake in it: if you have exactly 1 of the seafood resources, it values the tech lower than if you had zero. For land resources, having 1 is the same as having zero. So I removed that if, and also changed the scaling a bit. The point of my scaling change is to reflect that land resources can be improved by a single worker, whereas sea resources each consume a workboat, and are typically more difficult to defend...

But really all of this tech evaluation code and some of the improvement evaluation doesn't seem right to me. To choice which tech is best it basically just adds up a bunch of arbitrary numbers.. each number corresponds to something the tech gives you, but the value of the numbers seems to be just plucked out of the air. For the time being, I've put a bit more randomness into the tech choices in the form of a random multiplier between 110% and 90%; just so that the AI is less likely to all take the same tech path and neglect important techs which happen to be undervalued. But I intend to put a bit more work into it.

When I'm choosing techs, I think of the benefits in terms of a % boost to the productivity of my empire.
For example, if I've got just 1 city, with 2 population. Going from working 2 grassland forests to working a wheat farm and a gold mine is a massive boost. So getting the right techs and getting the worker are hugely important. Whereas later in the game, when I have like 20 cities working mostly high-yield tiles, getting an extra gold mine and wheat farm is nowhere near as important. So I think it would be good to estimate the % boost to various parts of the economy.

Similarly for military units. Currently the AI basically sees that a tech will grant a new unit, and adds a more or less flat number for the tech to account for the new unit. It doesn't actually care how strong or weak the unit is. I'd like to change it so that it can estimate how much of an advantage this particular new unit would be. A musketman is only slightly stronger than macemen / longbows, but a rifleman is a big step up from both of them. I'd like the AI to be able to understand this an thus value the techs more appropriately. I suppose to do this the AI would have to consider the % improvement for a variety of different roles. For example, macemen usually aren't better than longbows for defence, but they are a big boost to offence. Trebuchets are only stronger for bombarding and collateral damage vs cities, they are weaker than existing units for every other purpose. Riflemen are a big step up for both defence and offense, but don't help with bombarding...

I'm not in a great hurry to revamp everything, so this might take me awhile... but I'm starting to gain some confidence in editing the existing AI. So maybe I'll be motivated and able to make all these changes soon.

 

[Yakk]

Having the AI fully understand the stats of units for the purpose of "is this unit an upgrade" is hard.

It would be easier to hand-feed this to the AI.

How? Fix the iPower field.

iPower/Production should, in theory, give you the military power you get per unit of production you put into that unit.

iPower/Upgrade Cost should, in theory, give you the military power you get per unit of cash you put into upgrading existing units.

This doesn't work all that well, because iPower doesn't grow fast enough. So if the AI looks at a Maceman's iPower and cost, it says "meh, not much better than an axeman, not worth the bother".

This also leads to the in-game distortion of "I have 100 axemen and he has 75 macemen. We are clearly equal in Military Power!" on the power graph, which is ridiculous because 75 macemen will walk over 100 axemen like they aren't there.

Solve that, and learn the AI hints (this unit is a counter unit -- so if I know my enemy can build a unit that this counters that is badass, I should over-value this counter unit) in the XML file, and you could teach the AI how important new units really are. . .

 

[Afforess]

This also leads to the in-game distortion of "I have 100 axemen and he has 75 macemen. We are clearly equal in Military Power!" on the power graph, which is ridiculous because 75 macemen will walk over 100 axemen like they aren't there.
.

The problem is that a weaker civ who can't build 75 macemen will continue to spam axemen, and ruin their economy until they get close to the power of 75 macemen.... I've watched it happen.

 

[Fimbulvetr]

Shouldn't that only happen if there's no other choice?

    [COLOR="Green"]// If losing badly in war, need to build up defenses and counter attack force[/COLOR]
    if( bLandWar && (iWarSuccessRatio < -30 || iEnemyPowerPerc > 150) )

Only if there's a land war and we are losing ..
And in CvCityAI::AI_cityThreat there is

                    // Beef up border security next to powerful rival
                    if( GET_PLAYER((PlayerTypes)iI).getPower() > GET_PLAYER(getOwnerINLINE()).getPower() )
                    {
                        iTempValue *= std::min( 400, (100 * GET_PLAYER((PlayerTypes)iI).getPower())/std::max(1, GET_PLAYER(getOwnerINLINE()).getPower()) );
                        iTempValue /= 100;
                    }

which I guess is what causes this? We could reduce that a bit..

 

[Afforess]

Shouldn't that only happen if there's no other choice?

    [COLOR="Green"]// If losing badly in war, need to build up defenses and counter attack force[/COLOR]
    if( bLandWar && (iWarSuccessRatio < -30 || iEnemyPowerPerc > 150) )

Only if there's a land war and we are losing ..
And in CvCityAI::AI_cityThreat there is

                    // Beef up border security next to powerful rival
                    if( GET_PLAYER((PlayerTypes)iI).getPower() > GET_PLAYER(getOwnerINLINE()).getPower() )
                    {
                        iTempValue *= std::min( 400, (100 * GET_PLAYER((PlayerTypes)iI).getPower())/std::max(1, GET_PLAYER(getOwnerINLINE()).getPower()) );
                        iTempValue /= 100;
                    }

which I guess is what causes this? We could reduce that a bit..

Sure, but there has to be a way to detect if the enemies power is due to a result of stronger units, or a quantity of weak units. If Player A only can build axes, and Player B has rifles, Player A shouldn't be spamming more axes (at least not to the detriment of it's economy). Player A should use a different strategy, either beginning a war of attrition or try to find peace ASAP.

 

[Fimbulvetr]

Weaker units means less power per military unit..
What about this:

                    [COLOR="Green"]// Beef up border security next to powerful rival[B], just not too much if we can only produce weaker units[/B][/COLOR]
                    if( GET_PLAYER((PlayerTypes)iI).getPower() > GET_PLAYER(getOwnerINLINE()).getPower() )
                    {
                        [B]int iTempMultiplier = [/B]std::min( 400, (100 * GET_PLAYER((PlayerTypes)iI).getPower())/std::max(1, GET_PLAYER(getOwnerINLINE()).getPower()) );
                        [B]iTempMultiplier = (iTempMultiplier + std::min(iTempMultiplier, ( (100 * GET_PLAYER((PlayerTypes)iI).getNumMilitaryUnits())/std::max(1, GET_PLAYER(getOwnerINLINE()).getNumMilitaryUnits()) ))) / 2;[/B]
                        iTempValue *= iTempMultiplier;
                        iTempValue /= 100;
                    }

No changes in the bahavior if our units are equal or better, only if ours are worse: enemy has 200% power and 100% unit number -> (200+100) / 2 = 150% total. enemy power 200%, unit number 50% -> 125%

 

[Afforess]

Sure, I guess. The real problem is that Civ4 AI is too decentralized, really....

 

[Fimbulvetr]

Yeah but I prefer to stick to what I know I can do. Creating a whole new smart centralized AI for Civ4 is a bit above what I am capable of, and if I can somewhat improve the situation with a single additional line, I will. Of course I'm open for critizism and better ideas here.

 

[Afforess]

Yeah but I prefer to stick to what I know I can do. Creating a whole new smart centralized AI for Civ4 is a bit above what I am capable of, and if I can somewhat improve the situation with a single additional line, I will. Of course I'm open for critizism and better ideas here.

I know. I wasn't asking you to... just lamenting a bit.

Anyway, hopefully that dedicated AI programmer for Civ5 was worth the money!

 

[Fimbulvetr]

Anyway, hopefully that dedicated AI programmer for Civ5 was worth the money!

I have the feeling you'll know soon enough. Just don't expect too much, we don't know how many wheels that poor man had to reinvent during the whole development process