FF Where is the bottleneck (Jon?)

[somebody_else2]

Hello,

I've been modding the FF AI so that it would be more competitive. So far I have a very aggressive settling phase done (with own settler AI) and debugged. It now also builds a lot more fighters and bombers. However that is not my point.

I'm fighting with long turn times in the later part of the game. I have stopped the time that is spend in the main functions of CvAi. I come to about 13 seconds of total running time for about 70 turns. However in the worst case I have turn times of 13 seconds so it's not the AI which makes things slow.

Does somebody have an idea where the bottleneck could be? (Jon?)

 

[Maniac]

The whold mod is in python. That makes it slow.

 

[somebody_else2]

It's mostly a visual mod and AI mod. The other things work just as normal. In between turns the visual part is not update so it can't slow things down. The AI isn't either the cause (as I initially suspected) so what causes this slowdown?

 

[Jon Shafer]

It could be something in the AI, just not in the same place. I think the biggest expense may be when the AI tries to figure out what to do with its Construction Ships and where to build Starbases. Have you checked that?

Jon

 

[somebody_else2]

PROFILE : Spent 122ms in doConstructionShipAI, in total 1729022ms

Note that I changed the resolution from milliseconds to microseconds. Those 1729022 are therefore about 2 seconds for about 70 turns. That's fast enough. Considering that there are never that many ships around it can't be a show stopper.

doCityAIUpdate is pretty slow, of those 13 seconds I have profiled 12 were spent there. Is it even called with the last parameter being non -1? Here a "mark population as dirty if something changes and only reassign when dirty" strategy would solve the problem.

I have not checked the starbase AI. However there were no Starbases in the whole test game I played. (AI doesn't have enough gold as it expands too fast.)

I'm also getting heaps of error (about 0.3MB) in xml.log of the sort:

[43293.172] info type UNIT_PRIVATEER not found, Current XML file is: xml\GameInfo/CIV4EspionageMissionInfo.xml
[43293.172] info type UNIT_SPY not found, Current XML file is: xml\GameInfo/CIV4EspionageMissionInfo.xml
[43293.219] info type UNIT_PRIVATEER not found, Current XML file is: xml\GameInfo/CIV4EspionageMissionInfo.xml
[43293.219] info type UNIT_SPY not found, Current XML file is: xml\GameInfo/CIV4EspionageMissionInfo.xml
[43293.250] info type UNIT_PRIVATEER not found, Current XML file is: xml\GameInfo/CIV4EspionageMissionInfo.xml
[43293.250] info type UNIT_SPY not found, Current XML file is: xml\GameInfo/CIV4EspionageMissionInfo.xml
[43293.250] info type BONUS_COPPER not found, Current XML file is: xml\GameInfo/CIV4EspionageMissionInfo.xml
[43293.250] info type BONUS_HORSE not found, Current XML file is: xml\GameInfo/CIV4EspionageMissionInfo.xml

Those seem to come from the spy AI which as far as I know is not handled inside the mod so those come from the default AI. Perhaps overriding the corresponding function in Python with dummy do nothing functions would solve the problem. Here I however do not have the necessary insider info to know what to override.

 

[Jon Shafer]

Cool, I'll take a look at that...

Jon

 

[somebody_else2]

By changing CvEventInterface.py to

Spoiler :

# Final Frontier
# Civilization 4 (c) 2007 Firaxis Games

# Designed & Programmed by:	Jon 'Trip' Shafer

import CvUtil
import CvEventManager
import CvFinalFrontierEvents
from CvPythonExtensions import *

ModEventManager = CvFinalFrontierEvents.CvFinalFrontierEvents()
normalEventManager = CvEventManager.CvEventManager()


from profile import Profile
from pstats import Stats
from StringIO import StringIO
import sys

theStats = None
onEventCalls = 0

def getEventManager():
	return ModEventManager

def onEvent(argsList):
	'Called when a game event happens - return 1 if the event was consumed'

	global onEventCalls
	global theStats

	def real_onEvent():
		return getEventManager().handleEvent(argsList)

	#return real_onEvent()

	theProfiler = Profile()
	try:
		ret = theProfiler.runcall(real_onEvent)
	except AssertionError:
		# Crashs once for onTechAcquired. I don't know why but
		# it doesn't seem to cause problems. Perhaps a profiler bug.
		print "Profiler didn't like the " + argsList[0] + " event"
		return
	if theStats == None:
		theStats = Stats(theProfiler)
	else:
		theStats.add(theProfiler)

	if onEventCalls > 1000:
		old_stdout = sys.stdout
		sys.stdout = StringIO()
		theStats.strip_dirs().sort_stats('time').print_stats().print_callers()
		stats = sys.stdout.getvalue()
		sys.stdout = old_stdout
		print stats
		onEventCalls = 0
		
	onEventCalls += 1
	return ret
	
def applyEvent(argsList):
	context, playerID, netUserData, popupReturn = argsList
	return getEventManager().applyEvent(argsList)

def beginEvent(context, argsList=-1):
	return getEventManager().beginEvent(context, argsList)

You can get detailed profiling information about the Python code.

 

[somebody_else2]

Here are the modifications I spoke of earlier. I probably wont have the time necessary to work much on them in the near future therefore I upload what I have now.

What has changed:

• Very aggressive settling phase that sends out unprotected settlers to get more solar systems
• Completely rewrote the construction ship AI. It now will build quiet a lot starbases and it will build roads to other civs with open borders to allow trade. The starbase AI is now also very fast. (About 3-5 starbases per civ on )
• City Production selection rewritten. The weights on what to build need work but as soon as that's decided it works pretty well. The AI will build a lot squadrons now. In fact it can be decided on a per civ basis of what and how much it should compose it's army. Implementing battleship or fighter loving AIs is possible.
• No junk systems will be placed on the map anymore. Junk systems simply have way to much impact on the game. If you start in a junk system the game is basically over.
• Optimized the pirate spawning function a lot. It's effect however didn't change.

Problems:

• The AI doesn't know any victory except conquest. It doesn't try to go for culture, the spaceship equivalent it doesn't build the UN either.
• Worker and unprotected settlers have no fear any more. They do not flee when they see an enemy.
• Multiplayer will not work.
• The AI doesn't take it's trait bonuses into consideration when deciding what to build.
• The AI is incapable of using the invisible ships and doesn't use explores to locate them. The main reason is that I don't like the idea. It basically forces you to place an explorer in each city for defense or do I see that wrong?

In the attached zip I included:

• The modded CvFinalFrontierEvents.py, CvSolarSystem.py and CvGameUtils.py . Here all changes are marked with #MOD START and #MOD END.
• The new CvAI.py Here there have been so many changes that I stopped marking them
• The CvEventInterface.py with the profiler. For actual playing use the original file.

Note that you also need to enable the onUnitCreated and onUnitLost Events in the XML.

The Code is in a playable state although not much tested.

 

[somebody_else2]

Somehow the attachments didn't work

 

[Jon Shafer]

One possible way to address the "doCityAIUpdate" slowdown would be to remove the call made for every city at the start of each turn. This would mean that a city's population would only be completely reshuffled when founded or attacked. Other than that, only new Population will be assigned. I'm not sure how much of a detrimental effect, if any, this would have on the AI's performance though so I'm hesitant to enact this. I'll be doing some more testing on this to get a clearer answer.

Jon