[Dev] Some questions about Python development

dbkblk · Jan 23, 2015

Hi !
While merging Unicode patch for CivIV, i encountered some weird errors in a python module that we added: ANDAutomationsTab.py
Its goal is to create an automation tab in the BUG options.
While playing in Russian, the creation fails with:

Code:

Traceback (most recent call last):

  File "CvScreensInterface", line 947, in handleInput

  File "CvMainInterface", line 6596, in handleInput

  File "BugOptionsScreen", line 42, in showOptionsScreen

  File "BugOptionsScreen", line 142, in interfaceScreen

  File "BugOptionsScreen", line 149, in createTabs

  File "ANDAutomationsTab", line 286, in create

UnicodeDecodeError: 'ascii' codec can't decode byte 0xcf in position 0: ordinal not in range(128)
ERR: Python function handleInput failed, module CvScreensInterface

The involved code is:

Code:

for iI in range(gc.getNumBuildInfos()):
			if (team.isHasTech(gc.getBuildInfo(iI).getTechPrereq())):

				columnKey = self.determineColumnPlacement(5, iCount, (col1, col2, col3, col4, col5))

				control = str(remove_diacriticals(player.getNameKey() + gc.getBuildInfo(iI).getDescription() + "Check"))
				bEnabled = player.isAutomatedCanBuild(iI)

				szNewDescription = g_BuildNames[iI]
				screen.attachCheckBox(columnKey, control, szNewDescription , "CvOptionsScreenCallbackInterface", "handleNationalAutomatedBuildCheckboxClicked", control, bEnabled)
				screen.setToolTip(control, BugUtil.getText("TXT_KEY_NATIONAL_AUTOMATED_WORKERS_CAN_BUILD", szNewDescription))

				iCount += 1

Now that the names are coming in local codepage, there are others characters than ASCII. I would willingly accuse UTF as the guilty, but all others BUG tabs correctly render UTF now...
Any idea ?

EDIT: I think this loop makes the game to crash:

Code:

	while(loopCity):
		if (szName.rfind(ANDAutomationsTab.remove_diacriticals(loopCity.getName())) > -1):
			for iI in range(CyGlobalContext().getNumBuildInfos()):
				if (szName.rfind(CyGlobalContext().getBuildInfo(iI).getDescription()) > -1):
					CyMessageControl().sendModNetMessage(AutomatedSettings.getCanAutoBuildEventID(), CyGlobalContext().getGame().getActivePlayer(), loopCity.getID(), iI, int(bValue))
					#loopCity.setAutomatedCanBuild(iI, bValue)
					return 1
		(loopCity, iter) = player.nextCity(iter, False)

	return 0

I don't understand what is the point of this. Couldn't we loop over CityID instead of names ? That would be more crash-proof.

Afforess · Jan 23, 2015

dbkblk,

Looking at the code, I agree city ids would be a much better solution than the currently used city names. You would need to update the code that handles the ModNetMessage to use the city id as well.

dbkblk · Jan 23, 2015

@Afforess: Thank you for your answer. I have fixed it in another way by enabling python UTF8 support, replacing "szNewDescription" by "CvUtil.convertToUnicode(szNewDescription)" when invoking "setToolTip" and by removing "remove_diacriticals" which is counter-productive with UTF8.

Red Ant · Jan 23, 2015

szNewDescription ... Hungarian notation in Python code?

sz used to stand for "string - zero-terminated" ... makes sense when using C-style strings in C or C++, but in Python?

dbkblk · Jan 23, 2015

I think Zappara was used to C++ coding when he added this into the code

Afforess · Jan 23, 2015

dbkblk said:
I think Zappara was used to C++ coding when he added this into the code

Zappara never did any C++ coding. Hungarian notation is just the style throughout all the Civ4 .

dbkblk · Feb 8, 2015

@all: Is there any trick to do python debugging ? I would like to catch variables like in C++ debugging but i have no idea how to do it.
I've tried HAPDebugger but it gives me nothing more useful than standard python in-game debug messages.

platyping · Feb 9, 2015

For me, python exceptions is enough.
Of course, this will not catch any logic errors, or OOS errors. Logic errors you can never catch unless you read the codes yourself or encounter it while playing.

dbkblk · Feb 9, 2015

For sure, but we still have to "guess" the variable value. Isn't there any way to catch the values ?

platyping · Feb 9, 2015

When I need to know what are the values halfway through the codes, I simply add temporary codes to display the values on the screen itself.

oldnooob · Feb 9, 2015

@Platyping, do you see anything interesting in this log? I have been asking about these errors in several forums to no avail.

platyping · Feb 9, 2015

I have never bothered to read python logs when exceptions are enough for me to solve my bugs.
Anyway, logs make sense only to modders of the mod.

dbkblk · Mar 9, 2015

Does someone knows if this is possible to put gamefont icons in signs ?

masaykh · Mar 9, 2015

in PythonDbg.log
i see load_module encodings.ascii

but why not utf? unicode? or even russian encoding?
if you open Assets\Python\System\encodings\
you see all of them..

dbkblk · Mar 10, 2015

This is not that easy. Python probably loads an ASCII module but it uses UTF8. The problem we had is the game DOES NOT support UTF8 or Unicode. Nightinggale and I found a trick to convert UTF8 to the locale in use at init time to be able to use utf8 in the xml text files. The python part of the game fully support utf8, but the game engine doesn't and we don't have access to this. To resume, game reads utf8, convert it to the locale in use (windows codepage), then python reads the locale. There are no problems of char display in normal text.

The second major issue, is that the game used the Gamefont to render text on signs and citybillboards (you know, the bold text which looks different). Basically, the letters are images that are pasted next to each others instead of be rendered normally. You may haven't noticed that in the old Russian patch because all the Russian chars were entered in the Gamefont but we don't have any Russian entered here and made the city billboard to use the standard rendering, thus making it more universal. Editing Gamefont is a real... pain... To understand why we did that, just imagine a second to enter the 2000+ ideograms from Chinese

. The signs cannot be turned into rendered chars, or we haven't found the trick.

To conclude, python just reads what we give to him and is utf8 enabled already.

[Dev] Some questions about Python development

dbkblk

Emperor

Afforess

The White Wizard

dbkblk

Emperor

Red Ant

Chieftain

dbkblk

Emperor

Afforess

The White Wizard

dbkblk

Emperor

platyping

Sleeping Dragon

dbkblk

Emperor

platyping

Sleeping Dragon

oldnooob

Prince

platyping

Sleeping Dragon

dbkblk

Emperor

masaykh

Prince

dbkblk

Emperor

Similar threads