Infinite turn / grouping loop (Tester.py)

[need my speed]

I have seen this thread and tried merging the Tester.py within with 'my own' (it's RevDCM's one). That didn't work, curiously enough; the game said something about indents, but as far as I could see there was nothing wrong.

I would say I have confirmed this is the actual error, for manually switching players and forcing the specific civilisation in question to end the turn of all its units solves the problem, as does deleting the civilisation through World Builder. Problem is, however, that with the first solution, the AI screws up my own civilisation, and with the second, well, a whole continent (South America) would suddenly be empty. I don't want to lose the civilisation in question, for it's consistently the best performing leader / civilisation by far in all my games; in this game, it conquered all of South America, vassalised most of the Americas, and even preformed a huge and successful naval invasion against me.

Anyway! I could solve this by either getting the Tester.py to work, I suppose, but I can't get it to work. Could someone do this for me? Attached are RevDCM's Tester.py and civjo's Tester.py. civjo's solution, a few key presses and their functions, should be integrated in RevDCM's one. I have zero idea why I couldn't do this on my own, but, alas. Thank you in advance!

 

[Dancing Hoskuld]

the game said something about indents, but as far as I could see there was nothing wrong.

It can appear that the code is correct if it just happens that the tab is the same as the number of spaces you use to indent but it is not necessarily interpreted as the same indentation when it runs. Always use either spaces or tabs and no mixture of them for indentation of your code.

 

[isenchine]

Would this work better for you?

With Notepad++, you can replace those spaces by tabs. Even if you don't quite see it in the process, you can copy/paste the number of spaces (usually 4) and copy/paste a tab as a replacement.

Edit: removed attachment: not valid.

 

[need my speed]

Mhm, I am actually not sure if that worked, as in, I saw nothing happening, but, the turn passed correctly... And then the exact same error came up six or so turns later. This, I cannot seem to solve; even when I delete France, which causes the error this time, I believe, later on in the inbetween turn it still crashes.

I suspect the frequency of these crashes has to do with the many naval actions the AI undertakes; so far I have seen two huge naval invasions, whereas normally I barely see this happening. But, I really have no idea - all I know is that all these crashes making playing impossible, and that I have no easy fix. A preventive fix would be preferable, but a consistently working reactive one would be fine too. Alas...

Perhaps anyone here knows more? Though that would be going through quite some effort I suppose, as I'm playing with my own mod. Here is my current savegame and here is my mod. Thanks a lot in advance.

 

[isenchine]

Do you still have that indent error? Or are you just talking about infinite loops on and on? And you have crashes now?

The CivJo thing is not supposed to prevent you from infinite loops, it is supposed to help you locate the problem with the unit that is causing the loop. Then, you can force it to stop its turn or to force all units of a player to stop their turn.

Also, depending on the key pushed, reports are written in the PythonDbg.log file. In the WB, you can then locate the unit and delete it.

 

[need my speed]

I don't even know anymore.

I started a new game, but always around the same time - when the AI starts using Galleons and Frigates and all - this crash or whatever it is pops up. Sometimes I manage to solve it, but then it pops up nine turns later, or indeed, the very next turn.

What happens is that, at some point during the inbetween turn time, the music stops playing, and the 'waiting for other civilizations' text stops blinking. I cannot find anything useful in the log files. My latest savegame, as well as my previous savegame, is here. And while my mod can still be found in my previous post in this topic, a slightly more recent version is here.

 

[Nightinggale]

This is one of the reasons why I always avoid using python for game logics. When using the dll and the game freezes, I can profile and see which loop use 100% of the cpu time, which works even if you have no idea what went wrong. I can also step through it using the debugger to see why the stop condition is never reached. Using only those two approaches, I have solved every single freeze I have encountered in the dll and it isn't even that hard or time consuming to do once you have figured out how to use the tools in question. The dll also provides the ability to use asserts, which helps to detect variables containing "incorrect" data.

This mean if this happened in the dll, I would presumably be able to locate the bug in a few minutes and might have a fix within 10 minutes, even without prior knowledge of the code in question. However this doesn't apply to python code. Thinking hard I can't think of anything other than the primitive approach of just looking at the code and hope to spot the mistake, which is time consuming and partly relies on luck. How do you guys hunt python bugs, which doesn't trigger exceptions?

 

[need my speed]

I do have a dump of a .exe.dmp thing, I just have no idea how to actually use it; a tutorial I read uses a different version of Microsoft Visual Studio. Does this help?

 

[platyping]

How do you guys hunt python bugs, which doesn't trigger exceptions?

Bugs which do not trigger exceptions:

1) Logic Errors. You want to +1, but -1 instead.
No solution, read through codes.

2) Game hangs. Infinite loop.
Generally, this will happen only for while loops. Just look for "while".

3) Game crashes.
When you try to do something to an object that no longer exists.
Add in some null checks.

In general, if you are the coder, you test bit by bit while you mod.
For instance, when I code my pedia, I don't write the whole pedia before I test. That is suicidal...

If you are importer, then you better test the standalone mod comp fully before you import. Even if codes are written by established coders, they can still come with bugs.

 

[isenchine]

That and printing values of variables to the PythonDbg.log file.

need my speed: unfortunately, your mod is huge. I thought of looking at it today but there's no way that I can run it on the old laptop I have now. Maybe tomorrow I can transfer it to my home desktop (not connected to the intenet).

However, the only thing that I can really do is to try to locate the problematic unit.

Nightinggale: I suggest that you read the thread referred to in the first post if you want to have a better picture of the problem. Locating the problematic code in the dll is one thing but to prevent the AI from endlessly changing its mind about what to do is another thing. The Python code in question serves only to locate the unit(s) that is(are) part of the infinite loop problem (it's not a "freeze" or CtD).

need my speed, one question: can you provide a debug dll? Do you even have the source code files of the version of RevDCM you are using?

You're talking about Galleons and Frigates and it makes me think that it was reported more than once that Privateers are the culprit, can't say if it's because of their hidden nationality, the "always hostile" tag or the UNITAI_PIRATE_SEA. This said, I have same Privateers in my mod and apparently without problems.

It was also reported that it could happen with very specific unit combinations, like Galleys joining Galleon groups. Grouping/ungrouping units seem a big issue for the AI and can create infinite loops. Perhaps it is also the case with Privateers joining a selection group of Galleons, Frigates.

 

[need my speed]

I know about the Privateer thing, indeed. I also know about the Galleons grouping with Galleys, Herumor (of Harad) actually did that the previous turn, around where the Mediterranean Sea connects with the Atlantic Ocean - but when I forced a peace between him and Peter (of Russia) things were fine.

The source files are here. I would investigate how to get such a debug .dll - apparently it isn't the above .dmp file? - but I need to leave for an hour or so now, sorry.

 

[isenchine]

A debug dll is a CvGameCoreDLL.dll file about three times the size of a normal one. It is the same source code files compiled in one dll in a different way.

With a debug dll, you can run the game from within the compiler and have all sorts of tools at your disposal to debug your game. That's Nightinggale's job!

You can also just replace your dll with the debug dll. Your game will run slower but then you can get important messages/hints (asserts) about potential problems. Like that, you can already solve some crashes. It's not really helpful in the case of an infinite loop, but we can't rule out yet that you don't have other problems than that.

If you can find a debug dll that can run your game, it would be easier. Otherwise, we have to compile one from the source files you provided. I would volunteer to do that tomorrow but I have one problem with that. I'm not an informatician and I don't quite understand all the magic involved. I usually end up with projects in the compiler that have all the same name CvGameCoreDLL and don't know how to differentiate them easily (I can't rename them). I even have a project like that with zero files (all deleted) but I can't get rid of the line in the list of projects! Nightinggale, do you have any tips on this (I have MS Visual C++ 2008 Express)?

 

[Nightinggale]

All this writing about the dll makes me wonder if the problem is in python or the dll. It is triggered by a python change, but python calls the dll. Also it would be possible to analyze the dll at runtime to tell which functions python calls and that way get an indication of what python is doing. For all we know the dll could act up due to being called with an invalid argument and the dll ends up in an undefined state. Not looking into the dll might be a mistake, particularly since it seems that we don't really have tools to analyze what python is doing at runtime.

the game said something about indents, but as far as I could see there was nothing wrong.

Try to set Notepad++ to show whitespace. It might be due to replacing space with tabs or vice versa.

A debug dll is a CvGameCoreDLL.dll file about three times the size of a normal one. It is the same source code files compiled in one dll in a different way.

Strictly speaking, it is compiled in the same way, except the debug dll skips the last step (optimization) and instead it has extra data to tell which line in the source belongs to which instruction in the compiled dll. Other than that they should be identical.

That's Nightinggale's job!

I had a job interview? and I passed? I remember nothing of that. I must be getting

You can also just replace your dll with the debug dll. Your game will run slower but then you can get important messages/hints (asserts) about potential problems.

Makefile 2 can make a hybrid called assert. It is an optimized dll with asserts enabled. It is slower as it has to perform the assert checks, but it's almost as fast as release dlls (regular dlls). The intended goal for this is to always use this type when playing your own mod, which helps to catch bugs. It will require a proper debug dll if you want to attach the debugger, but most failed asserts can he dealt with without the debugger.

Even xml only modders benefits from using an assert dll. For instance if you give a free promotion PROMOTION_RANGGER, it will trigger an assert due to being an unknown string while a release dll will just silently fail and change it to NO_PROMOTION. The vanilla assert text for this error is rather poor and is aimed at people who debug right away. However it's still better than silently fail and a worst case scenario is for people to show up here and ask for help due to an unreadable assert. Perhaps a bit annoying, but still better than what the release dll would do.

It's not really helpful in the case of an infinite loop, but we can't rule out yet that you don't have other problems than that.

That depends on the code. It is possible that an infinite loop will trigger an assert. I think it is vanilla code, which has a loop to decide what an AI unit should do. It's intended to run 1-3 times and it asserts when it has tried 100 times and still haven't decided. Other than that, as I mentioned before the failproof dll infinite loop detection is to profile (also with makefile 2). Being told how much time it spend on each line reveals which loop which eats up 100% of the time. It's impossible to miss.

If you can find a debug dll that can run your game, it would be easier. Otherwise, we have to compile one from the source files you provided.

I have bad experience with using debug dll files provided by other people. It's like there are hardcoded paths or something in them. However they always just work when I compile myself.

The solution to this problem, which works best in my experience is to place the mod in git. Other people can then pull all the files and if set up correctly, everybody can compile out of the box and run their own tests. It makes fixing a problem like this one easier and faster and it doesn't involve as much guesswork. Git also helps in case you end up with "this feature is broken. It worked last weekend. How did I break it?" as it logs all changes down to each line change.

I usually end up with projects in the compiler that have all the same name CvGameCoreDLL and don't know how to differentiate them easily (I can't rename them).

I don't see the problem in having the same name for all project files except it might be a bit confusing if you have multiple projects open at the same time. Luckily it is quite easy to rename a project.

Close the project. You really don't want to keep the files in memory while doing this. Also backup in case something goes wrong.

The project files are partly autogenerated, which mean the only files you need for distribution (and hence renaming) are:
CvGameCoreDLL.sln
CvGameCoreDLL.vcxproj
CvGameCoreDLL.vcxproj.filters

Filters is poorly named as it is really the folder structure for the source file. If you don't use it and have all files in one "folder", the file will not be present.

Last open the .sln file. It will have the following line near the top:

Project("{some number}") = "CvGameCoreDLL", "CvGameCoreDLL.vcxproj", "{another number}"

The numbers appear to be random or hashes of something. I never touched those. The first string is what the project is called internally while the next is the name of the vcxproj file. Rename both to what you want. Save and the new name should be there.

I'm using msvc 2010, but it should be the same in 2008, though I remember something about vcproj file extensions as it was 2010, which added the x.

 

[isenchine]

I meant to write this more clearly before but why do you assume it is a Python problem? If I understand correctly, need my speed had a problem with an infinite loop, remembered the thread referred to in his first post and tried to merge the Tester.py solution file in his mod. Doing that, he got the "indent" error.

That Tester.py is a tool to determine which unit is currently "playing" during the infinite loop, then you can delete it in WB and resume your game. Otherwise, you are stuck and can't finish your game - very frustrating. It's not meant as an elegant solution, just as a quick fix to move on to the next turn.

Also, I should have written: "It's Nightinggale's domain!". But it can be your job if you find all this interesting enough!

Thanks a lot for the tips, I will try them tomorrow.

 

[need my speed]

Mhm-hm, the original error has been solved, but, I do not have either a preventive fix nor a guaranteed-to-work fix (such as the advertised Tester.py, allowing me to find out the cause and fix that), and considering a new error happened about nine turns later and then one turn later... I wonder why my mod suffers so from it, and how I can prevent it.

 

[Nightinggale]

I meant to write this more clearly before but why do you assume it is a Python problem? If I understand correctly, need my speed had a problem with an infinite loop, remembered the thread referred to in his first post and tried to merge the Tester.py solution file in his mod. Doing that, he got the "indent" error.

Rereads posts
Yeah it doesn't explicitly say it's a python problem. However all the code I see is python, which tricked me. However without the source for the dll, it is impossible to tell what goes on.

I will say again that providing easy access to the mod (including source code) is the way to go if other people should investigate a problem. Git works great for something like that, but it's not the only solution.

 

[isenchine]

At least, we do have the source files (see post #11).

 

[Nightinggale]

At least, we do have the source files (see post #11).

Yeah, but it requires access to the full mod to run tests. One silly thing about this thread is that if I had just gained the mod and a savegame, which freezes I could likely have found the cause within the time I have already spent reading the thread. I don't mind assisting people to make them solve the problems themselves, but somehow needing somebody else to compile a debug dll doesn't sound like a promising start.

 

[isenchine]

The full mod is in post #4 and again #6. If the Source files in post #11 do not work, there is another set (with other dates) in the full mod. Tomorrow, I will continue to see if I can continue the savegames with a debug dll. I could play a few turns for both savegames, now it crashes after the end of turn - I don't understand what's going on.

EDIT: I was tired: the set of source files is the same - sorry about that, I mixed it with a copy of Legends of Revolution that was on my computer. What is missing however is the file CvGameCoreDLL.h. I took the one from BtS.

@ need my speed: in the meantime, I have reviewed the Tester.py file. In the last version, I forgot one important line. Try to play with the combination of keys. Alt+0, 1, 2 or 3 should bring you an immediate message on screen. Rename the Tester_isenchineFEB to Tester of course.

 

[Nightinggale]

I had a job interview? and I passed? I remember nothing of that. I must be getting

Now it's official. I'm old and blind. I didn't notice the links to the mod itself. In the future if I am to assist, people should just give me the link, preferably to a git server and then short and simple state the problem. That will allow me to dig right into the code. All this human talk is tricky and confusing

Now I downloaded the rar files and a brief look indicates I should have everything. I will set up makefile 2.5+project files and I should be able to properly investigate this issue.