Single Player bugs and crashes v38 plus (SVN) - After the 20th of February 2018

@Thunderbrd since that two cultural units had errors (wrong path, I reported this one) I can fix them now, since their art never was in FPK - I didn't knew, that ALL module art was left unpackaged.
Ok, yes if you want to fix a file path error in the xml, go for it.
 
@Thunderbrd a valuable patch before the eventual release:

Quoting @billw2015 on the discord channel:
I have a game that went from < 30s per turn immediately to > 40m per turn. https://1drv.ms/u/s!Aj1eMcUFhShGhDKiuLSFFsQ4LZVd
billw BC-0978.CivBeyondSwordSave
I'm on SVN 10610
i have per turn saves so i could narrow down the turn on which it happened, but i don't think it will be that diagnostically relevant without some ideas of what might cause this problem
He provided his save to evaluate in the link above.
Two hours later...
oh damn the VS diagnostic tools work without any problem, wasn't expecting that
Code:
int CvUnit::getHealRateAsType(const CvPlot* pPlot, bool bHealCheck, UnitCombatTypes eHealAsType) const
98% time spend in there over a 2 minute period
all of it iterating through movedUnitIds
someone used std::find on a set instead of set.find
guess maybe that was it, down from 40+ m to <5m, still lame though

Patch to optimize the std sets
He patched the search method of finding in a std::set from using std::find on a std::set to using std::set::find function instead.
https://cdn.discordapp.com/attachme...9/597902376963407888/StdSetOptimization.patch
In case you can't download this patch, create a text file ending in .patch and use TortoiseSVN function to apply the patch:
Code:
Index: Sources/CvSelectionGroup.cpp
===================================================================
--- Sources/CvSelectionGroup.cpp    (revision 10658)
+++ Sources/CvSelectionGroup.cpp    (working copy)
@@ -8705,7 +8705,7 @@
 
         while (pUnitNode != NULL)
         {
-            if (movedUnitIds.empty() || std::find(movedUnitIds.begin(), movedUnitIds.end(), pUnitNode->m_data.iID) == movedUnitIds.end())
+            if (movedUnitIds.empty() || movedUnitIds.find(pUnitNode->m_data.iID) == movedUnitIds.end())
             {
                 pLoopUnit = ::getUnit(pUnitNode->m_data);
 
Index: Sources/CvUnit.cpp
===================================================================
--- Sources/CvUnit.cpp    (revision 10658)
+++ Sources/CvUnit.cpp    (working copy)
@@ -9049,7 +9049,7 @@
 
                     while (pUnitNode != NULL)
                     {
-                        if (movedUnitIds.empty() || std::find(movedUnitIds.begin(), movedUnitIds.end(), pUnitNode->m_data.iID) == movedUnitIds.end())
+                        if (movedUnitIds.empty() || movedUnitIds.find(pUnitNode->m_data.iID) == movedUnitIds.end())
                         {
                             pLoopUnit = ::getUnit(pUnitNode->m_data);
 
@@ -19462,7 +19462,7 @@
 
                 while (pUnitNode != NULL)
                 {
-                    if (movedUnitIds.empty() || std::find(movedUnitIds.begin(), movedUnitIds.end(), pUnitNode->m_data.iID) == movedUnitIds.end())
+                    if (movedUnitIds.empty() || movedUnitIds.find(pUnitNode->m_data.iID) == movedUnitIds.end())
                     {
                         pLoopUnit = ::getUnit(pUnitNode->m_data);
FYI dunno if it shows any other issues but the enemy moved 650 units that were then being looked up in this set to calculate healing (modifiers I guess?). 650 moved units sounds insane in 978 BC but I don't know.
so the original round that took ages takes 3 m compared to further rounds that take 1m so there is definite still some room for improvement. i expect there is some caching that could be done here to elimiate the cost of these lookups

I tested his patch too. I can definitely finish his turn now, in 6 minutes on my computer.

My comment:
Using std::set::find reduces the search time because of the internal structure of a std::set.
While std::find performs a linear search to find the element, the std::set::find function performs a red-black tree search so it's much more efficient.
If the needle does not exist in a set, doing std::find has to traverse the whole list, while doing std::set::find moves through only a certain branch of the tree before the search ends.
This has more complication when the set is large enough.
 
Last edited:
I want to include a quick and safe fix to NPC's being listed in the greatest works ranking list (top 8 civ) that pops up every now and then in game.
Notably annoying is it that the Neanderthal NPC civ usually tops those list.

Just mentioning it in case TB suddenly announce a new release before I get it in, I'm currently a bit stuck in PPIO modding so I won't do it before I get that wrapped up (max 24 hours).
 
@Thunderbrd a valuable patch before the eventual release:

Quoting billw on the discord channel:


He provided his save to evaluate in the link above.
Two hours later...





He patched the search method of finding in a std::set from using std::find on a std::set to using std::set::find function instead.
https://cdn.discordapp.com/attachme...9/597902376963407888/StdSetOptimization.patch
In case you can't download this patch, create a text file ending in .patch and use TortoiseSVN function to apply the patch:
Code:
Index: Sources/CvSelectionGroup.cpp
===================================================================
--- Sources/CvSelectionGroup.cpp    (revision 10658)
+++ Sources/CvSelectionGroup.cpp    (working copy)
@@ -8705,7 +8705,7 @@
 
         while (pUnitNode != NULL)
         {
-            if (movedUnitIds.empty() || std::find(movedUnitIds.begin(), movedUnitIds.end(), pUnitNode->m_data.iID) == movedUnitIds.end())
+            if (movedUnitIds.empty() || movedUnitIds.find(pUnitNode->m_data.iID) == movedUnitIds.end())
             {
                 pLoopUnit = ::getUnit(pUnitNode->m_data);
 
Index: Sources/CvUnit.cpp
===================================================================
--- Sources/CvUnit.cpp    (revision 10658)
+++ Sources/CvUnit.cpp    (working copy)
@@ -9049,7 +9049,7 @@
 
                     while (pUnitNode != NULL)
                     {
-                        if (movedUnitIds.empty() || std::find(movedUnitIds.begin(), movedUnitIds.end(), pUnitNode->m_data.iID) == movedUnitIds.end())
+                        if (movedUnitIds.empty() || movedUnitIds.find(pUnitNode->m_data.iID) == movedUnitIds.end())
                         {
                             pLoopUnit = ::getUnit(pUnitNode->m_data);
 
@@ -19462,7 +19462,7 @@
 
                 while (pUnitNode != NULL)
                 {
-                    if (movedUnitIds.empty() || std::find(movedUnitIds.begin(), movedUnitIds.end(), pUnitNode->m_data.iID) == movedUnitIds.end())
+                    if (movedUnitIds.empty() || movedUnitIds.find(pUnitNode->m_data.iID) == movedUnitIds.end())
                     {
                         pLoopUnit = ::getUnit(pUnitNode->m_data);



I tested his patch too. I can definitely finish his turn now, in 6 minutes on my computer.

My comment:
Using std::set::find reduces the search time because of the internal structure of a std::set.
While std::find performs a linear search to find the element, the std::set::find function performs a red-black tree search so it's much more efficient.
If the needle does not exist in a set, doing std::find has to traverse the whole list, while doing std::set::find moves through only a certain branch of the tree before the search ends.
This has more complication when the set is large enough.

Well that was a new thing on me, using a patch with the SVN function for applying one. I seem to have been able to work it out I think.

I don't usually mess around with those kinds of while loops unless it is to copy what has been done with an understanding enough to duplicate a method. Trying to write while loops is usually one of those things that I avoid out of fear of doing them wrong somehow. This:
Code:
if (movedUnitIds.empty() || movedUnitIds.find(pUnitNode->m_data.iID) == movedUnitIds.end())
makes sense to me but this (the original):
Code:
if (movedUnitIds.empty() || std::find(movedUnitIds.begin(), movedUnitIds.end(), pUnitNode->m_data.iID) == movedUnitIds.end())
does not. Somehow it does not surprise me that it's very slow in comparison as it seems to be unnecessarily complicating things there.

All this basically means I would never have spotted this in a million years and I pray we can find a lot more like it to help speed things up in general. I'll try to keep an eye out for this. Not sure by who or when this method was originally established. I might have continued its use if it was an example of what I needed somewhere.

I CAN say that many of those units that are doing heal checks at such an early era are animal and barb units in the wild. That count of units is not surprising to me in the least. Sounds like this will help to speed up later era turns a LOT.

I want to include a quick and safe fix to NPC's being listed in the greatest works ranking list (top 8 civ) that pops up every now and then in game.
Notably annoying is it that the Neanderthal NPC civ usually tops those list.

Just mentioning it in case TB suddenly announce a new release before I get it in, I'm currently a bit stuck in PPIO modding so I won't do it before I get that wrapped up (max 24 hours).
Ok, I'll update what I have as soon as I've fixed things up but I'll wait to bundle the final release until you've got that in place. Please make haste. I was going to upload v39 tonight so I'm holding off for ya here. Probably good to give us a little playtesting time anyhow.
Get with DH, i believe he knows best about modular art stuff....
I think I know what he would say from this discussion having been brought up a long while back. At the moment, I'm making an exec decision to simply allow those art files to remain in the modules loose as they are. I believe that's what he would advise if he were to visit here now. He might even get aggravated I thought to do otherwise. lol
 
Ok, I'll update what I have as soon as I've fixed things up but I'll wait to bundle the final release until you've got that in place. Please make haste. I was going to upload v39 tonight so I'm holding off for ya here. Probably good to give us a little playtesting time anyhow.
Got it done faster than I thought.
@raxo2222 would you mind checking if it works, I think autoplay skips the pop up, so to be sure you get it you would have to stop the autoplay 2 turns before a turn increment of 50 (turn: 48, 98, 148, etc.) and play a turn or two to get it.
I know the neanderthals won't show up if the list shows at all, it's just to make sure that no python errors appear due to some slip of the finger on my keyboard. ^^
 
I think I know what he would say from this discussion having been brought up a long while back. At the moment, I'm making an exec decision to simply allow those art files to remain in the modules loose as they are. I believe that's what he would advise if he were to visit here now. He might even get aggravated I thought to do otherwise. lol

I knew it also, but just wanted to be sure, and yes u r correct, stay in Modules folder. .
 
Got it done faster than I thought.
@raxo2222 would you mind checking if it works, I think autoplay skips the pop up, so to be sure you get it you would have to stop the autoplay 2 turns before a turn increment of 50 (turn: 48, 98, 148, etc.) and play a turn or two to get it.
I know the neanderthals won't show up if the list shows at all, it's just to make sure that no python errors appear due to some slip of the finger on my keyboard. ^^
It works correctly :)
At least PPIO version.

Without PPIO this works correctly too :)
 
Last edited:
@Thunderbrd a valuable patch before the eventual release:

Quoting billw on the discord channel:


He provided his save to evaluate in the link above.
Two hours later...





He patched the search method of finding in a std::set from using std::find on a std::set to using std::set::find function instead.
https://cdn.discordapp.com/attachme...9/597902376963407888/StdSetOptimization.patch
In case you can't download this patch, create a text file ending in .patch and use TortoiseSVN function to apply the patch:
Code:
Index: Sources/CvSelectionGroup.cpp
===================================================================
--- Sources/CvSelectionGroup.cpp    (revision 10658)
+++ Sources/CvSelectionGroup.cpp    (working copy)
@@ -8705,7 +8705,7 @@
 
         while (pUnitNode != NULL)
         {
-            if (movedUnitIds.empty() || std::find(movedUnitIds.begin(), movedUnitIds.end(), pUnitNode->m_data.iID) == movedUnitIds.end())
+            if (movedUnitIds.empty() || movedUnitIds.find(pUnitNode->m_data.iID) == movedUnitIds.end())
             {
                 pLoopUnit = ::getUnit(pUnitNode->m_data);
 
Index: Sources/CvUnit.cpp
===================================================================
--- Sources/CvUnit.cpp    (revision 10658)
+++ Sources/CvUnit.cpp    (working copy)
@@ -9049,7 +9049,7 @@
 
                     while (pUnitNode != NULL)
                     {
-                        if (movedUnitIds.empty() || std::find(movedUnitIds.begin(), movedUnitIds.end(), pUnitNode->m_data.iID) == movedUnitIds.end())
+                        if (movedUnitIds.empty() || movedUnitIds.find(pUnitNode->m_data.iID) == movedUnitIds.end())
                         {
                             pLoopUnit = ::getUnit(pUnitNode->m_data);
 
@@ -19462,7 +19462,7 @@
 
                 while (pUnitNode != NULL)
                 {
-                    if (movedUnitIds.empty() || std::find(movedUnitIds.begin(), movedUnitIds.end(), pUnitNode->m_data.iID) == movedUnitIds.end())
+                    if (movedUnitIds.empty() || movedUnitIds.find(pUnitNode->m_data.iID) == movedUnitIds.end())
                     {
                         pLoopUnit = ::getUnit(pUnitNode->m_data);



I tested his patch too. I can definitely finish his turn now, in 6 minutes on my computer.

My comment:
Using std::set::find reduces the search time because of the internal structure of a std::set.
While std::find performs a linear search to find the element, the std::set::find function performs a red-black tree search so it's much more efficient.
If the needle does not exist in a set, doing std::find has to traverse the whole list, while doing std::set::find moves through only a certain branch of the tree before the search ends.
This has more complication when the set is large enough.

The two code segments containing these usages of std::find should have been removed a long time ago. @Thunderbrd added them as a fix an bug which was later later fixed inside the FreeListTrashArray.

@Thunderbrd removed a simmilar code segment from CvSelectionGroup a few months ago in svn10538 https://sourceforge.net/p/caveman2c...618195271846340a0a8077:10537&diformat=regular
 
Last edited:
These two code segments should have been removed a long time ago. @Thunderbrd added them as a fix an bug which was later later fixed inside the FreeListTrashArray.

@Thunderbrd removed a simmilar code segment from CvSelectionGroup a few months ago in svn10538 https://sourceforge.net/p/caveman2c...618195271846340a0a8077:10537&diformat=regular
I found two similar entries in CvGameInterface.cpp on line 1011 and 1053
According to notepad++ those are only two places.

There are 12 uses of std::find(some stuff inside)
Spoiler :

Search "std::find(.*)" (12 hits in 8 files)
\Caveman2Cosmos\Sources\CvBuildingList.cpp (2 hits)
Line 203: if (std::find(m_aaiGroupedBuildingList->begin(), m_aaiGroupedBuildingList->end(), m_eSelectedBuilding) != m_aaiGroupedBuildingList->end())
Line 227: if (std::find(m_aaiGroupedBuildingList->begin(), m_aaiGroupedBuildingList->end(), m_eSelectedWonder) != m_aaiGroupedBuildingList->end())

\Caveman2Cosmos\Sources\CvCityAI.cpp (1 hit)
Line 12697: //if (iWorstPlot == -1 || std::find(aWorstPlots.begin(), aWorstPlots.end(), iWorstPlot) != aWorstPlots.end())

\Caveman2Cosmos\Sources\CvGameInterface.cpp (2 hits)
Line 1011: if (movedUnitIds1.empty() || std::find(movedUnitIds1.begin(), movedUnitIds1.end(), pUnitNode1->m_data.iID) == movedUnitIds1.end())
Line 1053: if (movedUnitIds2.empty() || std::find(movedUnitIds2.begin(), movedUnitIds2.end(), pUnitNode2->m_data.iID) == movedUnitIds2.end())

\Caveman2Cosmos\Sources\CvPlayer.cpp (3 hits)
Line 19889: if (GET_TEAM(getTeam()).isHasTech(eTech) || std::find(rootPath.begin(), rootPath.end(), eTech) != rootPath.end())
Line 26966: return (std::find(m_triggersFired.begin(), m_triggersFired.end(), eEventTrigger) != m_triggersFired.end());
Line 27058: std::vector<EventTriggerTypes>::iterator it = std::find(m_triggersFired.begin(), m_triggersFired.end(), eTrigger);

\Caveman2Cosmos\Sources\CvPlayerAI.cpp (1 hit)
Line 29576: // if (std::find(aiPlotGroups.begin(),aiPlotGroups.end(), pPlotGroup->getID()) == aiPlotGroups.end())

\Caveman2Cosmos\Sources\CvSelectionGroup.cpp (1 hit)
Line 6359: if (std::find(aCargoGroups.begin(), aCargoGroups.end(), pGroup) == aCargoGroups.end())

\Caveman2Cosmos\Sources\CvUnitFilters.cpp (1 hit)
Line 102: return std::find(m_eCombats.begin(), m_eCombats.end(), eCombat) != m_eCombats.end();

\Caveman2Cosmos\Sources\CvUnitList.cpp (1 hit)
Line 205: if (std::find(m_aaiGroupedUnitList->begin(), m_aaiGroupedUnitList->end(), m_eSelectedUnit) != m_aaiGroupedUnitList->end())

Probably some of them are fine and others should be changed.
 
Last edited:
The two code segments containing these usages of std::find should have been removed a long time ago. @Thunderbrd added them as a fix an bug which was later later fixed inside the FreeListTrashArray.

@Thunderbrd removed a simmilar code segment from CvSelectionGroup a few months ago in svn10538 https://sourceforge.net/p/caveman2c...618195271846340a0a8077:10537&diformat=regular
As I said, I may have used them as they were the model I found elsewhere of the code functionality I was looking for. Unfortunately, I had no clue there would've been an issue there, and if I removed an instance, it was for a different reason than this.
I found two similar entries in CvGameInterface.cpp on line 1011 and 1053
According to notepad++ those are only two places.

There are 12 uses of std::find(some stuff inside)
Spoiler :

Search "std::find(.*)" (12 hits in 8 files)
\Caveman2Cosmos\Sources\CvBuildingList.cpp (2 hits)
Line 203: if (std::find(m_aaiGroupedBuildingList->begin(), m_aaiGroupedBuildingList->end(), m_eSelectedBuilding) != m_aaiGroupedBuildingList->end())
Line 227: if (std::find(m_aaiGroupedBuildingList->begin(), m_aaiGroupedBuildingList->end(), m_eSelectedWonder) != m_aaiGroupedBuildingList->end())

\Caveman2Cosmos\Sources\CvCityAI.cpp (1 hit)
Line 12697: //if (iWorstPlot == -1 || std::find(aWorstPlots.begin(), aWorstPlots.end(), iWorstPlot) != aWorstPlots.end())

\Caveman2Cosmos\Sources\CvGameInterface.cpp (2 hits)
Line 1011: if (movedUnitIds1.empty() || std::find(movedUnitIds1.begin(), movedUnitIds1.end(), pUnitNode1->m_data.iID) == movedUnitIds1.end())
Line 1053: if (movedUnitIds2.empty() || std::find(movedUnitIds2.begin(), movedUnitIds2.end(), pUnitNode2->m_data.iID) == movedUnitIds2.end())

\Caveman2Cosmos\Sources\CvPlayer.cpp (3 hits)
Line 19889: if (GET_TEAM(getTeam()).isHasTech(eTech) || std::find(rootPath.begin(), rootPath.end(), eTech) != rootPath.end())
Line 26966: return (std::find(m_triggersFired.begin(), m_triggersFired.end(), eEventTrigger) != m_triggersFired.end());
Line 27058: std::vector<EventTriggerTypes>::iterator it = std::find(m_triggersFired.begin(), m_triggersFired.end(), eTrigger);

\Caveman2Cosmos\Sources\CvPlayerAI.cpp (1 hit)
Line 29576: // if (std::find(aiPlotGroups.begin(),aiPlotGroups.end(), pPlotGroup->getID()) == aiPlotGroups.end())

\Caveman2Cosmos\Sources\CvSelectionGroup.cpp (1 hit)
Line 6359: if (std::find(aCargoGroups.begin(), aCargoGroups.end(), pGroup) == aCargoGroups.end())

\Caveman2Cosmos\Sources\CvUnitFilters.cpp (1 hit)
Line 102: return std::find(m_eCombats.begin(), m_eCombats.end(), eCombat) != m_eCombats.end();

\Caveman2Cosmos\Sources\CvUnitList.cpp (1 hit)
Line 205: if (std::find(m_aaiGroupedUnitList->begin(), m_aaiGroupedUnitList->end(), m_eSelectedUnit) != m_aaiGroupedUnitList->end())

Probably some of them are fine and others should be changed.
Definitely worthy of post release review - that said, we need to be careful that it's not, for whatever reason, the best use for where it is employed. @alberts2 or @Anq: can you say when it would be a good thing to utilize or is it just plain a bad idea always?
 
As I said, I may have used them as they were the model I found elsewhere of the code functionality I was looking for. Unfortunately, I had no clue there would've been an issue there, and if I removed an instance, it was for a different reason than this.

The First segment was removed because it slowed the graphics engine alot like the remaining two do with the turn time. You imported this slow code from the MP game at three places. One was removed but i assumed that the other two had been removed in the same commit but they still exist.
 
One was removed but i assumed that the other two had been removed in the same commit but they still exist.
Yeah, it wasn't removed on the merit of its removal but for another reason, whatever that had been at the time so I had no idea the other two should be. Unless you had pointed it out - I vaguely recall you pointing something out, which I procedurally went and did as instructed, not looking much at why and if it might be an indication that it needed done elsewhere.
 
It would be save to use with a std::vector but it should not be used with std::set or std::map.
Is it preferable in any given case?
 
I remember that CvSelectionGroup thing. I was curious so I took a look. I noticed only 1 spot was fix, so I kept these files around so I wouldn't forget. files are old svn rev now, but it should be what you guys are talkin about.
 

Attachments

I remember that CvSelectionGroup thing. I was curious so I took a look. I noticed only 1 spot was fix, so I kept these files around so I wouldn't forget. files are old svn rev now, but it should be what you guys are talkin about.
I'll take a look at a compare later tonight
 
Is it preferable in any given case?
std::vector (and some other containers, e.g. array, deque, list, queue) has no member find function, std::find is the only option here. Basically if a container has its own find function then use it, as it will have the best chance to leverage its knowledge about the internal structure for better performance, otherwise use std::find. However there are faster alternatives to std::find in a vector if it is known to be sorted: binary search.
 
  • Like
Reactions: Anq
Back
Top Bottom