[Religion and Revolution]: Bugs and Todos

[agnat86]

I've done some playtesting, and I've encountered some minor bugs/errors.

1. Coca Leaves has the wrong art when loaded into a transport unit. The art for Cocoa Pods is shown instead. See Screenshot 1, where I hover my cursor over the unit button to show that it is indeed a Coca Leaves unit, not a Cocoa Pods unit.

2. When a nation is defeated, messages appear saying "you have made peace with " (Screenshot 2)

3. When a unit is led by a great general, both the Leadership I and II promotions become available. See Screenshot 3. At least, I assume that this is not intentional, and that both Leadership promotions are not supposed to go on the same unit, at least not at the same time.

 

[Nightinggale]

1. Coca Leaves has the wrong art when loaded into a transport unit. The art for Cocoa Pods is shown instead. See Screenshot 1, where I hover my cursor over the unit button to show that it is indeed a Coca Leaves unit, not a Cocoa Pods unit.

Works for me

2. When a nation is defeated, messages appear saying "you have made peace with " (Screenshot 2)

Fix one bug and you introduce a new one. I was asked by the king to make peace with a player I had already beaten and I got 2 units for accepting. Ray fixed it by automatically making peace with dying players.
Presumably this bug can be fixed by not displaying the message unless both players are alive.

 

[Nightinggale]

The black icons in the screenshot are not taken from the GameFonts.

They are taken from:
...\Assets\Art\interface\symbols\resource_icons_64ea.dds

After much experimentation with this file I managed to get the icons to display. The problem appears when the file is wider than 2048 pixels and it is currently 4096. The problem with the 2048 width is that it leaves room for 32 icons and we currently have 35.

I have attached a version of the file with 32 icons because 32 out of 35 icons is better than 0 out of 35. The 3 missing icons are coca leaves, seal skins and horses.

Also searching the forum I learned that dds files can (and should) be compressed with DXT3. Doing so reduce the filesize from 2.1 MB to 512 kB. The 32 icon version is reduced to 256 kB.

As for a more permanent solution I figure there is no choice other than making a file with 32 icons. The easiest way is to just kick out 3 icons, but that would result in a worse graphical experience on computers, which can handle more than 32 icons.

Instead we could add a new file with the first 32 icons and then pick which file to read based on some setting. Presumably "Low Resolution Textures" would be a candidate as we are already talking low end GPUs, though making our own is an option.

Next if the game is in 32 icon mode, it should give some yields new texture index. For instance fish and food both give food meaning it wouldn't be way off to always show food. There is also likely something to merge regarding fur. All it takes is finding 3 icons, which are close enough to other icons to make them share graphics and it will work.

EDIT: found the easiest solution possible. I reduced filesize to 2240 (64*35). Then I scaled it down to 2048. The icons are now 7.5% smaller than vanilla and they display correctly and aren't black. No need for extra code to determine which file to use or reduced icon set or anything like that.

The reduced resolution is hard to detect ingame, but editing the file is much easier if you know each icon is supposed to be 64x64 pixels meaning the svn version shouldn't be scaled. We can always argue if the released version should be just low resolution or two resolutions where people can move a file if they experience the problem. Either solution is fine by me. It shouldn't be hard to code a file selection based on "Low Resolution Textures" and I will look into it if that is the solution Ray wants. In fact I may look into it anyway as M:C could end up with this addition regardless of what happens in RaR.

 

[Schmiddie]

After much experimentation with this file I managed to get the icons to display. The problem appears when the file is wider than 2048 pixels and it is currently 4096. The problem with the 2048 width is that it leaves room for 32 icons and we currently have 35.

I have attached a version of the file with 32 icons because 32 out of 35 icons is better than 0 out of 35. The 3 missing icons are coca leaves, seal skins and horses.

Also searching the forum I learned that dds files can (and should) be compressed with DXT3. Doing so reduce the filesize from 2.1 MB to 512 kB. The 32 icon version is reduced to 256 kB.

As for a more permanent solution I figure there is no choice other than making a file with 32 icons. The easiest way is to just kick out 3 icons, but that would result in a worse graphical experience on computers, which can handle more than 32 icons.

Instead we could add a new file with the first 32 icons and then pick which file to read based on some setting. Presumably "Low Resolution Textures" would be a candidate as we are already talking low end GPUs, though making our own is an option.

Next if the game is in 32 icon mode, it should give some yields new texture index. For instance fish and food both give food meaning it wouldn't be way off to always show food. There is also likely something to merge regarding fur. All it takes is finding 3 icons, which are close enough to other icons to make them share graphics and it will work.

EDIT: found the easiest solution possible. I reduced filesize to 2240 (64*35). Then I scaled it down to 2048. The icons are now 7.5% smaller than vanilla and they display correctly and aren't black. No need for extra code to determine which file to use or reduced icon set or anything like that.

The reduced resolution is hard to detect ingame, but editing the file is much easier if you know each icon is supposed to be 64x64 pixels meaning the svn version shouldn't be scaled. We can always argue if the released version should be just low resolution or two resolutions where people can move a file if they experience the problem. Either solution is fine by me. It shouldn't be hard to code a file selection based on "Low Resolution Textures" and I will look into it if that is the solution Ray wants. In fact I may look into it anyway as M:C could end up with this addition regardless of what happens in RaR.

I would prefer only to reduce the size in the published version but not in the SVN. Otherwise we will loose quality. I will discuss this with Ray when I'm back. We should be careful here.

 

[Nightinggale]

I would prefer only to reduce the size in the published version but not in the SVN. Otherwise we will loose quality. I will discuss this with Ray when I'm back. We should be careful here.

I completely agree that we should try not to reduce resolution unless it is needed. This is why I propose having two versions of the file in releases. It would appear that the majority of computers do not have the size limit, which is really an argument against lowering the resolution for everybody.

I'm wondering about modifying CvAssetInfoBase::read()

pXML->GetChildXmlValByName(szTextVal, "Path");
setPath(szTextVal);

This can become

if (gDLL->getGraphicOption(GRAPHICOPTION_LOWRES_TEXTURES))
{
    pXML->GetChildXmlValByName(szTextVal, "PathLowRes");
}
if (szTextVal.empty())
{
    pXML->GetChildXmlValByName(szTextVal, "Path");
}
setPath(szTextVal);

That will give XML the power to add a low resolution version of not only this file, but a bunch of files. Also while standard resolution is the default, people with black icons can just switch on low resolution textures, restart and the problem will go away. Presumably GPUs with this problem would be candidates for using low resolution textures anyway.

Note that the standard resolution file will be used regardless of settings unless XML actually adds a low resolution path. This will minimize XML work.

 

[raystuttgart]

1. Coca Leaves has the wrong art when loaded into a transport unit. The art for Cocoa Pods is shown instead. See Screenshot 1, where I hover my cursor over the unit button to show that it is indeed a Coca Leaves unit, not a Cocoa Pods unit.

Works perfectly in the SVN version of the mod.
Are you sure, that this bug is not in your private modmod only ?

2. When a nation is defeated, messages appear saying "you have made peace with " (Screenshot 2)

That is actually not really a bug.

The message is correct.
To prevent some issues, once a nation is defeated it makes peace with all nations before leaving the game.

All the other log messages and log entries about that nation being defeated should also come up.

But I can take a look at that.

3. When a unit is led by a great general, both the Leadership I and II promotions become available. See Screenshot 3. At least, I assume that this is not intentional, and that both Leadership promotions are not supposed to go on the same unit, at least not at the same time.

It is corrected now.

 

[raystuttgart]

After much experimentation with this file I managed to get the icons to display.

Great.

The problem appears when the file is wider than 2048 pixels and it is currently 4096. The problem with the 2048 width is that it leaves room for 32 icons and we currently have 35.

There are actually icons in that file that are not needed.
(And most likely never will.)

1) Coal
2) Medical Herbs
3) (Walruss) Ivory
4) Natural Rubber

So I can simply remove those icons.
(And of course readjust the indexes of the yields.)

Also searching the forum I learned that dds files can (and should) be compressed with DXT3. Doing so reduce the filesize from 2.1 MB to 512 kB. The 32 icon version is reduced to 256 kB.

Compression is often related to loss of quality.
That should be checked properly by Schmiddie when he is back from vacation.
He is our graphics specialist and knows best about such things.

Currently I would not want to compress the file without knowing for sure that it won't result in noteable loss of quality.

It shouldn't be hard to code a file selection based on "Low Resolution Textures" and I will look into it if that is the solution Ray wants.

Sorry, but RaR will not have logic switching between "High Resolution graphic files on strong computers" and "Low Resolution graphic files on weak computers".

I consider that a very bad design decision.
We would need to develop and maintain two sets of graphics, which will never happen.
There are many other bad effects of such a design decision, which we do not need to discuss about though.

 

[agnat86]

Works perfectly in the SVN version of the mod.
Are you sure, that this bug is not in your private modmod only ?

Yes, it was, I somehow had forgotten to copy one of the later changes to coca leaves into the UnitInfos.xml. Sorry for wasting your time.

That is actually not really a bug.

The message is correct.
To prevent some issues, once a nation is defeated it makes peace with all nations before leaving the game.

All the other log messages and log entries about that nation being defeated should also come up.

But I can take a look at that.

Cool!
I realized that it wasn't a bug and that it would be necessary to make peace with a nation after it was defeated. This is merely a layout issue, just about the messages.

One other thing though.
Rangers can behave quite weirdly. If you use them to attack settlements, and they conquer them, then the settlements won't change ownership until another unit enters them (see screenshot). Also, if they are in a Forest where they are invisible, hostile units can enter the same plot without attacking them, leading to the weird situation where you cannot attack an enemy because they are in the same plot as your Ranger is.

This probably has something to do with their invisibility.

 

[Nightinggale]

Compression if often related to loss of quality.
That should be checked properly by Schmiddie when he is back from vacation.
He is our graphics specialist and knows best about such things.

Currently I would not want to compress the file without knowing for sure that it won't result in noteable loss of quality.

Fair enough. I did some reading on this and one of the most noteworthy changes is that the alpha channel is stored in 2 bits instead of 8. Great when you have an alpha channel for icons where it is on/off. Horrible if you make something with ghostly effects.

It does something to the color pixels as well, but I'm uncertain of the effect that has on the image and if it is visible ingame.

DXT3 is designed with GPUs in mind. This mean that the game can copy the image to the GPU without decompressing it first and it reduce the need for video memory by 75%. This will also reduce memory I/O by 75% whenever the image is drawn. As a result the average waiting time for the DLL to read memory is reduced, which will result in a faster game.

This mean compression is not much about filesize itself. Instead it is a tradeoff between performance and graphics. The impact on either is unknown and should ideally be investigated. I wonder the general lack of compression (not just this file, but all files) is the reason why RaR feels a bit sluggish even after I improved performance as much as I did in the DLL.

Sorry, but RaR will not have logic switching between "High Resolution graphic files on strong computers" and "Low Resolution graphic files on weak computers".

I consider that a very bad design decission.
We would need to develop and maintain two sets of graphics, which will never happen.
There are many other bad effects of such a design decision, which we do not need to discuss about though.

I disagree that it is a bad design decision. However removing unused icons and reducing the width of the file to max 2048 pixels is the best solution.

I saw somewhere that there could be problems if the image size isn't sized in 2^n. Presumably the safest is to make it 2048 wide regardless of number of icons. Maybe it's related to DXT3 compressing 4x4 blocks and the real needed size is 4*n. Either way 2048 should be safe.

 

[raystuttgart]

@Nightinggale:

I created a new version of the file:
(It is uploaded to SVN.)

1) I removed the 4 unnecessary icons
(And adjusted XML to that of course.)

2) I reduced size of the file to about half of the original size without loosing quality or icon size
(By simply cutting away the black part from the images. No compression was needed.)

Could you check if the black icon problem is solved ?
(I simply cannot test the issue, because the problem never appeared on my machine.)

The new file works perfectly on my own machine.
(However I don't know if it will do so on other machines as well.)

I saw somewhere that there could be problems if the image size isn't sized in 2^n.

That is correct.

That issue is known very long already. (KJ_Jansson told me about that years ago.)
Can't really explain it though.

resource_icons_64 ea.dds is generally pretty easy to handle though, compared to GameFonts.
(It is fairly simple to add or remove icons once you have good looking icons.)

 

[raystuttgart]

Rangers can behave quite weirdly.
...
This probably has something to do with their invisibility.

Possible.

The "Invisibility" stuff was actually a result of some of my wild prototyping for "Path to the Throne".
(I had really crazy ideas with "Invisibility" features. )

Don't think though, that I can get this solved before Release 2.1 should be published.
Currently have almost no time for modding ...

However, I don't think it is really a major issue.
They simply behave a bit "different".

 

[raystuttgart]

I deactivated the "Peace message" if the Player is dead.
(New revision in SVN.)

Now almost all small issues that came up should be corrected again.

There are still 2 small issues left:

1) Rangers behave a bit strangely when conquering cities
2) Bargaining with Natives (new Feature of RaR) causes OOS in Multiplayer

I don't think I can find the time to take a look (and fix) before Release 2.1 will be published.

Actually both of these issues are probably already in the mod since a very long time and never seem to have caused big problems.
(They should still eventually be fixed of course. They are just not critical.)

 

[Nightinggale]

2) I reduced size of the file to about half of the original size without loosing quality or icon size
(By simply cutting away the black part from the images. No compression was needed.)

Both the compressed and uncompressed format has a fixed number of bits for each pixel. This mean reducing the number of pixels by 50% will reduce the filesize by 50%. Compressing the file will reduce the size by 75% compared to the uncompressed file of the same size.

Could you check if the black icon problem is solved ?
(I simply cannot test the issue, because the problem never appeared on my machine.)

It works. Now comes the difficult task of finding a computer old enough to experience this problem while being new enough to handle RaR at an acceptable speed

2) Bargaining with Natives (new Feature of RaR) causes OOS in Multiplayer

I wasn't aware of this. Do you have a link with some more info?

 

[raystuttgart]

It works.

Ok, great.

I wasn't aware of this. Do you have a link with some more info ?

The issue was mentioned here.
(In my following posts I gave information about the comments to look for in the code.)

There is no further information provided though.
Actually it is probably not even necessary to get more information.

My main problem is that I cannot set up MP games to test.
But if you are able to do that, you might simply provoke the issue by using the Bargaining Feature:

* Start Trading with Natives to sell Goods with a Wagon Train
* Ask for a higher price

If you could take a look and fix that bug, that would be great.

 

[Nightinggale]

2) Bargaining with Natives (new Feature of RaR) causes OOS in Multiplayer)

I found the bug. The answer is handled in CvDiplomacy.py.

if (gc.getPlayer(gc.getGame().getActivePlayer()).[COLOR="Red"][B]tryGetNewBargainPriceBuy[/B][/COLOR]()):
	if (diploScreen.counterPropose() == 1):
		self.setAIComment(self.getCommentID("AI_DIPLOCOMMENT_BARGAIN_BUY_SUCCESS"))
	else:	
		self.setAIComment(self.getCommentID("AI_DIPLOCOMMENT_BARGAIN_BUY_SUCCESS"))
else:
	self.setAIComment(self.getCommentID("AI_DIPLOCOMMENT_BARGAIN_FAIL"))

It calls CvPlayer::tryGetNewBargainPriceBuy(). In this function it calls GC.getGameINLINE().getSorenRandNum(randomBase, "Bargaining Buy");

The problem is how randomness works. A random seed is used to make no more than one random number and then it is discarded. The new random seed is calculated based on the old one.

The point where it goes wrong here is that the random function is called is where the trading player is figuring out what to send on the network. As a result the trading player gets a new random seed while any other player will stay with the old one.

It looks to me like the problem would go away if the choice of bargaining is send to CvPlayer::handleDiploEvent() and this function then calls tryGetNewBargainPriceBuy(). The key difference here is that handleDiploEvent() is called in sync across the network.

Alternatively the bargaining system can use C++ randomness instead of game randomness. That will prevent the random seed from changing. However that might hurt debugging as loading a savegame without randomize random seed will in fact make the outcome of bargaining completely random.


I investigated the DDS compression further. It turns out that the forum lacks proper info on DDS files and compression. Some say only DXT3 works, some say DXT5 works too and some say to use DXT3 because vanilla use it. It's even stated that not using compression will not work, which we know is simply not true.

The compression will not affect quality of some images while it can seriously affect other images negatively. It depends on which colors are used, or rather which colors are used compared to each other.

I wrote a script to get the combine size of all uncompressed DDS files and it turns out that compressing all of them will reduce the size by 348 MB. It also turned out that 1013 out of 3751 are uncompressed.

Conclusion: no clear conclusion at all. Compressing everything will reduce memory I/O and you likely will be able to tell a difference in overall performance. At the same time it might affect images negatively and if compressed, more than 1000 images should be checked where before and after images are compared to tell if the compression produce an acceptable result. This mean compressing the images requires somebody with a whole lot of time to devote to this task, which appears to be the biggest issue.

 

[raystuttgart]

@Nightinggale:

1) Barganining OOS in MP:

Great that you found the cause.
It would be great if you could get this fixed.

If possible I would like to stay with "game randomness" for Bargaining.
But if that is the best / easiest solution, I can live with "C++ randomness" for that feature as well.

2) Compression of DDS:

There is no need to overexagerate here.

At the moment we don't even know if compressing all those DDS would really significantly improve performance.
DDS have never shown to be really problematic.

The problematic parts considering graphics have always been Textures (NIF) and Animations (KFM).
And that we have already improved a lot.

Yes, RaR is huge and contains an incredible amount of art which might affect performance of course.
We knew about that and we still wanted to have all these great graphics.

But generally RaR runs really performant considering its size.
On modern machines it runs really fast.

Loading might be a bit long on some machines.
But turn times seem to be completely ok.

Additionally there are 3 major issues that would come with "compression":

1) Potential loss of quality of graphics
2) Potential introduction of bugs
3) Lots and lots of efforts

From my side, the topic "Compression of DDS" is pretty much dead.
Unless Schmiddie - our graphical specialist - will say that he believes this is a good idea and wants to take that task.

 

[Schmiddie]

@Nightinggale:

2) Compression of DDS:

There is no need to overexagerate here.

At the moment we don't even know if compressing all those DDS would really significantly improve performance.
DDS have never shown to be really problematic.

The problematic parts considering graphics have always been Textures (NIF) and Animations (KFM).
And that we have already improved a lot.

Yes, RaR is huge and contains an incredible amount of art which might affect performance of course.
We knew about that and we still wanted to have all these great graphics.

But generally RaR runs really performant considering its size.
On modern machines it runs really fast.

Loading might be a bit long on some machines.
But turn times seem to be completely ok.

Additionally there are 3 major issues that would come with "compression":

1) Potential loss of quality of graphics
2) Potential introduction of bugs
3) Lots and lots of efforts

From my side, the topic "Compression of DDS" is pretty much dead.
Unless Schmiddie - our graphical specialist - will say that he believes this is a good idea and wants to take that task.

In any case I say "NO" with regard to the compression of dds files at this point of time and for all graphics by a script.

Ray has listed the major issues, especially the loss of quality. RaR is still under heavy development and improvement. Every time you compress a DDS file you will loose quality (even if you will not see this directly ingame). If you need to change such graphic again and you compress it once more, loss of quality continues again and again...

In the past this already lead to a unacceptable loss of quality of some of the button atlases and I had to create them once more (a lot of work / waste of time). So from my side I will only compress a graphic if I'm sure that this never will have to be changed.

I know that there are already some graphics which could be already compressed now, but I had not the time to do that so far. Since this currently has no high priority for me, I will do it by next opportunity. So please, do not compress any DDS files!

 

[Nightinggale]

If possible I would like to stay with "game randomness" for Bargaining.
But if that is the best / easiest solution, I can live with "C++ randomness" for that feature as well.

It doesn't look like it is possible to goto handleDiploEvent() and then set up possible new answers based on the outcome of that. That mean we can't use game random.

However I noticed something interesting about std::rand(), which could be quite useful here. It appears to be doing the very same thing as gam random, which is to calculate a new random seed based on the old one whenever a random seed is requested. We can set the random seed with std::srand().

If we call std::srand(m_ulRandomSeed) whenever m_ulRandomSeed is changed, then std::rand() will provide randomness relying on game random but at the same time it will not change the game randomseed and it will be safe to use when bargaining on one computer only.

In any case I say "NO" with regard to the compression of dds files at this point of time and for all graphics by a script.

I don't think anybody wants to do that with a script. I used a script to calculate the combined size of uncompressed DDS files. I use script to read from game files without second thought, but scripts, which writes to files are potentially very dangerous. As scripts can't judge the quality of graphical changes, I don't think it would ever make sense to use a script for changing graphics.

The more I think about DXT compression, the more I hate it. I would prefer if the game had used png instead as they have great compression for our kind of images and they are lossless. However unlike DXT, pngs can't be decompressed by GPUs.

 

[raystuttgart]

@Nightinggale:

If you can fix the OOS in MP for Bargaining, that would be great.
Simply choose the solution that you consider best.

If you can't fix it, world will not end either.
We could tell people then to not use that feature in MP.

@Schmiddie:

Let's forget about the compression of DDS for now.
It simply sounds like too much effort and risk (loss of quality, bugs).

I really don't believe it is critcal anyways.

By the way:
I will be gone again until next weekend.

 

[Nightinggale]

I committed a fix for the bargaining issue. I had a bit of a testing issue meaning it's only tested in singleplayer. As far as I'm aware, there aren't any network sync issues left.