Art Files

alberts8 · Sep 7, 2015

I found another nightmare hidden inside the c2c fpk files.

Code:

07.09.2015 18:47:29 - ------------------------------------------------------------------------------------------
07.09.2015 18:47:29 - START: Content (Byte by byte)
07.09.2015 18:47:29 - Determine the file count of all source folders...
07.09.2015 18:47:29 - File Count: 36389
07.09.2015 18:47:29 - Scan: Z:\New folder (13)
07.09.2015 20:05:21 - Found 14378 duplicates with 299.536.937 Bytes in source folder 'Z:\New folder (13)'
07.09.2015 20:05:21 - Groups: 3877
07.09.2015 20:05:21 - Scanned Archives: 0
07.09.2015 20:05:21 - Duplicates: 14378 (39%) (285,66 MB)
07.09.2015 20:05:21 - Elapsed time: 01:17:52

There are 36389 files inside the fkp's and 14378 of them are duplicates :eek:

. That is just messed up and there must be a way to get rid of them.

Another thing, i finally tested if having fpk files instead of loose files has any bad side effects. It turns out that MAF's happen earlier if the art files are inside fpk files. With loose art files the game goes MAF with 2.963.940kb memory usage but if the art files are inside fpk's it does that with only 2.700.568kb. That's not that much but it makes a difference in more advanced or huge games.

Toffer90 · Sep 7, 2015

alberts2 said:
I found another nightmare hidden inside the c2c fpk files.

Spoiler :

Code:

07.09.2015 18:47:29 - ------------------------------------------------------------------------------------------ 07.09.2015 18:47:29 - START: Content (Byte by byte) 07.09.2015 18:47:29 - Determine the file count of all source folders... 07.09.2015 18:47:29 - File Count: 36389 07.09.2015 18:47:29 - Scan: Z:\New folder (13) 07.09.2015 20:05:21 - Found 14378 duplicates with 299.536.937 Bytes in source folder 'Z:\New folder (13)' 07.09.2015 20:05:21 - Groups: 3877 07.09.2015 20:05:21 - Scanned Archives: 0 07.09.2015 20:05:21 - Duplicates: 14378 (39%) (285,66 MB) 07.09.2015 20:05:21 - Elapsed time: 01:17:52

There are 36389 files inside the fkp's and 14378 of them are duplicates. That is just messed up and there must be a way to get rid of them.

Another thing, i finally tested if having fpk files instead of loose files has any bad side effects. It turns out that MAF's happen earlier if the art files are inside fpk files. With loose art files the game goes MAF with 2.963.940kb memory usage but if the art files are inside fpk's it does that with only 2.700.568kb. That's not that much but it makes a difference in more advanced or huge games.

The main problem lies in getting the nif files to use a texture that is not assumed to be in the same folder as the nif is; if two nifs use the same texture but are in different folders the texture has to be in both folders.

I will investigate if a nif could define a texture file through a file path instead of just a file name.

If the investigation fails the solution would be to have far more nif files in the same folder than we currently have.

Edit: A lot of assets use the same nifs but with different textures; which requires a duplicated nif that only differs in what texture file it uses (a small text string inside the nif is different; so it would not be a recognized as a duplicate file). The only way to solve this one would be if the texture used by a specific nif was defined in the xml that calls for the nif instead of inside the nif. Dunno if this is possible to accomplish in the dll; it's probably decided in the exe.

Sparth · Sep 7, 2015

Toffer90 said:
If the investigation fails the solution would be to have far more nif files in the same folder than we currently have.

It can be done in this way. Other mods do this and its working. We lost some clearness but it can be worth.

Also art files sometimes are named identical but content is not identical. Even if size is the same. For example DDS files (textures or buttons). This same.

Most of possibles cuts was already done in here. Only in Units2.FPK I manually removed ~4500 useless files. Others, like textures, was resized and repacked.

Toffer90 · Sep 7, 2015

Sparth said:
It can be done in this way. Other mods do this and its working. We lost some clearness but it can be worth.

Then I will start making it so whenever I find applicable cases; I've already seen this a couple of times while I've been working with the mod but they can be hard to find manually.

Sparth said:
Also art files sometimes are named identical but content is not identical. Even if size is the same. For example DDS files (textures or buttons). This same.

Yes, DDS has fixed sizes that only depend on image resolution and type of compression. The number of duplications reported by alberts2 are probably accurate though, as he likely went a bit deeper than just checking file sizes. ^^

Sparth said:
Most of possibles cuts was already done in here. Only in Units2.FPK I manually removed ~4500 useless files. Others, like textures, was resized and repacked.

I'm aware of your tremendous effort into this area

goodjob

and was frankly surprised by what alberts2 reported here.
Did you see the edit to my first post, perhaps the nifs with only 1 different string was flagged by alberts2's search?

Sparth · Sep 8, 2015

Toffer90 said:
Did you see the edit to my first post, perhaps the nifs with only 1 different string was flagged by alberts2's search?

If it was possible to add in XML these lines instead in NIF it will be great. We have tremendous amount of identical NIF files which got only different texture

strategyonly · Sep 8, 2015

Are we talking about files like these?? whereas on the (pic1) panther files and getting matched as duplicates in (pic2) Sabre??

These files are needed in BOTH folders even though they are named the same...

Toffer90 · Sep 8, 2015

An example of both cases:

Spoiler :

alberts8 · Sep 9, 2015

Sparth said:
Also art files sometimes are named identical but content is not identical. Even if size is the same. For example DDS files (textures or buttons). This same.

Toffer90 said:
The number of duplications reported by alberts2 are probably accurate though, as he likely went a bit deeper than just checking file sizes. ^^

I'm aware of your tremendous effort into this area goodjob and was frankly surprised by what alberts2 reported here.
Did you see the edit to my first post, perhaps the nifs with only 1 different string was flagged by alberts2's search?

Those 14378 duplicates are 100% identical files not just files with the same size or name. I usesd a byte by byte comparison tool to be sure.

Sparth · Sep 9, 2015

alberts2 said:
Those 14378 duplicates are 100% identical files not just files with the same size or name. I usesd a byte by byte comparison tool to be sure.

They can have identical size but content its not identical. Its nature of DDS files and sometimes NIF files too.

alberts8 · Sep 9, 2015

Sparth said:
They can have identical size but content its not identical. Its nature of DDS files and sometimes NIF files too.

That is the same for every other existing file type. That's the reason why i used a byte by byte comparer to check for duplicates. The number of duplicates is correct.

Sparth · Sep 9, 2015

alberts2 said:
That is the same for every other existing file type. That's the reason why i used a byte by byte comparer to check for duplicates. The number of duplicates is correct.

Here you have example (one of many) of same size (with different content) files
http://imgur.com/XROaPpp

Left: Agent unit with Asian skin
Right: Agent unit with African skin

agent.nif model is the same for both units
agent_diff.dds main texture different for both units
timebomb.dds same texture for dynamit and pistol

If we move all units to one folder we can just leave one timebomb.dds file for all Agents, but NIF and main texture should be different anyway.
Or we move texture location from NIF to XML. Then we win the day and we can remove tons of NIF files.

Yudishtira · Sep 10, 2015

All units with brown horses (for example) have the same skin dds for the horse. Exact same. But they have to because the nif definition expects its skin to be local. Perhaps they could all be converted to 'global' paths, but obviously it would be a huge amount of work for the few with the expertise in Nifskope to do it.

Or am I missing something??? :confused:

ETA: Sorry pretty much the same point Sparth made, except I'd be more confident that he knows what he's talking about...

Sparth · Sep 10, 2015

I only presume that if moving texture path from NIF to XML was possible then already should be done in other mods. But isn't

alberts8 · Nov 20, 2015

Sparth said:
Here you have example (one of many) of same size (with different content) files
http://imgur.com/XROaPpp

Left: Agent unit with Asian skin
Right: Agent unit with African skin

agent.nif model is the same for both units
agent_diff.dds main texture different for both units
timebomb.dds same texture for dynamit and pistol

If we move all units to one folder we can just leave one timebomb.dds file for all Agents, but NIF and main texture should be different anyway.
Or we move texture location from NIF to XML. Then we win the day and we can remove tons of NIF files.

Those duplicates have to go at some point since they increase the sice of the mod and the engine has to load manny identical textures multiple times. I ran it again today this was the result

Code:

20.11.2015 22:11:35 - --------------------------------------------------
20.11.2015 22:11:35 - Search: File size + File content SHA-1 (160-Bit)
20.11.2015 22:11:35 - Determine the file count of all source folders...
20.11.2015 22:11:35 - File Count: 35699
20.11.2015 22:11:35 - Scan: Z:\New folder (7)\art
20.11.2015 22:20:48 - Found 13549 duplicates with 279.086.694 Bytes in source folder 'Z:\New folder (7)\art'
20.11.2015 22:20:48 - Groups: 3.655
20.11.2015 22:20:48 - File Comparison Count: 44.287.061
20.11.2015 22:20:48 - Duplicates: 13549 (37%) (266,16 MB)
20.11.2015 22:20:48 - Elapsed time: 00:09:13

The only working solution would be to put all unit art files into the same folder. In newer games using the gamebyro engine it is possible to use relative paths for textures inside NIF's but we can't.

Toffer90 · Nov 20, 2015

alberts2 said:
In newer games using the gamebyro engine it is possible to use relative paths for textures inside NIF's but we can't.

Oh it's possible with relative texture paths in nifs for bts. I've already done it in a few cases I've found. (I've not been looking for these cases though.) Just take a look at the camel rider (not in FPK).
Some of the cultural versions have a shared/identical texture that has not been duplicated. It worked fine in game.

Before the next FPK packing I could try to find and set up more of these shared texture cases.

Sparth · Nov 20, 2015

Toffer90 said:
Before the next FPK packing I could try to find and set up more of these shared texture cases.

Sharing textures is for sure good idea but IMO its not a big deal beacuse most of the units is using different, unique textures. Yes they are named the same, yes they have the same size, but they are different

If we only have way to put texture paths to XML then we can remove tons of duplicated NIF files...

Edit: After fast calculations I think we can reduce size of the mod by 25-36 Mbs.

Toffer90 · Nov 20, 2015

Sparth said:
Yes they are named the same, yes they have the same size, but they are different

It's my impression that these would not show up as identical in alberts search.

Edit: I stick to the policy TB mentioned. ↓↓ But it would be a lot of work to try and rename the many thousands of files this don't apply to.

Thunderbrd · Nov 20, 2015

Yes they are named the same, yes they have the same size, but they are different

Not a good precedence methinks. Best to ensure unique naming. imo.

alberts8 · Nov 21, 2015

Sparth said:
Sharing textures is for sure good idea but IMO its not a big deal beacuse most of the units is using different, unique textures. Yes they are named the same, yes they have the same size, but they are different
If we only have way to put texture paths to XML then we can remove tons of duplicated NIF files...

Edit: After fast calculations I think we can reduce size of the mod by 25-36 Mbs.

Sadly it's not possible to move texture paths from NIF files to XML. It would only be possible if we had the source code for the exe.

The NIF files are only identical because all the textures have the same file names.

Moving to sharing textures would save ~100MB from ~10000 less DDS files that would be something.

Sparth · Dec 2, 2015

Toffer, can you at these NIF files and test 10-20 of them (just unpack to art/units directory)
https://docs.google.com/uc?authuser=0&id=0B6VuHphw5zHzU1lfbDhRMUZVWFk&export=download

I edited these NIF files and added relative paths to duplicated textures from BTS/C2C. In my computer its works but its a lot of works and I want info from another user before I start editing rest of files.

Art Files

Emperor

C2C Modder

C2C Team Member

C2C Modder

C2C Team Member

C2C Supreme Commander

C2C Modder

Attachments

Emperor

C2C Team Member

Emperor

C2C Team Member

Spiritual/Creative

C2C Team Member

Emperor

C2C Modder

C2C Team Member

C2C Modder

C2C War Dog

Emperor

C2C Team Member

Similar threads