FPK live (tm)

Well this is a hell of a discussion to walk into at lunch time ;)

So from what I can tell from trying to speed read through it all, it looks like y'all are saying you cannot store a folder: mods/Caveman2Cosmos/Unloaded/Art... and then keep the game from indexing all the loose art files in that folder. Am I right that this is the challenge we're up against?

That would be interesting news to begin with since I've long kept all art files unpacked in the Caveman2Cosmos folder in folders named for their own FPKs. That would mean I've been hella limiting my own memory and causing far longer load times than normal. Obviously, the file paths to access those art files have been in the FPKs all along, but to understand that these files have been double-loaded into the mod all this time is ... wow.

Before I try to sort through all the discussions and suggestions on how to address this, I just want to clarify that this IS the issue you're saying we have a problem with?
 
That would be interesting news to begin with since I've long kept all art files unpacked in the Caveman2Cosmos folder in folders named for their own FPKs. That would mean I've been hella limiting my own memory and causing far longer load times than normal. Obviously, the file paths to access those art files have been in the FPKs all along, but to understand that these files have been double-loaded into the mod all this time is ... wow.
So this would explain why you waited minutes for mod to load, when I had it loaded within 20 seconds on subsequent launches.
Also this was why you and whisperr had crashes mid Atomic era on now Gigantic map (I scaled down generated map sizes after this discussion).
I think I even asked if you had any loose art within Caveman2Cosmos folder.

And yeah any loose art in Caveman2Cosmos folder is loaded - no matter where it is.
Unpacked art would have to be outside Caveman2Cosmos altogether.

Or live repository would have to be another folder.
So if someone wanted to work on art, then someone should copy only select files to gameplay mod.
 
Last edited:
So from what I can tell from trying to speed read through it all, it looks like y'all are saying you cannot store a folder: mods/Caveman2Cosmos/Unloaded/Art... and then keep the game from indexing all the loose art files in that folder. Am I right that this is the challenge we're up against?

That would be interesting news to begin with since I've long kept all art files unpacked in the Caveman2Cosmos folder in folders named for their own FPKs. That would mean I've been hella limiting my own memory and causing far longer load times than normal. Obviously, the file paths to access those art files have been in the FPKs all along, but to understand that these files have been double-loaded into the mod all this time is ... wow.

Before I try to sort through all the discussions and suggestions on how to address this, I just want to clarify that this IS the issue you're saying we have a problem with?
That's essentially what's being discussed, yup.
 
Working on a mod is different than a full game, there are some quite unreasonable restrictions we simply have to adapt to rather than try to break.
Anything is possible if you are willing to put the effort in. We can subvert the entire loading flow if we want to, there are techniques to redirect all file operations for a process, for instance. That is overkill in this case though, however when it comes to more serious problems like memory it might be time to look at this sort of thing (as a last resort).

Am I right that this is the challenge we're up against?
Yep pretty much! You *can* stop it from being looked at by making the folder hidden, but that has its own problems, and has to be done locally (SVN doesn't have a hidden flag you can commit). I'm working up some options now.
That would be interesting news to begin with since I've long kept all art files unpacked in the Caveman2Cosmos folder in folders named for their own FPKs.
You are in for a pleasant surprise then! This to me is pretty good motivation to move forward with solving this problem though.
 
Okay so here is the possible solutions I tried out and what they imply:

Have a hidden unpacked folder:
- It needs to be set locally, because SVN can't store hidden attributes on things. So a script or right click the folder and hide it basically.
- SVN misses changes in there maybe? I just tested and it seems to see the hidden folder and contents okay actually.
- Won't appear unless your explorer is set to show hidden folders. Confusing for people! Again manually need to go and show hidden folders in explorer.
- Weird. Just weird that source content folder devs are meant to use is hidden...
- Error prone in the same manner as TB experienced above: if you forget to hide it your load times will die!

Restructure mod folders:
- Everyone needs to update their mod directory in ini file or command line.
- In game menu doesn't recognize it correctly. It still sees Caveman2Cosmos, but tries to load it from the root folder instead of the subfolder, which obviously doesn't work.
- Breaks save games! This is insurmountable unless coupled with other save breaking changes. HOWEVER it does actually enforce that that saves are broken with a dialog instead of errors, so in this respect it is actually useful.
- Update all the references to assets in the main game. All references to main game files are broken and need an extra ..\ added. Pretty simple search replace, dunno how hard to confirm all references are fixed though.

Directory junctions:
This is the solution I just came up with now (used it before for the same purpose in another game). It means that SVN folder HAS to be outside of the mods folder. But instead of doing an export/copy we just make directory junctions (a type of link) from the folders we want the game to load in the SVN directory straight into the Mods directory. We can also copy config files and other minor things that we don't want to change.
- It needs to be setup once initially, I already made a script that puts up a directory browser and makes the links. It takes < 1 second to run. This only has to be done ONCE, not every update.
- It means people *can't* put SVN into mods directory at all, it HAS to be somewhere else, however it is functionally identical, so I don't see this as a problem, other than you have to initially move it somewhere else.
upload_2019-8-1_22-49-55.png
 
Have a hidden unpacked folder:
- It needs to be set locally, because SVN can't store hidden attributes on things. So a script or right click the folder and hide it basically.
- SVN misses changes in there maybe? I just tested and it seems to see the hidden folder and contents okay actually.
- Won't appear unless your explorer is set to show hidden folders. Confusing for people! Again manually need to go and show hidden folders in explorer.
- Weird. Just weird that source content folder devs are meant to use is hidden...
- Error prone in the same manner as TB experienced above: if you forget to hide it your load times will die!

Restructure mod folders:
- In game menu doesn't recognize it correctly. It still sees Caveman2Cosmos, but tries to load it from the root folder instead of the subfolder, which obviously doesn't work.
Both unaceptable in my opinion.

Junction seems ok, but I much prefer the fourth option.
Our own export script would be nice, one that pretty much does it all to set up the mod. All it has to do is ask the user what directory to export to, and then work its magic. ^^
We could have this folder structure on the SVN with such an install script:
"Anywhere\SVN folder named whatever\"
--- "Caveman2Cosmos\"
--- --- All that we currently have.
--- "Install C2C.bat"
Similar to "junction" as one must have the SVN folder elsewhere than directly inside the working copy that you play from.
Different to "junction" as you need to do an export before playing on a different revision.
Less prone to mess up regarding custom modifications or in regards to having multiple C2C SVN folders setup at the same time.
Will require a custom export/install batch script that do the FPK packing and potential dll compiling (checkbox? default off?) before exporting the correct stuff to the correct spot.
 
Last edited:
Different to "junction" as you need to do an export before playing on a different revision.
What about doing local modifications, you need to export then as well right? Except for dll changes of course we can output direct to the mod folder, which helps with the issue TB mentions about accidentally checking in the wrong version. But changing an xml file and then have to do export again? That seems painful to me.

Less prone to mess up regarding custom modifications or in regards to having two different C2C SVN folders at the same time.
Custom modifications yes, but how many people does this apply to? I think SVN ignore can fix this anyway right?
I don't see how two SVN folders would be a problem at all in this case specifically. In fact it is better, you can instantly switch between them by just running the install script to recreate the junctions for whichever version you want to use.

Will require a custom export/install batch script that do the FPK packing and potential dll compiling (checkbox? default off?) before exporting the correct stuff to the correct spot.
Yeah I am definitely happy to do this.
 
Not sure what you mean, a local modification would never need to be in the SVN folder....
Local modification as in, you want to make changes and then later submit them. So first they are local modifications. Like any normal changes we make that we want to put into SVN.
 
Local modification as in, you want to make changes and then later submit them. So first they are local modifications. Like any normal changes we make that we want to put into SVN.
I would make those changes directly in the folder I play from and then copy the files (or even lines using winmerge if there are changes within a file I don't want to commit) that I want to commit into the SVN when I'm ready to commit my changes.
No need to update SVN or export anything while I'm working on something.
Thing is that I'm often working on stuff that I have no intention of committing at the same time as I'm working on something I actually will commit.
Custom modifications yes, but how many people does this apply to? I think SVN ignore can fix this anyway right?
I think there's quite a few SVN users who have some custom modifications and modmods are used quite frequently. I don't know about the practicality of using SVN ignore.
I don't see how two SVN folders would be a problem at all in this case specifically. In fact it is better, you can instantly switch between them by just running the install script to recreate the junctions for whichever version you want to use.
Ok, that's not a problem then.
I just don't like the idea of not separating the files that I'm currently playing around with from the repository files.
 
Last edited:
I would make those changes directly in the folder I play from and then copy the files I want to commit into the SVN when I'm ready to commit my changes. No need to update SVN or export anything while I'm working on something.
Thing is that I'm often working on stuff that I have no intention of committing at the same time as I'm working on something I actually will commit.
Hmm I would never want to work like that, it is way too much manual copying stuff around for me!
However it occurs to me that this isn't an either/or situation AT ALL.
We can have both a junction based install AND a copy based install methods and people can use whichever one they want. They both just require that SVN isn't in the mods directory.
Just need some feedback from other people if they would be happy to move to a system like this.
 
However it occurs to me that this isn't an either/or situation AT ALL.
We can have both a junction based install AND a copy based install methods and people can use whichever one they want. They both just require that SVN isn't in the mods directory.
Just need some feedback from other people if they would be happy to move to a system like this.
I was thinking along the same line right now.
 
Here is my junction directory install script for anyone interested, drop it in the Tools directory in an SVN outside of Mods dir and run it. It will delete any existing copy of the Mod in the Mods dir and make the junctions.
 

Attachments

I'll have to look into this deeper. One reason that I've ALWAYS played with the svn repository in the mods folder directly is because if I change the name of the mod folder, copy the mod folder, or move the mod folder, the debug dll won't properly attach to the process for some reason. I must be playing in the same mod folder that I built the debug dll in it seems. This MAY not be an issue with the way we are proposing to do things here.

Regardless, none of those solutions sound appealing for various reasons.

One thing I wanted to ask earlier... Is there a way to get the mod to only load art assets in a particular folder or isolate (without hiding) a folder from loading? I realize that's probably one of the first things we've looked at. Just seems there should be some kiind of way to shield the folder from loading what's in it. That said, if there was, I figure we wouldn't be considering any of this.

What about the option of having an unloaded art folder setup in My Games? The unpacker could automatically set that up and we could add/manipulate art files there and the FPK packer and updater could refer to that location? Can we always assume that would be the same system path? The savegames always go there don't they?


Another thought: if we had a hidden folder for this, transfers to and from SVN or Git can ALWAYS be the FPKs only and it could be the unpacker and packer and updater that is expected to be used (and only if you want to manipulate art files) and it would unpack to the hidden unpacked folder and pack and update from it. I see no reason why that wouldn't work actually. You'd never commit that so being hidden is actually perfect. The commits and updates would only work with the FPK compiled art. THIS sounds to be the most appealing to me, assuming my previous questions are barking up the wrong tree.
 
What about the option of having an unloaded art folder setup in My Games? The unpacker could automatically set that up and we could add/manipulate art files there and the FPK packer and updater could refer to that location? Can we always assume that would be the same system path? The savegames always go there don't they?
The idea is to not have FPK's but only loose art files on the SVN repository, so to have the loose art in "My games" would require the SVN folder to be there too, unless you want to move stuff out of the repository with a script and then ignore the insane list of deleted files whenever you are about to make a commit.
It is possible to reliably identify the path to the civIV folder within "Documents\My Games" btw.
One thing I wanted to ask earlier... Is there a way to get the mod to only load art assets in a particular folder or isolate (without hiding) a folder from loading?
This is handled by the exe, but if anyone could find a way to manipulate the exe it may very well be billw.
Another thought: if we had a hidden folder for this, transfers to and from SVN or Git can ALWAYS be the FPKs only and it could be the unpacker and packer and updater that is expected to be used (and only if you want to manipulate art files) and it would unpack to the hidden unpacked folder and pack and update from it. I see no reason why that wouldn't work actually. You'd never commit that so being hidden is actually perfect. The commits and updates would only work with the FPK compiled art.
That is plan B I guess, to just drop any attempt at having the loose art on the repository.
 
Last edited:
That is plan B I guess, to just drop any attempt at having the loose art on the repository.
Leaning more towards this in my thinking at the moment but I want to hear what others say. I have no problem with having numerous autopacked smaller FPKs as THE way we store and transmit all art files (outside the loose ones in the art folder that would still be allowed but usually like... why?)

Particularly when we have such an easy and quick unpacking batch tool! (which would be really nice if it could also be instructed to ensure the folder 'unpacked' is hidden.)
 
Last edited:
It means people *can't* put SVN into mods directory at all, it HAS to be somewhere else,
They both just require that SVN isn't in the mods directory.
Just need some feedback from other people if they would be happy to move to a system like this.

And this is how I always have my SVN folder. I Never Ever use it to play from. Or work from either. Kept in a completely different directory from where Game directory is. And from where my working copy is. Reduces for me the chance of committing a file that may have a mistake. Because the working file is where the adjustments are made. Then a copy is put into the Mod for play testing. If it all tests out okay, then I put the file into the SVN and Commit.
 
Last edited:
And this is how I always have my SVN folder. I Never Ever use it to play from. Or work from either. Kept in a completely different directory from where Game directory is. And from where my working copy is. Reduces for me the chance of committing a file that may have a mistake. Because the working file is where the adjustments are made. Then a copy is put into the Mod for play testing. If it all tests out okay, then I put the file into the SVN and Commit.

All though I have not modded for a long time. I use a similar system to JosEPh.
 
And this is how I always have my SVN folder. I Never Ever use it to play from. Or work from either. Kept in a completely different directory from where Game directory is. And from where my working copy is. Reduces for me the chance of committing a file that may have a mistake. Because the working file is where the adjustments are made. Then a copy is put into the Mod for play testing. If it all tests out okay, then I put the file into the SVN and Commit.

All though I have not modded for a long time. I use a similar system to JosEPh.
I'd like to see a system that supports either method is basically what I'm looking to achieve here.
 
Back
Top Bottom