Working on a mod is different than a full game, there are some quite unreasonable restrictions we simply have to adapt to rather than try to break.
Thunderbrd
C2C War Dog
Well this is a hell of a discussion to walk into at lunch time 
So from what I can tell from trying to speed read through it all, it looks like y'all are saying you cannot store a folder: mods/Caveman2Cosmos/Unloaded/Art... and then keep the game from indexing all the loose art files in that folder. Am I right that this is the challenge we're up against?
That would be interesting news to begin with since I've long kept all art files unpacked in the Caveman2Cosmos folder in folders named for their own FPKs. That would mean I've been hella limiting my own memory and causing far longer load times than normal. Obviously, the file paths to access those art files have been in the FPKs all along, but to understand that these files have been double-loaded into the mod all this time is ... wow.
Before I try to sort through all the discussions and suggestions on how to address this, I just want to clarify that this IS the issue you're saying we have a problem with?
So from what I can tell from trying to speed read through it all, it looks like y'all are saying you cannot store a folder: mods/Caveman2Cosmos/Unloaded/Art... and then keep the game from indexing all the loose art files in that folder. Am I right that this is the challenge we're up against?
That would be interesting news to begin with since I've long kept all art files unpacked in the Caveman2Cosmos folder in folders named for their own FPKs. That would mean I've been hella limiting my own memory and causing far longer load times than normal. Obviously, the file paths to access those art files have been in the FPKs all along, but to understand that these files have been double-loaded into the mod all this time is ... wow.
Before I try to sort through all the discussions and suggestions on how to address this, I just want to clarify that this IS the issue you're saying we have a problem with?
So this would explain why you waited minutes for mod to load, when I had it loaded within 20 seconds on subsequent launches.
Also this was why you and whisperr had crashes mid Atomic era on now Gigantic map (I scaled down generated map sizes after this discussion).
I think I even asked if you had any loose art within Caveman2Cosmos folder.
And yeah any loose art in Caveman2Cosmos folder is loaded - no matter where it is.
Unpacked art would have to be outside Caveman2Cosmos altogether.
Or live repository would have to be another folder.
So if someone wanted to work on art, then someone should copy only select files to gameplay mod.
Last edited:
That's essentially what's being discussed, yup.
billw2015
King
- Joined
- Jun 22, 2015
- Messages
- 837
Anything is possible if you are willing to put the effort in. We can subvert the entire loading flow if we want to, there are techniques to redirect all file operations for a process, for instance. That is overkill in this case though, however when it comes to more serious problems like memory it might be time to look at this sort of thing (as a last resort).
Yep pretty much! You *can* stop it from being looked at by making the folder hidden, but that has its own problems, and has to be done locally (SVN doesn't have a hidden flag you can commit). I'm working up some options now.
You are in for a pleasant surprise then! This to me is pretty good motivation to move forward with solving this problem though.
billw2015
King
- Joined
- Jun 22, 2015
- Messages
- 837
Okay so here is the possible solutions I tried out and what they imply:
Have a hidden unpacked folder:
- It needs to be set locally, because SVN can't store hidden attributes on things. So a script or right click the folder and hide it basically.
- SVN misses changes in there maybe? I just tested and it seems to see the hidden folder and contents okay actually.
- Won't appear unless your explorer is set to show hidden folders. Confusing for people! Again manually need to go and show hidden folders in explorer.
- Weird. Just weird that source content folder devs are meant to use is hidden...
- Error prone in the same manner as TB experienced above: if you forget to hide it your load times will die!
Restructure mod folders:
- Everyone needs to update their mod directory in ini file or command line.
- In game menu doesn't recognize it correctly. It still sees Caveman2Cosmos, but tries to load it from the root folder instead of the subfolder, which obviously doesn't work.
- Breaks save games! This is insurmountable unless coupled with other save breaking changes. HOWEVER it does actually enforce that that saves are broken with a dialog instead of errors, so in this respect it is actually useful.
- Update all the references to assets in the main game. All references to main game files are broken and need an extra ..\ added. Pretty simple search replace, dunno how hard to confirm all references are fixed though.
Directory junctions:
This is the solution I just came up with now (used it before for the same purpose in another game). It means that SVN folder HAS to be outside of the mods folder. But instead of doing an export/copy we just make directory junctions (a type of link) from the folders we want the game to load in the SVN directory straight into the Mods directory. We can also copy config files and other minor things that we don't want to change.
- It needs to be setup once initially, I already made a script that puts up a directory browser and makes the links. It takes < 1 second to run. This only has to be done ONCE, not every update.
- It means people *can't* put SVN into mods directory at all, it HAS to be somewhere else, however it is functionally identical, so I don't see this as a problem, other than you have to initially move it somewhere else.
Have a hidden unpacked folder:
- It needs to be set locally, because SVN can't store hidden attributes on things. So a script or right click the folder and hide it basically.
- SVN misses changes in there maybe? I just tested and it seems to see the hidden folder and contents okay actually.
- Won't appear unless your explorer is set to show hidden folders. Confusing for people! Again manually need to go and show hidden folders in explorer.
- Weird. Just weird that source content folder devs are meant to use is hidden...
- Error prone in the same manner as TB experienced above: if you forget to hide it your load times will die!
Restructure mod folders:
- Everyone needs to update their mod directory in ini file or command line.
- In game menu doesn't recognize it correctly. It still sees Caveman2Cosmos, but tries to load it from the root folder instead of the subfolder, which obviously doesn't work.
- Breaks save games! This is insurmountable unless coupled with other save breaking changes. HOWEVER it does actually enforce that that saves are broken with a dialog instead of errors, so in this respect it is actually useful.
- Update all the references to assets in the main game. All references to main game files are broken and need an extra ..\ added. Pretty simple search replace, dunno how hard to confirm all references are fixed though.
Directory junctions:
This is the solution I just came up with now (used it before for the same purpose in another game). It means that SVN folder HAS to be outside of the mods folder. But instead of doing an export/copy we just make directory junctions (a type of link) from the folders we want the game to load in the SVN directory straight into the Mods directory. We can also copy config files and other minor things that we don't want to change.
- It needs to be setup once initially, I already made a script that puts up a directory browser and makes the links. It takes < 1 second to run. This only has to be done ONCE, not every update.
- It means people *can't* put SVN into mods directory at all, it HAS to be somewhere else, however it is functionally identical, so I don't see this as a problem, other than you have to initially move it somewhere else.
Both unaceptable in my opinion.
Junction seems ok, but I much prefer the fourth option.
Similar to "junction" as one must have the SVN folder elsewhere than directly inside the working copy that you play from.
Different to "junction" as you need to do an export before playing on a different revision.
Less prone to mess up regarding custom modifications or in regards to having multiple C2C SVN folders setup at the same time.
Will require a custom export/install batch script that do the FPK packing and potential dll compiling (checkbox? default off?) before exporting the correct stuff to the correct spot.
Last edited:
billw2015
King
- Joined
- Jun 22, 2015
- Messages
- 837
What about doing local modifications, you need to export then as well right? Except for dll changes of course we can output direct to the mod folder, which helps with the issue TB mentions about accidentally checking in the wrong version. But changing an xml file and then have to do export again? That seems painful to me.
Custom modifications yes, but how many people does this apply to? I think SVN ignore can fix this anyway right?
I don't see how two SVN folders would be a problem at all in this case specifically. In fact it is better, you can instantly switch between them by just running the install script to recreate the junctions for whichever version you want to use.
Yeah I am definitely happy to do this.
Not sure what you mean, a local modification would never need to be in the SVN folder....
billw2015
King
- Joined
- Jun 22, 2015
- Messages
- 837
Local modification as in, you want to make changes and then later submit them. So first they are local modifications. Like any normal changes we make that we want to put into SVN.
I would make those changes directly in the folder I play from and then copy the files (or even lines using winmerge if there are changes within a file I don't want to commit) that I want to commit into the SVN when I'm ready to commit my changes.
No need to update SVN or export anything while I'm working on something.
Thing is that I'm often working on stuff that I have no intention of committing at the same time as I'm working on something I actually will commit.
I think there's quite a few SVN users who have some custom modifications and modmods are used quite frequently. I don't know about the practicality of using SVN ignore.
Ok, that's not a problem then.
I just don't like the idea of not separating the files that I'm currently playing around with from the repository files.
Last edited:
billw2015
King
- Joined
- Jun 22, 2015
- Messages
- 837
Hmm I would never want to work like that, it is way too much manual copying stuff around for me!
However it occurs to me that this isn't an either/or situation AT ALL.
We can have both a junction based install AND a copy based install methods and people can use whichever one they want. They both just require that SVN isn't in the mods directory.
Just need some feedback from other people if they would be happy to move to a system like this.
I was thinking along the same line right now.
Thunderbrd
C2C War Dog
I'll have to look into this deeper. One reason that I've ALWAYS played with the svn repository in the mods folder directly is because if I change the name of the mod folder, copy the mod folder, or move the mod folder, the debug dll won't properly attach to the process for some reason. I must be playing in the same mod folder that I built the debug dll in it seems. This MAY not be an issue with the way we are proposing to do things here.
Regardless, none of those solutions sound appealing for various reasons.
One thing I wanted to ask earlier... Is there a way to get the mod to only load art assets in a particular folder or isolate (without hiding) a folder from loading? I realize that's probably one of the first things we've looked at. Just seems there should be some kiind of way to shield the folder from loading what's in it. That said, if there was, I figure we wouldn't be considering any of this.
What about the option of having an unloaded art folder setup in My Games? The unpacker could automatically set that up and we could add/manipulate art files there and the FPK packer and updater could refer to that location? Can we always assume that would be the same system path? The savegames always go there don't they?
Another thought: if we had a hidden folder for this, transfers to and from SVN or Git can ALWAYS be the FPKs only and it could be the unpacker and packer and updater that is expected to be used (and only if you want to manipulate art files) and it would unpack to the hidden unpacked folder and pack and update from it. I see no reason why that wouldn't work actually. You'd never commit that so being hidden is actually perfect. The commits and updates would only work with the FPK compiled art. THIS sounds to be the most appealing to me, assuming my previous questions are barking up the wrong tree.
Regardless, none of those solutions sound appealing for various reasons.
One thing I wanted to ask earlier... Is there a way to get the mod to only load art assets in a particular folder or isolate (without hiding) a folder from loading? I realize that's probably one of the first things we've looked at. Just seems there should be some kiind of way to shield the folder from loading what's in it. That said, if there was, I figure we wouldn't be considering any of this.
What about the option of having an unloaded art folder setup in My Games? The unpacker could automatically set that up and we could add/manipulate art files there and the FPK packer and updater could refer to that location? Can we always assume that would be the same system path? The savegames always go there don't they?
Another thought: if we had a hidden folder for this, transfers to and from SVN or Git can ALWAYS be the FPKs only and it could be the unpacker and packer and updater that is expected to be used (and only if you want to manipulate art files) and it would unpack to the hidden unpacked folder and pack and update from it. I see no reason why that wouldn't work actually. You'd never commit that so being hidden is actually perfect. The commits and updates would only work with the FPK compiled art. THIS sounds to be the most appealing to me, assuming my previous questions are barking up the wrong tree.
The idea is to not have FPK's but only loose art files on the SVN repository, so to have the loose art in "My games" would require the SVN folder to be there too, unless you want to move stuff out of the repository with a script and then ignore the insane list of deleted files whenever you are about to make a commit.
It is possible to reliably identify the path to the civIV folder within "Documents\My Games" btw.
This is handled by the exe, but if anyone could find a way to manipulate the exe it may very well be billw.
That is plan B I guess, to just drop any attempt at having the loose art on the repository.
Last edited:
Thunderbrd
C2C War Dog
Leaning more towards this in my thinking at the moment but I want to hear what others say. I have no problem with having numerous autopacked smaller FPKs as THE way we store and transmit all art files (outside the loose ones in the art folder that would still be allowed but usually like... why?)
Particularly when we have such an easy and quick unpacking batch tool! (which would be really nice if it could also be instructed to ensure the folder 'unpacked' is hidden.)
Last edited:
JosEPh_II
TBS WarLord
And this is how I always have my SVN folder. I Never Ever use it to play from. Or work from either. Kept in a completely different directory from where Game directory is. And from where my working copy is. Reduces for me the chance of committing a file that may have a mistake. Because the working file is where the adjustments are made. Then a copy is put into the Mod for play testing. If it all tests out okay, then I put the file into the SVN and Commit.
Last edited:
All though I have not modded for a long time. I use a similar system to JosEPh.
Thunderbrd
C2C War Dog
I'd like to see a system that supports either method is basically what I'm looking to achieve here.
