FPK live (tm)

Did you only go into the menu only or start an actual game? I tried this and the game crashes.
Go in to game AND start a new game. What exactly did you try? I did:
  1. move fpks to c:\temp
  2. break point in DDL_PROCESS_ATTACH
  3. run mod with VS attached
  4. hit breakpoint
  5. put fpks back
  6. continue from breakpoint
  7. everything worked
 
What if someone else changes the same file you have changed without you noticing?
Then you just copy the file back into the svn after updating you might overwrite those changes!

This actually happens.
I do things similarly to JosEPh in that I have my copy of the SVN folder different to my modding and playing folder. I have a Sync program that comes with a compare files component that I use to see when there are changes between the two. I Put my work that I want to commit into my SVN copy then do a final update from the repository which used to tell me if there were any changes that affected the files I was changing. If there were I would inspect those changes and do some testing if they were in conflict. No problems then I commit. Currently I have an incorrect set up which I will fix when I get back.

i.e.

Because before I commit I Update the SVN 1st! Then look at what has been changed. So that has not happened for me. Attention to details and not a rush job, which can happen with the other method.
ditto
 
Go in to game AND start a new game. What exactly did you try? I did:
  1. move fpks to c:\temp
  2. break point in DDL_PROCESS_ATTACH
  3. run mod with VS attached
  4. hit breakpoint
  5. put fpks back
  6. continue from breakpoint
  7. everything worked

Doesn't work for me for some reason i get crashes in the bts.exe. I try it again later.
 
What if someone else changes the same file you have changed without you noticing?
Then you just copy the file back into the svn after updating you might overwrite those changes!
It may have happened before, most SVN user with write access mess up something in their first weeks of modding experience, like forgetting to update before committing so that it may be hours or days since they updated when they commit. These kinds of errors can happen regardless of how the modder organize their files.
I never update the SVN folder before adding my changes to the folder, my changes are based on one revision, and that is the revision that must be present in the SVN folder I put them in.
Right before I'm ready to commit my changes I do an update and look for any red conflict merges, I would even inspect green successful merges as even successful merges may be two changes that are incompatible with each other.
After inspecting merges I usually do another update just to be sure as the inspection could have taken more than 5 minutes and, hey, you never know what could happen in 5+ minutes.

Why would anyone working like this update their SVN folder while working on changes in the other folder, that just complicates life. Update without exporting should only be done when the user is not working on anything in their working copy, if they intend to start working on something they should do the export first, or work directly in the svn folder and export their changes afterwards to test them.
 
Last edited:
So my intention with this proposal is to have a system that keeps the loose art IN source control, instead of the FPKs, and then work up a solution that keeps that being as easy as it is to develop right now
I was going to ask you what the benefits even are for this. My only concern has been the time it takes to upload or download a given file becoming too long to manage for weak internet connections. So with FPKs the size they are, this is not as much a concern. I'm a little concerned we'll start seeing some bugs as it is now because we found at some point you couldn't have more than some 8 or so FPKs without some things going wrong and fixed things, or so it seemed, by cutting down how many we had. I now wonder if it was a misdiagnosis and an incidental fix took place in the reorganization.

However, you gave a few compelling reasons
- Modifying art contents is now trivial. Just modify the unpacked folder contents
- Modifying art contents is more robust. You cannot accidentally override an existing path, you cannot accidentally fail to correctly merge FPKs and delete someones changes.
- SVN operations are faster. Only the actual art files that changed will need to be updated/committed etc.
- SVN has a true record of what actual art files changed because the files go into a directory and just stay there.
- We can exclude other things from the mod directory, like the source, further improving load time.

So I get why you're looking to do this.

As I said the only limitation is that you can't fetch SVN directly into the mods folder with the solutions.
Ok, so this is what bothers me.

We have to maintain the mod folder in 2 locations. Ugh. That is just a rough situation for a lot of us. But if that's basically ALL it means, then it may be workable.

However you can choose to "install" the mod (with a script) from wherever you fetch it to in such a manner that you never need to copy anything to the mods folder and back to run the game.
So any time you update the mod (fetch), you have to fetch it in the modding folder location and then run the install script? Are you saying that we cannot mod directly into the folder that the mod runs from? (AKA, edit the files and code in the folder which is inside the mods folder) Would take some getting used to, having to run the install script with every adjustment. I suppose it could be done but it doesn't sound pleasant. Adds an extra step for every single most minor change and one easily forgotten leading to wondering why your changes aren't showing up when you run the game.

It is automatically sharing the parts of the SVN directory that are needed for the mod to run, and more importantly NOT the parts that would make it slow. This is (for all intents and purposes) identical to having SVN in the mods folder, without the drawbacks.
I'm not sure what you mean by automatically sharing. This is the part that makes me think you're saying we can change files directly in the mods folder version and it will update back to the repository version and vice versa? That there's an automatic copy-over system in place? Even for file deletions?


People who DO have SVN in mods folder need to move it out, a one time operation which took me about 30 seconds. Cut+Paste+Done. Then they need to run the install script to link the SVN folders to their mod folders. If even this is too much I can make a script that will just do it all automatically for them. I can also, along with the FPK auto packing in the DLL, add a dialog that will warn if someone didn't do this and tell them what to do.
This doesn't sound too bad, so long as we're not stressing the memory on some systems to demand those systems are maintaining 2 copies of the mod, which isn't a light demand if it's not a fairly modern system. I might need to store the 2nd copy in a D drive instead of the C due to memory limitations on the solid state drive.

Just trying to imagine the reality of working with such a structure. I'm not sure about some of the other arguments for other methods yet. My concern is we have people trying to use the SVN getting lost and giving up on that and this adds... steps.
 
So any time you update the mod (fetch), you have to fetch it in the modding folder location and then run the install script?
Negative. It is ONE copy that appears in two places. It is a form of shortcut. A directory junction means that the actual same directory appears in two places on your file system. If you make any modifications inside that directory those modifications are being done to the same exact files and subdirectories. Not copies, but the actual same files.
In math parlance you made the directory tree into an acyclic graph.

Are you saying that we cannot mod directly into the folder that the mod runs from?
You absolutely CAN. However it would not have most of the assets you want to modify, the loose art and source code would NOT have shortcuts in the mods directory, because the entire point is to not include them. XML and python WOULD be there because it is loaded directly.

I'm not sure what you mean by automatically sharing. This is the part that makes me think you're saying we can change files directly in the mods folder version and it will update back to the repository version and vice versa? That there's an automatic copy-over system in place? Even for file deletions?
See my above answers.

I might need to store the 2nd copy in a D drive instead of the C due to memory limitations on the solid state drive.
Like I said it is one copy just shared, so no extra space.

My concern is we have people trying to use the SVN getting lost and giving up on that and this adds... steps.
Well, the whole SVN setup isn't exactly user friendly to being with. I would prefer to just write a tool for our mod that does install and update for them (I think RoM had one). No need for tortoise svn, or copy pasting urls or whatever, and no need to worry what it is actually doing behind the scenes. This tool can also integrate detecting crash dumps and making zips, and linking to the forums/discord etc. etc.

But right now I just want to sketch out what I think are improvements for a more sustainable long term development model: this project has been around for more than 10 years, I think it has another 10 in it. I want to try and help make that more likely by lowering barriers to entry, making contribution easier, making iteration faster, reducing or eliminating sources of human error where automation can get it right every single time (FPK packing is an example of this), making releases easier to organize and stabilize etc.
 
You absolutely CAN. However it would not have most of the assets you want to modify, the loose art and source code would NOT have shortcuts in the mods directory, because the entire point is to not include them. XML and python WOULD be there because it is loaded directly.
Would this cause any problems with attaching a debug dll to the process successfully?
See my above answers.
Some fascinating magic indeed.

So you would have to manipulate graphic and source files from the 2nd folder location but when you place a dll/pdb in the assets, which you'd have automatically happen if you run the batch generator, that would immediately be updated to the mod folder as well. hmm... And if you change anything about the graphics, you need to run the updater.

It sounds like you'd need to run commits from the external folder? Or can you run commits from the mod folder as well? Updates would generally be done on the external folder?


So if someone was to do a fresh checkout, they'd have to just run a clearly named install batch that it comes with it. That batch would install the cloned folder in the mod folder location, while packing the FPKs in that location (but not the modder's location - FPKs would only exist in the mod folder location?)
 
Would this cause any problems with attaching a debug dll to the process successfully?
I see no reason why? As far as Civ is concerned the linked files are all in the mod folder, it can't tell any difference at all.

And if you change anything about the graphics, you need to run the updater.
Not if the original purpose of this thread is realized and I make the FPK live (tm) system :D

It sounds like you'd need to run commits from the external folder? Or can you run commits from the mod folder as well? Updates would generally be done on the external folder?
Forget about the mod folder entirely after running the initial setup, you do everything in the external folder from then on.

So if someone was to do a fresh checkout, they'd have to just run a clearly named install batch that it comes with it.
If fresh = delete the directories / root then yes.

Hopefully you noted alberts is preferring directory restructure rather than this approach (out of what I presented) and I can say it has some definite benefits for simplicity if someone actually does need to know how the system works, along with making the repository itself neater. Certainly if this project was new that would be the way to go, personally I go 50/50 between that and this solution at this point.

Another caveat of this system: junctions/hardlinks only work on the same drive, so your SVN must be on same drive as mods folder. If you checked out to mods folder previously then this shouldn't be a problem. But it could some day cause someone some slight confusion or annoyance.
 
Last edited:
Hopefully you noted alberts is preferring directory restructure rather than this approach (out of what I presented) and I can say it has some definite benefits for simplicity if someone actually does need to know how the system works, along with making the repository itself neater. Certainly if this project was new that would be the way to go, personally I go 50/50 between that and this solution at this point.
I can life with the junction method as well i see no problems with using it.

Others might not because of the additional scripted step to setup the junctions, create
fpk's and compile the dll. At least that was what i gathered from the conversations and that kinda made the solution which just works the one to choose.
 
Forget about the mod folder entirely after running the initial setup, you do everything in the external folder from then on.
Ok, as I thought. And just to clarify, if we do modify, say, an xml file in the mod folder version, those edits would automatically be also made in the external folder's version of that file right? So you couldn't do that with sources - they'd only be located in the external - but for more basic XML you could?

I see no reason why? As far as Civ is concerned the linked files are all in the mod folder, it can't tell any difference at all.
For whatever reason, if I have a debug dll compiled for the mod and I try to load a game saved on a different mod folder name so I then have to change the name of the mod folder to load their save, the debug dll will need to be recompiled for VS to attach to the process properly. I'm thinking this won't be a problem here because we're not changing operative mod folder/mod names and I think that's the key problem it hits.


Just to clarify what I sorta brought up earlier, the mod folder version doesn't have to be in the same drive as the external linked folder right? I could have it in the D drive while the mod folder is in C?

Hopefully you noted alberts is preferring directory restructure rather than this approach (out of what I presented) and I can say it has some definite benefits for simplicity if someone actually does need to know how the system works, along with making the repository itself neater. Certainly if this project was new that would be the way to go, personally I go 50/50 between that and this solution at this point.
In noted that he had another opinion but I haven't researched his option to the extent of understanding as far as this one yet so I can get a full comparative analysis going in my head.

I can life with the junction method as well i see no problems with using it.

Others might not because of the additional scripted step to setup the junctions, create
fpk's and compile the dll. At least that was what i gathered from the conversations and that kinda made the solution which just works the one to choose.
I'm not thinking having to run an install batch after checking out a new repo folder to a non-modules location would be tough to tell people they have to do. I think anyone could probably manage that.

The ability to start the game, then load the mod from there might be so habitual for many though that this inability to run things like that anymore might be a little uncomfortable for some. Though you could still do that as long as you used an updater batch after any update I suppose.
 
Ok, as I thought. And just to clarify, if we do modify, say, an xml file in the mod folder version, those edits would automatically be also made in the external folder's version of that file right?
All the files mirrored between the external SVN folder and the working copy folder would actually be the same files, one is just a mirror of the other, so a change in one is the same as a change in the other.
It may appear to be two files as you have two folders for it at two different places but on your hard drive there would only be one of each mod file.
None of the files in your working copy would be real in a way, simply a mirror of most of the files that are in the external folder.
This is true for most files from the repo, some files will not be mirrored like the source files and unpacked art, those will only be found in the external repo folder.
 
I'm not thinking having to run an install batch after checking out a new repo folder to a non-modules location would be tough to tell people they have to do. I think anyone could probably manage that.

Exactly what i said to others.
 
Last edited:
All the files mirrored between the external SVN folder and the working copy folder would actually the same files, one is just a mirror of the other, so a change in one is the same as a change in the other. For most files, some files will not be mirrored like the source files and unpacked art, those will only be found in the external folder.
That's how I thought I'm understanding this but I just want to make sure I'm not being stupid and misunderstanding.

Part of what I'm internally debating in this is if I'd want the source files in the mod folder despite being able to keep them in the external one only.
 
No i meant that i said exactly that to other modders but they said that additional step is too much to handle.
In the context of these discussions, many of us, self included, can easily get overwhelmed to the point of wanting to just say forget ALL that.

This is such a simple step that if it's boiled down to just that, I think we can make it work.
 
Others might not because of the additional scripted step to setup the junctions, create
fpk's and compile the dll. At least that was what i gathered from the conversations and that kinda made the solution which just works the one to choose.
There would be no manual extra step to build FPKs and DLL. DLL is still just like normal in the repo, no extra step there. FPKs are build using the system I propose in the first post in this thread. i.e. entirely automatically. So no manual step there. Only manual step is run one extra script whenever you have done a *clean* check out.

those edits would automatically be also made in the external folder's version of that file right? So you couldn't do that with sources - they'd only be located in the external - but for more basic XML you could?
You could but why on earth would you want to! You have one folder structure that does have everything in it, and then another that doesn't, just do all edits in the one that does!

For whatever reason, if I have a debug dll compiled for the mod and I try to load a game saved on a different mod folder name so I then have to change the name of the mod folder to load their save, the debug dll will need to be recompiled for VS to attach to the process properly. I'm thinking this won't be a problem here because we're not changing operative mod folder/mod names and I think that's the key problem it hits.
I don't see there being a problem. The actual Mods\Caveman2Cosmos folder itself is NOT a link it is a real folder containing a set of links (Assets etc.). You can change that real folder name to whatever you want to get saves to work.

Just to clarify what I sorta brought up earlier, the mod folder version doesn't have to be in the same drive as the external linked folder right? I could have it in the D drive while the mod folder is in C?
No you can't sorry. I did add that extra note in an edit on that last post as it might be surprising. However you aren't in a worse place than before because of this limitation. Before you could either have the mod IN the mods directory which is obviously on the same drive, or you could have it on another drive and have to manually copy/export from that drive, which you can still do just fine in the new system.

The ability to start the game, then load the mod from there might be so habitual for many though that this inability to run things like that anymore might be a little uncomfortable for some. Though you could still do that as long as you used an updater batch after any update I suppose.
With the hardlinks you definitely CAN still start the game and load the mod from there, that is one of the reasons for this and not doing a directory restructure: it maintains that capability. /edit maybe I misunderstand what you mean here. There is no script to run after update with this system, only after clean get.

No i meant that i said exactly that to other modders but they said that additional step is too much to handle.
Hopefully that was when the additional step was running a script after every update (which would be annoying), whereas with the junctions it is just running it after every *clean* get. Although those are way more commonly required than they should be (roll on git :mischief:).
 
In the context of these discussions, many of us, self included, can easily get overwhelmed to the point of wanting to just say forget ALL that.

Yes, I hope it is understood, I am explaining in detail these suggestions, with the intent of turning the final products into zero, one or occasionally two click solutions.
 
You can change that real folder name to whatever you want to get saves to work.
You won't be able to start C2C without the mod folder being exactly Caveman2Cosmos since PPIO was merged, so going forward there should be no players who have C2C saves made where the folder is named differently.
The exception will be v39 saves though, these may still have been made under a different mod folder name than Caveman2Cosmos.
 
Back
Top Bottom