FPK live (tm)

[olduser]

And this is how I always have my SVN folder. I Never Ever use it to play from. Or work from either. Kept in a completely different directory from where Game directory is. And from where my working copy is. Reduces for me the chance of committing a file that may have a mistake. Because the working file is where the adjustments are made. Then a copy is put into the Mod for play testing. If it all tests out okay, then I put the file into the SVN and Commit.

What if someone else changes the same file you have changed without you noticing?
Then you just copy the file back into the svn after updating you might overwrite those changes!

This actually happens.

 

[billw2015]

I'd like to see a system that supports either method is basically what I'm looking to achieve here.

So my intention with this proposal is to have a system that keeps the loose art IN source control, instead of the FPKs, and then work up a solution that keeps that being as easy as it is to develop right now. I think I have a solution that is in fact *easier* than it is right now for a lot of people, and more difficult for nobody. As I said the only limitation is that you can't fetch SVN directly into the mods folder with the solutions. However you can choose to "install" the mod (with a script) from wherever you fetch it to in such a manner that you never need to copy anything to the mods folder and back to run the game. It is automatically sharing the parts of the SVN directory that are needed for the mod to run, and more importantly NOT the parts that would make it slow. This is (for all intents and purposes) identical to having SVN in the mods folder, without the drawbacks.
Alternatively if you like to work as toffer describes you can choose to *actually* install it (again with a script), that will just copy the required stuff to the mod folder. Or probably we can modify the svn export settings such that it ignores unpacked data etc.

Upsides to doing things this way have mentioned a few times, I will reiterate:
- Modifying art contents is now trivial. Just modify the unpacked folder contents
- Modifying art contents is more robust. You cannot accidentally override an existing path, you cannot accidentally fail to correctly merge FPKs and delete someones changes.
- SVN operations are faster. Only the actual art files that changed will need to be updated/committed etc.
- SVN has a true record of what actual art files changed because the files go into a directory and just stay there.
- We can exclude other things from the mod directory, like the source, further improving load time.

Downsides:
- People who DO have SVN in mods folder need to move it out, a one time operation which took me about 30 seconds. Cut+Paste+Done. Then they need to run the install script to link the SVN folders to their mod folders. If even this is too much I can make a script that will just do it all automatically for them. I can also, along with the FPK auto packing in the DLL, add a dialog that will warn if someone didn't do this and tell them what to do.

Is there another downside I'm missing? This seems like a one time minor inconvenience for a load of benefits.

 

[Anq]

Doesn't packing before launch consumes a lot of time? And it puts a lot of strain on the hard drive. Not every one has an SSD to withstand tens of thousands of read/writes every time they launch the exe.

I'm also running from the SVN, although I do have a separate checkout of the trunk, since I switch to and from branches in subfolders in my Mods repository from time to time. I never use the export function but check-out a full copy of some directory.

 

[raxo2222]

Doesn't packing before launch consumes a lot of time? And it puts a lot of strain on the hard drive. Not every one has an SSD to withstand tens of thousands of read/writes every time they launch the exe.

Yeah, packaging takes a lot of time - I was testing these new tools.

 

[billw2015]

Doesn't packing before launch consumes a lot of time? And it puts a lot of strain on the hard drive. Not everyone has an SSD to withstand tens of thousands of read/writes every time they launch the exe.

It would NOT pack every time. It would pack when things change. And even then I can make it only pack changes. There are many ways to solve this problem, I am not worried about it. Just off the top of my head (all these are automatic, no change to user actions):
- Only pack when the number of new files is more than 100. Up until then keep new files in the art dir instead.
- Only pack when SVN version changes, keep local changes in art dir instead.
- Only pack once a week.
- Only pack in the background when you launch and use the packs from THIS launch on the NEXT run.

I spent years in my job working on precisely these kind of problems, but if in the end I can't solve it in a manner that doesn't impact people then it just won't be used.

I'm also running from the SVN, although I do have a separate checkout of the trunk, since I switch to and from branches in subfolders in my Mods repository from time to time. I never use the export function but check-out a full copy of some directory.

Yeah for you things get easier I think. You can keep both your SVNs outside of the mod, and to switch you just run the script to make the links in the repo you want to use. It doesn't require renaming a directory or copying any files on your part. It is just flicking a switch between them.

 

[olduser]

Restructure mod folders:
- Everyone needs to update their mod directory in ini file or command line.
- In game menu doesn't recognize it correctly. It still sees Caveman2Cosmos, but tries to load it from the root folder instead of the subfolder, which obviously doesn't work.
- Breaks save games! This is insurmountable unless coupled with other save breaking changes. HOWEVER it does actually enforce that that saves are broken with a dialog instead of errors, so in this respect it is actually useful.
- Update all the references to assets in the main game. All references to main game files are broken and need an extra ..\ added. Pretty simple search replace, dunno how hard to confirm all references are fixed though.

This seems to be the easiest way to set this up. The save compatiblity will be broken sooner or later anyways.

 

[Anq]

- Update all the references to assets in the main game. All references to main game files are broken and need an extra ..\ added. Pretty simple search replace, dunno how hard to confirm all references are fixed though.

Eventually we're releasing a stable version that's at Mods\Caveman2Cosmos, so don't move the Resource folder and don't change the theme files. Geez, things are harder to solve.

Why not create a separate folder that's a sibling to our svn /trunk?

Adding it this way is far better than moving the whole mod structure.

 

[olduser]

Eventually we're releasing a stable version that's at Mods\Caveman2Cosmos

That would not work anyways because changing the modfolder breaks saves.

 

[Anq]

That would not work anyways because changing the modfolder breaks saves.

I mean don't change the mod folder, but put the unpacked art files outside of svn /trunk.

How hard is it to keep the current path?

Other mods always release a version that can be played at Mods\Modname

 

[billw2015]

This seems to be the easiest way to set this up. The save compatiblity will be broken sooner or later anyways.

You think? Seems like the most difficult to me. My other solution boils down to just move your SVN repo out of mods and run a single script. How is everyone needs to change mod path, break all saves and update loads of refs easier than that?

Eventually we're releasing a stable version that's at Mods\Caveman2Cosmos, so don't move the Resource folder and don't change the theme files. Geez, things are harder to solve.

Why not create a separate folder that's a sibling to our svn /trunk?

Adding it this way is far better than moving the whole mod structure.

Yeah changing the folder structure is my least favourite option since the hard link option occurred to me.

I mean don't change the mod folder, but put the unpacked art files outside of svn /trunk.

Then everyone needs two repos to run the game? Seems more complicated.

I'm still not seeing any explanations of why the method I propose of a choice between hardlinks and SVN export is actually difficult in any way?

 

[Anq]

Then everyone needs two repos to run the game? Seems more complicated.

The normal /trunk repo will contain the FPKs, and for the artists they can check out the /Arts repo and run the scripts to update the FPK contents, and put them back to trunk. (Manually is just fine, don't need automation here)
It's not hard to do. Rather than making the /trunk structure unintuitive and hacking the File System with junctions...
Not every one is tech-savvy to know what the junction/hard-link is doing on their computers.

 

[billw2015]

The normal /trunk repo will contain the FPKs, and for the artists they can check out the /Arts repo and run the scripts to update the FPK contents, and put them back to trunk. (Manually is just fine, don't need automation here)

Seems *okay*, not really great. It solves one of the problems I mentioned at the cost of more fiddling for artists rather than less.

Rather than making the /trunk structure unintuitive...

Trunk structure doesn't change in my suggestions other than instead of FPKs you have a folder with their contents. This has no functional impact on anyone not changing art, and anyone who is can do it easily without needing extra branches, or running scripts.

 

[Anq]

Trunk structure doesn't change in my suggestions other than instead of FPKs you have a folder with their contents. This has no functional impact on anyone not changing art, and anyone who is can do it easily without needing extra branches, or running scripts.

I see. You may need to edit https://forums.civfanatics.com/goto/post?id=15509581#post-15509581 this post to say no restructuring as well as redirecting the resource files is needed.
(Edit: Ah my bad, you were only listing the options)

You'll also need to WARN every SVN users, that Junctions mean their local modification in the Mods\Caveman2Cosmos folder is also on their SVN "working copy", so careless people will either screw up the svn repo or let the svn repo screw up their local changes there. The cost to avoid exporting is too high.

Seems *okay*, not really great. It solves one of the problems I mentioned at the cost of more fiddling for artists rather than less.

But has the advantage of being simpler, not involving hi-tech NTFS junctions

 

[billw2015]

I see. You may need to edit https://forums.civfanatics.com/goto/post?id=15509581#post-15509581 this post to say no restructuring as well as redirecting the resource files is needed.

Yeah that post was presenting a set of *alternative* solutions. You can see under directory junctions i do not mention any restructuring.

You'll also need to WARN every SVN users, that Junctions mean their local modification in the Mods\Caveman2Cosmos folder is also on their SVN "working copy", so careless people will either screw up the svn repo or let the svn repo screw up their local changes there. The cost to avoid exporting is too high.

No, anyone currently NOT using SVN in their mods folder can go on like before, we just fix up the export or make a script that basically does the same thing. Nothing changes for them understanding wise.
Anyone currently using SVN in their mods folder also goes on like before, but they just need to do the setup first. Just as previously changes in their mods folder effected SVN, they would continue to do so.

But has the advantage of being simpler, not involving hi-tech NTFS junctions

Doesn't matter what the solution is if it has no effect on the user. However I think windows users understand shortcuts.

 

[olduser]

You think? Seems like the most difficult to me. My other solution boils down to just move your SVN repo out of mods and run a single script. How is everyone needs to change mod path, break all saves and update loads of refs easier than that?

Only needs to be done once after that it just works.

 

[billw2015]

Only needs to be done once after that it just works.

If other people are okay with it I certainly am. However once it is done a couple of things won't work anymore: old save games and running the mod from the menu. I don't know how important it is, save games are going to break anyway I guess. For running the mod I would like to make a proper installer anyway at some point, running it from the menu is the worst way to run it. But some people do it from what I gather.

 

[olduser]

Did you have any time to test if this really works?

Actually turns out maybe there is an entry point we can use: DllMain. It is called before any loading from what I can tell.

Yep confirmed:
https://i.imgur.com/3jOgf9r.gifv

 

[JosEPh_II]

What if someone else changes the same file you have changed without you noticing?
Then you just copy the file back into the svn after updating you might overwrite those changes!

This actually happens.

Because before I commit I Update the SVN 1st! Then look at what has been changed. So that has not happened for me. Attention to details and not a rush job, which can happen with the other method.

 

[billw2015]

Did you have any time to test if this really works?

Works in what respect? Certainly DllMain process attach is called BEFORE FPKs are loaded. I put the break point in, deleted my FPKs, ran the game, hit the breakpoint, restored the FPKs, and the mod launched correctly (assets are there in new game, I can see the FPKs open in process explorer). However indexing definitely occurs BEFORE the mod is loaded hence the discussion here.

 

[olduser]

Works in what respect? Certainly DllMain process attach is called BEFORE FPKs are loaded. I put the break point in, deleted my FPKs, ran the game, hit the breakpoint, restored the FPKs, and the mod launched correctly (assets are there in new game, I can see the FPKs open in process explorer). However indexing definitely occurs BEFORE the mod is loaded hence the discussion here.

Did you only go into the menu only or start an actual game? I tried this and the game crashes.