ModBBF, for adding many files (100s) outside of ModBuddy very quickly

Pazyryk

Deity
Joined
Jun 13, 2008
Messages
3,584
EDIT: This is a bit obsolete now...
  1. ModBuddy seems faster to me now.
  2. If you have 100s of large art files (like me) what you should do is this: Split your mod into two mods, one called "MyMod Media Pack" containing all of your many art files. Make your main mod require the Media Pack in dependencies (which work now). If you do all your difficult Lua/SQL/XML code in the non-media mod, then you will rarely need to build the larger Media Pack.


Download from the files database here: http://forums.civfanatics.com/downloads.php?do=file&id=17823

If you have a lot of art (say >10 large dds files) then Modbuddy gets really slow. This tool can add 200 .dds files (~0.5 M each) in 4 seconds, probably only limited by my system's ability to read/copy files. (OK, 31 seconds when I have ModBuddy, Civ5 and the Tuner running at the same time...) It should work for any other file extension too.

You specify (in User Settings at the top of the script):
  • your mod name (without version number)
  • path to MODS folder
  • and then in sets of three:
    * source folder for your files (anywhere on your computer)
    * destination folder in mod
    * file extension​

The script will:
  • find the "current" mod (the one with the highest version number)
  • read your files (only those with specified file extension in specified folders)
  • generate md5s
  • add to .modinfo right after the <Files> tag with import="1"
  • copy files to specified mod folders, creating folders/paths as needed

The script detects and skips files already in your mod (based entirely on .modinfo "<Files>" section), so you can call it repeatedly without harm (note: calling twice in succession actually has a nice side effect of forcing a mod browser update, so I always do that). So, the way I use this is that I generate and keep .dds files in the same folder as original .jpeg's (in my own Art folder far from Civ5 mod stuff). Whenever I make a new .dds (or build a new mod version from ModBuddy), I just run the script again and it only adds the files not already in the latest mod version. I add no art in ModBuddy at all (though it won't hurt if you do).

More details in header information in the script (pasted below). My apologies to all, it is written in PERL (which I use for work), but I have included instruction for obtaining PERL free and getting the needed modules.

Code:
#!/usr/bin/perl
use File::Path;
use File::Glob ':glob';
use File::Copy::Vigilant;
use Digest::MD5;
########## By Pazyryk. Use, alter and distribute as you please. ################
#
# Gets all files in specified folders (of specified extension) and copies into
# the "current" mod (the one with the largest version number) in the specified
# folders (creating folders and paths if necessary), without creating redundant
# items. It will add them to the .modinfo file with the correct md5 and
# "include=1". You can create as many source->destination folder sets as you
# need, each with its own specified file extension (User Settings below).
#
# WARNING! Script overwrites .modinfo file. Use at your own risk.
#
# Notes:
#  * When specifying source folders, files of non-specified extension are
#    ignored. (E.g., I keep .dds's in the same folder as original .jpeg's.) 
#  * If a file name already exists in .modinfo (between <File> and </File>)
#    then this script will skip that file, regardless of folder or import=1 or
#    =0. So you can call this script repetitively (e.g., by mistake or after
#    creating a new .dds file) without creating redundant items in your mod.
#  * The script uses the .modinfo file (and only that) to check for already
#    existing files. It will overwrite a file of the same name in the specified
#    destination folder if it isn't in .modinfo.
#
# Installing PERL and required moduals
#  * Free PERL download: http://www.activestate.com/activeperl/downloads.
#  * If your PERL build is like mine, then you will need to install two
#    moduals: Digest-MD5 and File-Copy-Vigilant. You can type "ppm" in any
#    command window to get the Perl Package Manager. If not installed, click
#    View->All Packages, then right-click->install on the needed modual. No,
#    you're not done yet. Look for the obscure green arrow (->) in the top
#    panel (mouse-over says "run marked actions") and press it. 

####################    User Settings    #######################################

$mod = "";		# e.g., "MyMod", not "MyMod (v 1)"
$mods_folder_path = "";	# Path to MODS folder; see examples below

$art_from[0] = "";	# Path to source files
$art_to[0] = "";	# Path relative to mod folder
$extension[0] = "dds";	# File type to copy/add (note: I've only tested dds)
			# Repeat last three lines with [1,2,...] as needed,
			# but they must be added in sets of three

# Notes:
# $mod must be without version!
# Example $mods_folder_path:
# "C:/Users/Me/Documents/My Games/Sid Meier's Civilization 5/MODS"
# (Note that spaces and ' are handled OK. Not sure about other weird chars.)
# Example art_from, art_to, extension triplet:
#	$art_from[0] = "C:/Users/Me/Desktop/MyArt/People";
#	$art_to[0] = "Art/HiRes2D/People";
#	$extension[0] = "dds";
#	Add more triplets as needed with index 1, 2, etc. 
#
# Paths can be relative if you like, so "" works as current dir (except
# $art_to[#] is always relative to mod folder).

# WARNINGS!
#  * The actual mod folder name must be of standard format "MyMod (v 21)" with
#    the same exact two spaces (the number can be any integer > 0).
#  * You should not have anything resembling " (v #)" in any of the User
#    Settings specified above!

################################################################################

I haven't learned how to make ModBuddy extensions yet. If someone includes detailed instructions below, I'll add it to this post.

No, I won't be translating this to Lua or some other language. But feel free to do so yourself.
 
Jeez it's quite around here. No bugs. No requests. No lol or controversy over the name. It's Best Buddy Forever, of course. A play on ModBuddy. Get it? Anyone? Is anyone there?
 
Jeez it's quite around here. No bugs. No requests. No lol or controversy over the name. It's Best Buddy Forever, of course. A play on ModBuddy. Get it? Anyone? Is anyone there?

Hi Pazyryk,

First, I like the name. :lol:

This is a great idea, however in practice it doesn't work for me, because I want to duplicate a file structure recursively, so for me this just trades the tedium of lots of clicking around in ModBuddy for the tedium of lots of copy/paste.

Requests:
  1. Allow recursion.
  2. Allow the user to specify multiple $extensions for one set of folders.
  3. When a <Files /> entry exists (i.e., when no files have yet been added to the mod), replace it with <Files> and </Files> automatically.

Thanks!
 
Nutty,

Sorry for the slow reply (I've been taking a little modding break). I'm just trying to get my head around your request. I think what you want it to do is to fully replicate a file structure from one place on your computer to the mod. In other words, you have a complete representation of the mod folder structure (with some files) on your computer somewhere, and you want ModBFF to replicate that structure in the mod. Do I understand correctly?

While that's a perfectly reasonable idea, it isn't compatible with the way I work. I have various assets in different places on my computer (fantasy 2D art here, music somewhere else) with no correspondence to how they should go in a mod. The script allows me to get these from where ever they may be and put them in the mod in a sensible way. The cost in this approach is that I have to specify destination folder structure, rather than the script trying to infer this based on file structure of source files.

It's certainly an interesting idea, though, and I'd consider writing it to do that as an alternative "mode". I personally hate complicated folder/subfolder structures (I'd far rather just have 100s of files in one folder, assuming they are related such as 2D popup art .dds files). But is there really any pressing need to have complicated folder structure in your mod? (Perhaps this is needed for 3D models?... I haven't dealt with this at all yet.)

#2. Perhaps wildcard functionality ("*" gets everything) would be useful? If you only have 2 file types, then just make 2 sets of "to" "from" "extension" sets.

#3. ModBFF can't make a modinfo file from scratch. It always has to follow ModBuddy. Is it possible to make a mod in ModBuddy that has nothing in the Files section?

I'd also consider making some additional changes to ModBFF, such as the ability to add things to OnModActivated and so forth (I'll have some programming time around Christmas). But only if there is sufficient interest from others.
 
Sorry for the slow reply (I've been taking a little modding break).

No problem, I totally understand. Thanks for following up.

#1.
Exactly. You're right in your assumption that this is for 3D models. It would be a pain for me to have all the models in one directory, especially where unit modders use varied naming conventions for their files.

#2.
I only need 4 filetypes, but in my use-case, 4 sets for each of many folders made the idea of using your tool for my purposes much worse. If it copied the entire folder structure, then (for me) wildcard functionality is not as important. To make wildcards work for me, I would want to include everything EXCEPT certain filetypes.

#3.
After creating an empty mod, before adding any files, the files section is <Files /> rather than <Files></Files>.

I'd probably use the OnModActivated feature, FWIW. While you're adding settings, perhaps you could have one for import="0".

Thanks!
 
OK. I can see how your suggestions are necessary to work with 3D models. Perhaps for simplicity it would be better if ModBFF just replicated file structure given a starting path (not the way I've been working, but now that I think about it, it seems logical). If you only have to specify one path, then it would be easy to add any number of "include" file types (with * wildcard) and "exclude" file types (with * wildcard) which should cover any possibility. The only loss in this is that you can't specify different extension types for different folders (but so what? jpegs are always excluded, dds's are always included, etc.).

I'll try to implement this. Unfortunately, I won't have much time until Christmas break. So it may be end of month before I can do it.
 
The only loss in this is that you can't specify different extension types for different folders (but so what? jpegs are always excluded, dds's are always included, etc.).

Couldn't you still use your current method of defining multiple sets to use different extension types for different folders?

I'll try to implement this. Unfortunately, I won't have much time until Christmas break. So it may be end of month before I can do it.

No worries. Where your script will be useful in the future, when I was looking for it, my task was a one-off where the tedium of dealing with ModBuddy was truly daunting.
 
It's also possible that the will "fix" whatever it is that slows ModBuddy builds so much, which would obsolete this utility (I'd be happy with that). I don't understand what ModBuddy is doing with big .dds files that takes so long. It's certainly doing something other than md5 generation and file copy, but nothing useful as far as I can tell.

Couldn't you still use your current method of defining multiple sets to use different extension types for different folders?

I guess I could do it so that you specify extensions (and excluded extensions) for a folder plus all sub-folders of that folder. That way you can have a set of extensions for your "Art" folder, another for your "Lua" folder, and so on. Possible. But is there really any need for this? Can you describe a scenario where this feature would be useful? In my work flow, I have certain file types that are never loaded (.jpeg, .txt) and others that are always loaded (.dds, .lua, .xml).
 
Sorry, I forgot to answer this question!

But is there really any need for this? Can you describe a scenario where this feature would be useful?

I was thinking you would retain separate sets for dealing with different folder trees. You're probably right regarding the extensions, but I see no problem leaving it in for those who want to be sure about what files they're including from each path entered.
 
@Nutty, I've sort of missed my promise to do this in December. My actual mod is taking priority and will for a while.

While I agree it would be nice to have a "recursive" file approach to this, I have to say it is not quite necessary even for 3D art assets. You can dump all of your art assets together into one or several folders, rather than having a separate file structure for each art asset. It may not be how you want to do it, but that is what you are stuck with (for now) if you want or need to use ModBBF rather than ModBuddy to add to your mod.

You should realize that every file added in a mod has to have a unique name even if you put it in a different folder. So you might want complex file structure from an organizational point of view, but it may actually cause confusion and colliding resources rather than help. I always prefix my assets so that related ones are together by alphabetical order.
 
@Nutty, I've sort of missed my promise to do this in December.

No biggie. I wouldn't have "needed" it until danrell's done with his new pack of units.

Either way, there was never any thought of it being strictly necessary, but I also had the thought that where 2 mods are using the same unit art resource, if the unit artist's file naming is left alone, there won't be multiple instances of the same art put into the VFS. Then again, I'm not sure that would result in a performance impact even if there were hundreds of units otherwise duplicated.
 
Back
Top Bottom