Nightinggale's Help Center and ModComp factory

Nightinggale · Mar 7, 2016

Looking at what is going on here on the forum, way too much time is wasted moving from one mod to another. This applies when starting a new mod/modcomp, merging modcomps into mods or even just helping people, who have become stuck. So much time is wasted that I have caught myself "why bother?", which isn't something we should be thinking. We need something faster and with less hassle. What we is is a modding industrial revolution

I have created a project for this very purpose. It can be found here: https://sourceforge.net/p/civ4-modcomps-by-nightinggale/code/ci/master/tree/

The core is the git server. It has a branch called BTS_3.19 and it is just that: DLL source, python and xml for vanilla. Well I added "project files", which allows compiling the DLL with all the features from the newest makefile.

Creating a new mod or modcomp:
Fork if you have to. Create a new branch from BTS_3.19 for your mod. The setup is made out of the box and should be able to compile on any computer with a compiler installed already.

Placing your mod on git:
Fork if you have to. Create a new branch from BTS_3.19 for your mod. Drop all your files into the git repository. Add all unversioned files and commit. Git prefers if you do not pack your graphic files, but keep them as individual files.

Benefits of using a common git foundation

the project files are set up already
all mods can be pulled (downloaded) into the same repository. It saves time when switching as it is just a click or two away
merging is a two click operation

Merging might be the most interesting one of everything mentioned here. With your mod selected, select the branch (modcomp) you want to include and click merge. If there aren't any conflicts (you and modcomp changed the very same line), then that's it. If the DLL code is modified, then it needs to be compiled again, but it takes a minute or so because the setup is already present.

It wouldn't be farfetched to say that once this system is populated with mods and modcomps, it would be realistic to add 5 modcomps this way in 10 minutes :crazyeye:

I added the ewi modcomp to the git server. It turned out not to have been made for BTS, but using git to revert all changes not commented to be present in the modcomp turned it into a BTS mod. This mean the upgrade took a few minutes and we should be able to enjoy the revived edition with BTS too. The goal is to add images to events using xml only, which sounds useful, though I admit I haven't actually tested it (yet?).

Getting help from me
Once in a while people complain about a crash or similar. If I am to help from now on, it would have to be through git using this system. It simply takes me way too long to look into each mod individually and set up the compiler for it and all that.

Setting up git for a mod

Spoiler :

SmartGIT guide

Spoiler :

TODO: setting up git to use multiple remotes to fetch modcomps from multiple people.

Using the compile server
I have set up a computer, which once in a while downloads from git, compiles the DLL and places it on a DropBox account.
Compile server file directory

Do note that this is just the DLL and that other files are needed as well. This is primary a service for people without compilers. If you want your mod to be included, post in this thread. The server requires git URL, mod name, branch name and target type. Post here even if you don't know the answers to those and we will figure it out.

keldath · Mar 8, 2016

A w e s o m e!

Sword_Of_Geddon · Mar 9, 2016

Oh wow Night, this is really ambitious! And very much appreciated....

Nightinggale · Mar 10, 2016

Sword_Of_Geddon said:
this is really ambitious

The most ambitious part is the hope that everybody switch to git. It would seem that people who aren't experienced in programming have a hard time realizing why git makes everything much easier, even in one man projects.

I'm not sure the rest of this is that ambitious, at least not on my scale. Compared to what else I do for modding, it's more like business as usual. I wanted an "engine" to handle all the boring tasks and with a bit of thinking, git can do everything I request in that regard. Setup is mostly copying game files from vanilla and project files and git settings from other mods. I don't think I had the need to create anything from scratch at all. The second mod I added (the event image one) was much quicker to do when using git. In fact I think I saved more time than I spend setting up the git system, which mean I spent a negative amount of time on the setup :crazyeye:

The git system I set up is very expandable. Say platyping wants to add his python modcomps, he can click the fork button on the project page, which gives a new git server where he has write access (git can fork without using the fork button, but it takes more skill). He can then add a branch, copy a modcomp, commit, make a new branch, copy the next modcomp and so on. Anybody can then check out one of the two git servers and then add the other one with "add remote". Because both remotes are based on the same "root" (the vanilla I made here), merging will be just as easy as when all the modcomps are on the same server.

The reason I picked platyping (without asking him, I hope that's ok) is because it's python only. Being python only mean it isn't windows only, which doesn't matter for git. Git is available for mac and linux as well as plenty of other platforms.

Again the ambitious part is to get people to adapt to this common git root system. The power in the system relies on modcomps being added using this root and the usefulness only grows as the number of modcomps grow. This mean it now all depend on other people and that is the only thing in this whole setup I can't fully control.

Also if I get people to use this system and periodically push their newest version to a public git server (or one I have access to), then providing feedback and assistance would be a whole lot easier.

Leoreth · Mar 10, 2016

Yeah I agree, this is a great idea and I hope people pick up on it.

What this made me think about, since you are afaik better versed in the C++ world, do you know of any way to set up a build server for the BtS DLL? Compiling the DLL is the most significant hurdle for people who are willing to mod the game, if we had a place where people could just push their sources and the server would build the DLL for them that would help immensely and also give people an incentive to move their code to Git.

need my speed · Mar 10, 2016

One day, when I have time - that might be months in the future, mind; I haven't even played Civilization IV for more than a month - I'll look into this. This seems like an awesome idea! And I'd really like to be able to do fancy DLL stuff.

And on that note, as far as I know - and I'm 99% sure here - your and platyping's solution worked for my mod. So thank you for that as well!

Nightinggale · Mar 10, 2016

Leoreth said:
What this made me think about, since you are afaik better versed in the C++ world, do you know of any way to set up a build server for the BtS DLL? Compiling the DLL is the most significant hurdle for people who are willing to mod the game, if we had a place where people could just push their sources and the server would build the DLL for them that would help immensely and also give people an incentive to move their code to Git.

I have been thinking about that, but it would be problematic. It would work something like this:

people select a server/branch to compile in a web interface
the server somehow forwards this to a build server
the build server pulls changes from the server in question
change checkout to the requested branch
runs the makefile from a script rather than MSVC to compile
compress the dll file and uploads it somewhere

I have no idea on how to do the first two. The rest could be doable if I had a build server. Sadly such a build server would be CPU intensive. Not only does that rule out the free servers, it also rules out the cheap servers

Another (more primitive) approach could be simply to make some sort of script, which compiles the newest DLL for each branch, unless it is unchanged since last run. Something like that could run in the background on a regular computer and could be executed like once a day. Sure it could mean it would take up to 24 hours before it would compile the DLL, but we could skip the server thing entirely. It would require people to add their mod server on a positive list to get included, but we would require that with a server based system as well.

I have been thinking something like making a zip with a complete installer. Something, which runs all the needed installers in order (either automatically or they are named 1,2,3...) and will end the install by copying the needed files into position. If it goes as planned, people should be able to just doubleclick the project file, hit F7 when the project is open and it will compile the DLL. Git takes care of placing project files, makefile and all that, meaning it should be enough to just install the compiler. Modiki has a guide and with the git setup, only some of the instructions of step 1 would be needed.

Any other ideas on this issue would be most welcome. None of what I have written here can be considered ideal and it would be great if somebody can come up with something better.

Leoreth said:
if we had a place where people could just push their sources and the server would build the DLL for them that would help immensely and also give people an incentive to move their code to Git.

The more I think about this sentence, the better it gets. I think it might actually hold the key to make the git approach a success. With this, we don't have to tell people to use git. Instead people will "demand" to use git and be connected to this system. Now if only the "remote compiling" system can be made without too much work :think:

Leoreth · Mar 10, 2016

Infrastructure of this sort definitely exists for the JRE world, on Github you can even directly integrate with your repositories. And since our modding work counts as open source I think most of it would work free of charge. For C++ I am mostly clueless though, it just goes to show how little I have to do with the Windows world outside of BtS modding.

But reading what you wrote I think I got too wrapped up in the typical SE "is there a tool for that" mindset. There's really not that much to do even on a bare server (AWS has a free tier and if I'm not completely mistaken Windows AMIs, so just setting up an instance there shouldn't be a problem). We're not at the stage yet where there is massive demand so I don't think performance should be that much of a concern.

There are Github hooks (no idea about sourceforge, just talking from experience here) to listen to and trigger pulls and builds whenever something is pushed to a branch. On the build server side there should be some basic scheduling to avoid getting swamped in coinciding pushes but otherwise something close to on demand seems feasible. I don't think people with major projects that compile their code dozens of times per day would be using this, it's mostly people who want to change one minor thing.

Optimally the build server would even be able to push the new DLL back to the repository/branch, although I'm not sure how to set that up with key exchanges etc. in a way that doesn't become too painful for an inexperienced user.

Actually the only thing I'm not too certain in is setting up the compiler on such a system. But your idea of one ready to go compiler installer is already great. Once that is working it would already help people a lot.

Nightinggale · Mar 10, 2016

Adding DLL files to git is a bad idea. It adds 3-4 MB (compressed) each time anything change in the code, regardless of how minor. Multiply this with the number of revisions which change the DLL in all branches combined and we are suddenly talking about hundreds of MB of data. When somebody downloads the git repository for the first time, it downloads everything, which mean adding the DLL files greatly increase the download time. Git stores compressed diff files for text based files, meaning changes to the source code will not generate huge amount of data.

Addgn graphic files cause the same issue, which is why one git repository should only contain one mod, not multiple mods like I just added multiple modcomps to one repository. Also as I mentioned earlier, save individual graphic files to git, not packed as changes would make git store a new packed file and use up all the disk space.

Storing compiled DLL files would have to be some sort of file release system, not git itself.

Leoreth said:
Actually the only thing I'm not too certain in is setting up the compiler on such a system. But your idea of one ready to go compiler installer is already great. Once that is working it would already help people a lot.

If you can find a server, which will allow us to install a windows based compiler from 2003, then we could be talking. Maybe we could use a linux based server with wine, though I have little experience setting up wine.

Leoreth · Mar 10, 2016

I use Git to roll out the development version of my mod to players (essentially a continuous open beta). This by necessity includes graphics files and the DLL, with the latter changing constantly. Currently the .git folder is at around 1GB over 1300 commits, not great but still manageable. But I see your point.

It looks like you can set up free AWS instances running Windows:

Spoiler :

They only have 1GB memory and one core so obviously not the greatest environment, but it's free.

Nightinggale · Mar 10, 2016

Leoreth said:
Currently the .git folder is at around 1GB over 1300 commits, not great but still manageable. But I see your point.

Back when I moved M:C from github to sourceforge, I deleted the DLL files. It turned out that they used 200 MB and nobody would ever need them again. It quickly grows and using packed graphics would really make the size grow.

Leoreth said:
It looks like you can set up free AWS instances running Windows:

Sadly it's a Free Tier offer. After a bit of research, it turns out that some people can get it, but not everybody and it will last 12 months. After that, it will cost money.

Leoreth said:
They only have 1GB memory and one core so obviously not the greatest environment, but it's free.

1 GB memory and one CPU core sounds like my laptop and that actually gives me an idea. I'm not really using it anymore since it's from 2004 and kind of slow. I have used it for modding though and has the compiler and git installed already. It requires 5+ minutes to compile, but speed isn't the most important factor here. It's also not really suited for playing the mods, but that doesn't matter either.

All I would need to do is to do is to write a script, which does:

pull from git
compile
if a new DLL file is compiled, copy it to some public location
repeat for each folder in a certain folder

The actual compile is the only thing I don't know offhand how to do. Perhaps it is as simple as writing a bat file where I copy paste the commands MSVC use to compile. Copy to a public location could be simple as a folder set to sync with dropbox.

With that working, all it takes to have such a service running is to configure the computer to wake up at the same time each day, run script and then shut down. It actually sounds way more doable than I thought. The only non-automatic item in this process would be to add new mods/git servers.

lfgr · Mar 11, 2016

Nightinggale said:
The actual compile is the only thing I don't know offhand how to do. Perhaps it is as simple as writing a bat file where I copy paste the commands MSVC use to compile.

It is:

Code:

"C:\Windows\Microsoft.NET\Framework\v4.0.30319\MSBuild.exe" CvGameCoreDLL.sln /p:Configuration=Release

Concerning your project as a whole, I also think that is a great idea. I mostly modded FfH in the last years, using mercurial rather than git, but the next time I'm doing a Bts modcomp I'll definitely use your git base.

Nightinggale · Mar 11, 2016

lfgr said:

Naturally the forum didn't notify me of this post (email problem... again) and I ended up with a more CLI based approach. Copy pasting the build commands didn't work and I ended up "debugging" the makefile to reveal environment variables. Copy pasting the INCLUDE variable into the build script made it compile just fine. The result is a perl script, which will compile the DLL when it is executed. Now I just need to set up a loop and a git interface and it should be able to generate the newest DLL for each git repository.

I thought a bit about distribution and I think the best solution is to place the files in a separate git repository. People can then get the timestamps from git and they can download the file they want from the web interface on SF. It's also beneficial for the script as it has to interact with git anyway. It will not have to learn a new interface to be able to tell if the upload is complete.

lfgr said:
Concerning your project as a whole, I also think that is a great idea. I mostly modded FfH in the last years, using mercurial rather than git, but the next time I'm doing a Bts modcomp I'll definitely use your git base.

Mercurial is likely also fine for modding and likely just as well suited as git. The common git base is the only thing I can think of offhand, which justifies tellling people to use git rather than mercurial.

I have been thinking. Maybe it requires a bit of discipline to use this. For instance if a needs a new feature, it should ideally start as a branch from vanilla, be coded and then merged into the mod. If all features interesting for other mods are hidden inside the mod itself, it becomes a bit more tricky to merge it into other mods. Still one can always cherrypick, which mean regardless of approach, the more mods using the system, the more codechanges are available for semi-automatic merging.

Nightinggale · Mar 12, 2016

Success

I managed to write a perl script, which pulls from git and then it compiles. It loops through all directories at a certain location, meaning adding more requests for DLLs is a matter of cloning into that directory and it will not change branch, meaning once the branch is set, it stays on that branch. That's pretty much how it should be

It's not particularly fast though. It took 6 minutes and 50 seconds to get just one DLL file. Add download and upload, it can handle around 150 DLL files if it works 24 hours/day. Horribly slow compared to what other computers can do, but it should be way more than what we need.

I still need some upload/distribution system. The git idea seems difficult because I have problems getting the script to use a login with git. It doesn't have to use a login if it just use git as read-only, but uploading DLL files is a different story. I think I will go with the dropbox sync approach. It seems like all I would have to do is to just copy a DLL file and everything else will be automatic and the script will not have to log in anywhere.

EDIT:
Now the script starts at login and the computer logs in automatically without human interaction. Next step is to schedule power on daily and.... the bios doesn't support that :cry:

It looks like they didn't intend the laptop to run as a server or server like device. However if pressing the power button is all I need to do, then pressing it randomly once or twice a day would be doable. It's less great than I planned, but it's still doable.

AbsintheRed · Mar 13, 2016

Nightinggale said:
It looks like they didn't intend the laptop to run as a server or server like device.

Those bastards... it might even seem like that they didn't thought of the possibilty that your laptop ends up in a .dll building server for Civ IV :eek:

Seriously though, wonderful job! Seeing all the progress you make is encouraging

Leoreth · Mar 13, 2016

Impressive!

Nightinggale · Mar 13, 2016

I finished the compile "server", added dropbox and added a link to the first post.

The design is simple. There are two text files. Those contains server URL and what should be compiled from each server. The idea is that not only is it public how each DLL file is compiled, it should also be editable even when the compile computer is turned off.

The system allows reusing a repository for multiple DLL files, like a Release and an Assert DLL. It can switch branch as well, but that will cause recompilations even when nothing changed. This could be fixable if it turns into a problem.

The concept is that the computer logs in automatically and the script runs at login. It waits 2.5 minutes for DropBox to sync the txt files. It's likely way too much, but I want to be sure to catch all server changes. It then loops everything from mods.txt and provides the zip files. When finished, it waits 10 minutes for sync and then it shuts down. All I have to do is press the power button and the rest is automatic. If it would start once every 24 hours as I originally planned, then it would compile automatically day after day without human interaction.

Now if only I could edit the txt files directly in the browser, then it would be perfect.

Oh and perfection would likely require modders to actually sign up their mods to be build this way. The only DLL, which is actively being build with this system is the develop version of Medieval Conquest. Yeah, I designed it to handle both Colonization and BTS. More specially I hardcoded it for the file layout in M:C and the common git base. In theory it could work on any mod using makefile 2.5, but right now the path to the Makefile is hardcoded.

AbsintheRed said:
Those bastards... it might even seem like that they didn't thought of the possibilty that your laptop ends up in a .dll building server for Civ IV

Yeah it's amazing a 2004 computer wasn't designed to fit the SDK needs for a 2005 game, which was SDK modable from 2006.

Lib.Spi't · Mar 13, 2016

People just don't envision the future like they used to!

Nightinggale · Mar 14, 2016

I added a branch called "All_ModComps". As the name indicates, it's just a merge of all ModComps (2 so far). The DLL is automatically compiled when the source is updated and the download link is in the first post. The idea is that as long as long as ModComps does nothing unless activated in XML, I might as well compile one DLL for all. It makes it easier for me and modders will not have to pick the one feature they like if they want multiple.

keldath · Mar 25, 2016

hey man, so im in need of assistance,

what do i need to do to get my files up on git and do all the stuff you wrote on the first post ?

Nightinggale's Help Center and ModComp factory

Deity

LivE LonG AnD PrOsPeR

Arbiter of the Sword

Deity

Bofurin

Rex Omnium Imperarium

Deity

Bofurin

Deity

Bofurin

Attachments

Deity

Emperor

Deity

Deity

Deity

Bofurin

Deity

Overlord of the Wasteland

Deity

LivE LonG AnD PrOsPeR

Similar threads