[General idea] Move from SVN to GitHub to ease the development

[dbkblk]

Hi,

It's been a while that i think this mod need to move on Github to ease the development.
Actually, the use of svn is nice but it have some limitations:
- This is not easy to contribute. We have to upload the modifications we made and 45° has to merge it. It is centralized.
- The mod has no issue tracker.

As it does not sound essential to move on, i think this would have many advantages over the current setup:
- Github make development really easy.
- Github has a web interface to easy review code, make commentaries for devs, make translation without needing tools.
- It has right managment. 45° could still be the chief and select the wanted contributors.
- It have an issue tracker, making it easier for users to report bugs and easier for devs to track the lasting one.
- It have some really nice git gui out there.
- It is easy to download the latest rev (for ex, this test link would always be the same: https://github.com/dbkblk/CMP_fr_FR/archive/master.zip)
- It is easy to create branches to work on a separate additions then merge it into trunk.
- The current SVN could be imported directly into github.
- It is easy to make a frontpage or a wiki for the mod.

I think it should be a good step to move on this tool as it could encourage users like me, Maska or Sogroon (for ex.) to review the files and make commentaries.

I've uploaded an example with the french translation of CivMegaPack here:
https://github.com/dbkblk/CMP_fr_FR

EDIT: Here is a link to understand how it works:
http://try.github.io/levels/1/challenges/1

And this post is the perfect example of what could be considerably made easier:
http://forums.civfanatics.com/showpost.php?p=13159224&postcount=111

What do you think about it ?

 

[Combat Wombat]

I'd vote for bitbucket

 

[45°38'N-13°47'E]

Never used GitHub, but there's a thing to consider. I'm not the owner of the mod. I'm a temporary Mod Manager. This mod still belongs to Afforess, although he allowed me and other people to modify it and take care of it.
From my point of view, it could maybe be easier to use GitHub, but I would have to learn everything from scratch on how to use it and I prefer spending my time in fixing the mod rather than learning how to use a new platform. And, as I said, Admin or Owner rights should go to Afforess, not to me. He's the one who should then decide what to do and who can do what.

 

[dbkblk]

And, as I said, Admin or Owner rights should go to Afforess, not to me. He's the one who should then decide what to do and who can do what.

I did not see that when analyzing the situation. However, Afforess may like this idea ?

From my point of view, it could maybe be easier to use GitHub, but I would have to learn everything from scratch on how to use it and I prefer spending my time in fixing the mod rather than learning how to use a new platform.

As smart as you are, you may need only 1 or 2 hours to understand the way it works.
I've found a website that contains a tutorial and explains the benefits and the workflows of GIT.
https://www.atlassian.com/git/

 

[dbkblk]

I'd vote for bitbucket

Bitbucket seems nice but it is limited as it is not free for more than 5 users.

 

[45°38'N-13°47'E]

I did not see that when analyzing the situation. However, Afforess may like this idea ?

He's following the forum and chimes in from time to time. I think he's currently engaged in a game with one of the latest revisions.

 

[Afforess]

I would totally support a move to github. My primary concern is that git does not always handle really large files (I am thinking the 1gb art fpks) very well. I would be happy to be the owner of the github account, and add the current users as contributors (with full read/write priveleges), I already have an active github account.

https://github.com/Afforess

Also, Github does not allow large files in their repository (files > 100mb). The overall repository size can be up to 1gb, but made up of smaller files. My quick analysis of RAND currently shows it to 1.75GB. Much larger than allowed. So we need to consider some options:

1.) Go with Github. Don't keep the movies & FPK files in the repo. Host those somewhere separately.

2.) Look for public Github alternatives (Bitbucket?). Bitbucket also discouarges large repos: https://confluence.atlassian.com/pages/viewpage.action?pageId=273877699

3.) Self-hosted git repository (no size limitations, but costs $). We could have something like a self-hosted Gitlab (Github clone) for personal use - https://www.gitlab.com/gitlab-ce/

4.) Stick with the current SVN

My 2 cents. No matter what git solution you pick, large files are a problem. Even with a self-hosted setup, git is slow with big binaries. Git was made for source code and text. This might be a good time to reconsider the current packaging/installer setup too, as I imagine this will break the current setup now. Despite that, I love git. It is superior to SVN in pretty much every other way. Significantly so.

Finally, I previously offered to donate server space for this project a while back. The offer still stands. I have access to very fast hosts (200mb/s up & down) and can easily host all the large files, or even a gitlab install.

 

[45°38'N-13°47'E]

Ok, looking at the problem from a user's point of view, I think hosting movies and fpk separately isn't a good solution. It makes installing the mod more complicated. So I would discard solutions that can't handle big files. And if we move to other solutions, we'll definitely need a new installer because Mantriur nightly build won't work anymore. But a new installer is certainly beyond my ability.
So while I certainly appreciate an easier approach for programmers (using Tortoise SVN is a pain), I guess end users ease of access has priority for the moment.
If we can make player's access and installation easy, I have no problem with moving to a new platform.

Edit: I would also discard every solution with a cost. I have no problem in paying a fee to have a good platform, but should the mod be managed by someone else in the future, I don't want to limit anyone because of the costs (although I think they're really affordable).

 

[dbkblk]

I totally agree with your analysis, as i didn't see the big file problematic neither. I'm trying to think about a solution which would be free and as easy as now for the end-user (which is the n°1 rule).

The nightly build isn't really necessary. A workaround should be to create a small launcher which check the repo and update / install the mod. I think i can handle that in C++. However, if the fpk files are oftenly modified, there is really no point in doing that.
Is there frequent changes in the big fpk files ?

EDIT: I've checked the fpk files. With the exception of "Vokarya.fpk", the others files have not been modified since 2012 or 2010.

 

[BlindKitty]

Well, there is one other idea that might work, but it would probably need someone more experienced than me to actually set up, if it's even feasible. It goes like that: there is a certain utility tool, named Jenkins, meant for continuous integration; it works well with Git. We could set up Git with just source code/text files, moving *.fpk files to another repository. Whoever is currently coding, would have to merge those two parts of mod on their own, to be able to commit to Git, but we all agree that it is still better idea than SVN, I think. And Jenkins could handle downloading Git code, art assets, merging them together and putting up complete mod for downloading with no interaction, given starting config. It is quite small on its own (like, 200 MB tops, I think, including plugins and such), and works well on both Linux and Windows (or so I heard, I'm only using it on Linux). If Afforess servers could take that, we could put there art assets, Jenkins, and ready-to-download mod.
Jenkins builds can be triggered by timer, by hand, or by commit, and it is easy to set up a few jobs - Stable build triggered by hand, when a version is deemed stable, nightly build for those feeling adventurous, and whatever else comes to mind, like version with MegaCivPack pre-packed into mod.
What to you think of that? It would be somewhat more complicated for the back-end users, likely 45* (I could keep an eye on Jenkins or something, but I don't really have much free time or relevant experience), especially during setup (although I could help there somewhat), but I would keep things simple for front-end users, who would be happy to just download the stable version every once in a while.

 

[dbkblk]

That could be a solution, i'll look into that.

1) Another workflow could be to keep using the same infrastructure, but use externals with subversion. This is not really clear in my mind at the moment but basically, the workflow could be:
- Use github for all the files except FPK and Sound. Work with branch, then merge it to master branch.
- Link the github server to svn with externals (that's just a "svn update" in the end).
- Upload the modified fpk or sounds, if any, on the svn server.
- Commit on the svn, as usual

The drawback with that is:
- It is easier to collaborate but that add layers to the dev process, which is not really nice

2) Another idea to dig in:
The mod have a stable and a development version.
- The stable version is just packed then diffused, so this is not a problem to pack it manually, as it happens just from time to time. I mean, we just have to grab the code and the fpk and upload it somewhere (on sourceforge ?).
- The devel version is for advanced users. How hard it is to grab a separate "graphical pack" and put it on the same folder for an advanced user ? Moreover, if there are clear instructions, that would be easy. It may also be script or included in a launcher.
Git have a file which is called .gitignore where you can put ignored files and folders. You can ignore *.fpk and Usersettings/ files and folders and just pull the update as you do in svn.
The only problem that need to be resolved is the update of fpk, as Vokarya do it from time to time. Considering that Vokarya.fpk is above the 100mb limit of github, this is not a real problem.

 

[dbkblk]

@45' and Vok: Could you describe briefly how is your workflow while updating the game ? Do you work directly in the mod folder or do you have a work folder and you copy the files into the mod folder ? etc.

 

[45°38'N-13°47'E]

@45' and Vok: Could you describe briefly how is your workflow while updating the game ? Do you work directly in the mod folder or do you have a work folder and you copy the files into the mod folder ? etc.

Basically I have an SVN folder on my pc where there's a copy of the current SVN version and this is the one I update and that I use to upload changes. I actually have many copies of the dll folder that I use for experiments; when I'm ready and all the changes I've done are working, I overwrite all my modified files from my experimental folders into that SVN folder and upload it via Tortoise SVN. That's it, more or less.

 

[Afforess]

I'm just not a SVN fan and going to experiment with mirroring the SVN (minus fpk's + large files) on Github. Git supports importing SVN repositories and can be used to "mirror" what the SVN has. At worst, I will just use this for personal use.

 

[dbkblk]

I thought about a pretty easy workflow for github:
- Make a "Base pack" with all the fpk, sounds, art hosted on sourceforge
- Put all the code and small binaries on github (it will be under 800mb)
- Put a ini file in which we write the base package version and a download link
- Use my launcher to grab the new files from github and a incremential base package update when the ini file specifiy there is a new version
- We can work with a master branch where we put the new revision on and a devel branch where we just update and experiment.
- When a fpk need to be updated, just pack it, upload it on sourceforge, then update the ini file with a new version number and a download link, so the launcher will know there's an update. If the file is named, for example, "BASE_PACK_UPDATE_694.7z", it will be easy to know that it correspond to the 694 version. This way, the version control process, still works !

BTW, i have to find out this dll problem.

 

[dbkblk]

@Afforess: I've uploaded r692 on a github repo, just to try it out. Here
I will go one week on holiday, but after i'll try to write a script which put all excluded binaries in the Assets/ folder to another folder, then to pack them as a base binary package.
The base pack will be uploaded on sourceforge, then the eventual package update.
Then, i'll need to write another script which detect which file have been modified since the base binary package and copy them to another folder.

This flow will allow to work on git. On the first installation, it will clone the repo and grab the base binary automatically with the launcher. When an "update binary package" is published, the launcher will grab it at the same time.
It is easy to write a file which contain all binaries update links and versions directly on the git repo.

I'm not sure to be clear, i need some sleep

The .gitignore is:

# Folders
Autolog/
UserSettings/*.ini
Assets/Modules/Custom_Civilizations_MCP/
Assets/Sounds/

# Checker components
checker/
platforms/
temp/

# Binaries
*.dll
*.exe
*.FPK
*.bik
*.dds
*.tga
*.jar

# SVN components
*.orig
*.svn/