1. We have added a Gift Upgrades feature that allows you to gift an account upgrade to another member, just in time for the holiday season. You can see the gift option when going to the Account Upgrades screen, or on any user profile screen.
    Dismiss Notice

Scripting a build of the DLL!

Discussion in 'Civ4 - Modding Tutorials & Reference' started by TC01, May 21, 2016.

  1. TC01

    TC01 Deity

    Joined:
    Jun 28, 2009
    Messages:
    2,216
    Location:
    Irregularly Online
    For a while now, I've wanted the ability to automatically fire off builds of the DLL whenever someone pushes commits to it's repository. As of today I'm pleased to say I've accomplished this, so I figured I'd post a brief tutorial for anyone else who might be interested in doing the same.

    Proof that this works for FF+. :)

    What's this for?

    Final Frontier Plus's DLL is managed by a git repository, stored on github. It would be really nice if, every time someone pushes a commit to the repository or opens a pull request, something somewhere automatically attempted to build the DLL. (Ideally, it would be tested somehow, but let's not get carried away here). If it succeeds, you can download a DLL and not have to build it yourself!

    This sort of thing is very easy to set up these days, with technology like Travis CI and similar services that provide github integration. But, for me at least, the challenges of doing that with the Civ 4 DLL are really twofold:

    1. I'm a Linux programmer. Outside of my Civ 4 modding work, I don't really do Windows development at all and have no idea how it works.
    2. The Civ 4 DLL compilation process depends on some very obsolete versions of things, including the VC++ 2003 toolkit, so even once I figure out how to make the Windows build systems work, I'll have to figure out how to install them.

    Okay, so how does this work?

    This tutorial assumes you already know how to build the DLL and have stored your sources in a git repository that's structured similarly to Final Frontier Plus's; crucially, you include everything needed to build the DLL in the repository.

    First, you log into AppVeyor and link it to your github account (assuming you're using github; it can be linked against other repository hosting as well). AppVeyor is a Windows CI (Continuous Integration) service that supports creating automated builds of Windows projects.

    Once you log in, you can select the "New Project" provider and then navigate to the correct github repository.

    If you want, you can just hit the Build button to see what happens! I'll walk you through the problems I ran into and had to fix; you will probably encounter at least some of them.

    Initial Configuration

    AppVeyor can be configured through their interactive online GUI or by writing a config file and sticking it in your repository. Not really sure what I was doing, I chose to use the online GUI, so this will explain that. I made some initial configuration tweaks first:

    Under "Settings > Build", I had to fill in "Visual Studio solution or project file" to point to my project solution file; I just entered "CvGameCoreDLL.vcxproj" since that file was in the root directory inside the repository. Otherwise, AppVeyor complained that multiple project solution files were available and was not sure what to build.

    Under "Notifications", I added a notification of the form "Github Pull Request", so that the AppVeyor bot would comment on PRs about the status of their builds.

    Installing Dependencies

    We need to install the dependencies listed here on the AppVeyor virtual machine that will actually be doing the compilation. Fortunately, Kael is generous enough to host downloads of the VC++ 2003 Toolkit and the required libraries, so it was just a matter of figuring out how to script the installation of the toolkit (since the libraries just need to be downloaded into a directory).

    That took some doing. With the help of this page I was able to script the installation of the VC toolkit. Unfortunately, I couldn't stop the installer from forking during the installation, so I put a "sleep for 2 minutes" into the script between the installation and the download of the library files (which get put into a directory generated by the installer, so it has to finish first).

    Code:
    Invoke-WebRequest http://kael.civfanatics.net/files/VCToolkitSetup.exe -OutFile ${env:TEMP}\VCToolkitSetup.exe
    & ${env:TEMP}\VCToolkitSetup.exe /s /v"/qb"
    Start-Sleep -s 120
    Invoke-WebRequest http://kael.civfanatics.net/files/msvcrt.lib -OutFile "C:\Program Files (x86)\Microsoft Visual C++ Toolkit 2003\lib\msvcrt.lib"
    Invoke-WebRequest http://kael.civfanatics.net/files/msvcrtd.lib -OutFile "C:\Program Files (x86)\Microsoft Visual C++ Toolkit 2003\lib\msvcrtd.lib"
    Invoke-WebRequest http://kael.civfanatics.net/files/msvcprt.lib -OutFile "C:\Program Files (x86)\Microsoft Visual C++ Toolkit 2003\lib\msvcprt.lib"
    
    Copy the above script into "Environment > Init Script" (toggle the init script to "PS"; this is Windows PowerShell), and hit Save.

    You'll notice we don't install the Windows Platform SDK. Fortunately for us, it appears to already be installed as part of the stock build system on these virtual machines! Thus the above is all that's needed.

    afxres.h and CvGameCoreDLL.rc

    If your DLL build includes CvGameCoreDLL.rc, you may run into problems, because the MFC libraries won't be installed on the builder and CvGameCoreDLL.rc has a dependency on a header stored in these libraries-- specifically, it contains the line:

    Code:
    #include "afxres.h"
    

    The internet suggested I could probably just change the include in CvGameCoreDLL.rc to read:

    Code:
    #include "windows.h"
    
    ...so, I did that, pushed the change, and the RC information was still put on the DLL when it finished building, so I'm going to assume this is fine.

    Build artifacts

    You're probably going to want to be able to get at the compiled DLL. You can do this from the "Build Artifacts" tab in the AppVeyor settings menu. If you configure things the way I have, it looks like the Debug configuration gets built by default (which may be what you want, if you're testing builds you or other people are pushing to your repository). So, if you want to register the debug DLL as a build artifact:

    Hit the "add artifact" button, as the relative path, enter "Debug\CvGameCoreDLL.dll" (obviously, you can adjust this for Release if you want the release target instead), and leave the other two fields blank. Then hit Save.

    Give it a try!

    At this point, try building again! If all goes well, your build should succeed and you should be able to download the built DLL from the Artifacts tab.

    I've only tested this with a single setup, and that worked after making all of the above changes. I think the FF+ DLL is relatively standard, as DLLs go, so I don't see any reason why this shouldn't work for other projects.
     
  2. DarkLunaPhantom

    DarkLunaPhantom Prince Supporter

    Joined:
    Feb 4, 2013
    Messages:
    422
    I agree that this seems much much easier than one might expect. (Although, I see that it failed the first thirteen times. :D )
     
  3. TC01

    TC01 Deity

    Joined:
    Jun 28, 2009
    Messages:
    2,216
    Location:
    Irregularly Online
    Heh, yep. It took some trial and error to get this working.
     
  4. Leoreth

    Leoreth 心の怪盗団 Moderator

    Joined:
    Aug 23, 2009
    Messages:
    33,585
    Gender:
    Male
    Location:
    Leblanc
    This is a godsend, I've been thinking about this a lot but didn't think I would get very far for the same reasons you mentioned.

    I'll definitely give it a try for my mod, this will especially help with community contributions. Thanks a lot!
     
  5. TC01

    TC01 Deity

    Joined:
    Jun 28, 2009
    Messages:
    2,216
    Location:
    Irregularly Online
    No problem! Let me know if you run into any issues, I'd be happy to help debug (and update this tutorial accordingly). :)
     
  6. Leoreth

    Leoreth 心の怪盗団 Moderator

    Joined:
    Aug 23, 2009
    Messages:
    33,585
    Gender:
    Male
    Location:
    Leblanc
    Heh, I just got things to work earlier today. Most of the issues I ran into had to do with setting up my repo so that I could use the configuration locally and on the server.

    One thing I still have to solve is that the DLL is included in my repo. While that's not the conventional way to separate source and artifact in CICD, I think it's better here because it gives inexperienced users easy access to a working up-to-date mod (I treat the develop branch as an open beta basically). So that leaves the question of how to get the server to push the new DLL back to develop if a pull request is merged, I'm currently trying to figure that out.
     
  7. TC01

    TC01 Deity

    Joined:
    Jun 28, 2009
    Messages:
    2,216
    Location:
    Irregularly Online
    Nice, glad it worked!

    Maybe this documentation will be helpful for you if you haven't already found it?
     
  8. Leoreth

    Leoreth 心の怪盗団 Moderator

    Joined:
    Aug 23, 2009
    Messages:
    33,585
    Gender:
    Male
    Location:
    Leblanc
    Yeah I found it, but haven't had time to play with it yet, so I don't know if it answers all my questions. It's more about which Git actions would trigger a build and push back into develop (e.g. I want merge commits to trigger a new DLL build but not pushing to develop).
     
  9. Nightinggale

    Nightinggale Deity

    Joined:
    Feb 2, 2009
    Messages:
    4,308
    This looks rather interesting and I will certainly investigate it closer at some point. Currently I have written a perl script to pull from git, compile and then place in a public dropbox. The problem with that approach is that it only executes on startup and when I manually request it. My big concern by moving to this is the fact that I use submodules, which makes it less strait forward to set up.

    I use sourceforge because unlike github there is no data limit on free projects as long as they are publicly available. Github had a 1 GB limit the last time I checked. Appveyor doesn't mention SF, but I did a bit of research and found a guide, which I feel should belong in this thread. I haven't actually tested it yet though. https://sorbusaria.wordpress.com/2016/12/26/how-to-deploy-from-appveyor-to-sourceforge/
     
    AbsintheRed likes this.

Share This Page