Mods and Civ4 - A Brief History of Time

Brad Oliver

Civ3/4 Mac programmer
Joined
Jan 9, 2002
Messages
843
Location
Glendale, AZ
One of the most common niggles with Mac Civ4 has been that Aspyr has never released an SDK for the Mac, which precludes building mods that require new game DLLs.

We've done a lot of work recently on this particular issue and I'm here to say that there's good news and bad news. ;)

The bad news is that it's considerably harder than we were expecting. The good news is that I think we've uncovered an interesting hack that can, in theory, allow it to work.

The bad news is that a certain upcoming game will use an unorthodox hack to support mod DLLs, but you probably won't notice unless you look closely. ;)

The good news is that once we get DLLs working in true plug-in form, we'll be patching all the relevant Mac Civ4 titles to make it work right.

If you want the nitty-gritty, the issue we ran into was that Civ4's DLL exports C++ classes through the DLL interface. From a technical standpoint, it means that Win32 imports and exports the vtable for the class, and does some magic so that non-virtual methods exported from a DLL are present in the vtable (which would otherwise only contain virtual methods). This is sadly not so with gcc, so we had to do some experimenting.

We first discovered that making all methods in the exported classes explicitly virtual works for some cases, but falls down for operator methods (e.g. operator =, the copy constructor, etc). It also fails for cases when the mod DLL's class contains extra methods or members that the importing template in the main Civ4 app is not expecting.

Needless to say, we've gone back and forth on this a few times, but I think we have a solution that will finally work, although it hasn't been put to the full test yet. We've already frozen the code for a certain app (*cough*) so we're holding off on further experimenting until it hits the wild so as not to delay it any longer - thus the "unorthodox hack" I mentioned above. I hope you all understand and bear with us while we regain our footing on Mac Civ4 once again. :)
 
Wow! Well done Aspyr, and thanks a million, Brad, for lifting the kimono.

It never looked as though it would be easy, and Aspyr's previous history of Civ modder support was not a good omen for the chances of you guys investing this much effort.

I am VERY impressed! Please, bring it on :goodjob:

PS. Sorry to be slow - it's early in the morning for me, but will we be able to build custom DLLs in Xcode, and will Aspyr give us the headers and/or libraries needed to do this? If so, will DLLs build from the same sources as the Windows DLLs?
 
PS. Sorry to be slow - it's early in the morning for me, but will we be able to build custom DLLs in Xcode, and will Aspyr give us the headers and/or libraries needed to do this? If so, will DLLs build from the same sources as the Windows DLLs?

If our current hack idea pans out, yes, although we'll surely have to make some changes to the source which I will make sure are well-indicated in our template SDK. This also means that you'll probably have to apply a series of hacks to any PC DLLs, not the least of which will be marking certain C++ class methods as virtual.

Getting *that* solution into your hands is a good few months away though, as we'll have to make sure it's finally feasible and roll it out in a series of patches for the 3 apps.
 
thank god, I was so close to breaking down and buying windoze versions and playing in parallels just to play BTS....sweetness.
 
Fantastic news. I think there is at least a small community of Mac users who mod (and thus sometimes play) Civ on the Windows side who would love the chance to help port some of the best Windows dll mods to the Mac side. I would imagine that the less complicated a mod, the easier to port. A series of hacks would be a small price to pay not to have to run Basecamp...
 
SDK = Software Development Kit
DLL = Dynamically Linked Library

Basically, the SDK is what you use to make changes to the game code; those changes are then implemented as a DLL file.
 
Hey Brad, how is the hack coming along?
I know I sound impatient and that this isn't going to be out for a while (if ever), but I was wondering if you have had any interesting breakthroughs (and/or roadblocks.)

Haven't looked into it more yet as we've had other more pressing issues to deal with first. We'll revisit it after BtS is in your hands.
 
Okay, I have to ask again, how is this coming along now? I know I'm impatient, but I do have BtS 3.19 in my hands, so thought I had better bring this subject up one more time before Brad leaves.
Python is enough to keep me busy for now, but I'm sure I'll want to move up to the Big Leagues some day. :D
 
Don't forget that our only source of information on this subject so far has probably dried up. Brad Oliver is leaving Aspyr - he may already have left. No one else from Aspyr ever posts here.
 
Don't forget that our only source of information on this subject so far has probably dried up. Brad Oliver is leaving Aspyr - he may already have left. No one else from Aspyr ever posts here.

This Friday (Sept. 11) is my last day.

Here's what I can tell you: progress is slow, but somewhat steady. One of our engineers has written either a beautiful or hideous hack to work around the main issue[1], but it's not 100% solid yet, and he's had a hard time tracking down the remaining issues. I don't know that we can reliably give a timeframe.

[1] The main issue is that the main game imports several (think thousands) of C++ classes from the DLL. As I understand it, gcc4 "helpfully" does some vtable alignment optimizations if one of these classes in the DLL contains either more data members or more (unexported) methods than are present in the same class in the main app, causing the main app to either jump to garbage routines or sometimes reference garbage data on these altered classes, since the main app does not know that the alignments have changed. Visual Studio does not do these sorts of alignment optimizations, so they are imported in a way that works fine.

Andy has written a Perl script that attempts to create wrapper classes for the DLLs that contain just the classes and members that are to be imported, but since there are thousands, tracking down the last few issues is like looking for a needle in a haystack. It's definitely been a far bigger deal than either of us anticipated. I expect, given his commitments on other projects, a solution is some time away.
 
[1] The main issue is that the main game imports several (think thousands) of C++ classes from the DLL. As I understand it, gcc4 "helpfully" does some vtable alignment optimizations if one of these classes in the DLL contains either more data members or more (unexported) methods than are present in the same class in the main app, causing the main app to either jump to garbage routines or sometimes reference garbage data on these altered classes, since the main app does not know that the alignments have changed. Visual Studio does not do these sorts of alignment optimizations, so they are imported in a way that works fine.

Andy has written a Perl script that attempts to create wrapper classes for the DLLs that contain just the classes and members that are to be imported, but since there are thousands, tracking down the last few issues is like looking for a needle in a haystack. It's definitely been a far bigger deal than either of us anticipated. I expect, given his commitments on other projects, a solution is some time away.

did a search on internet.. and found this (in http://man.he.net/man1/gcc-4 )
-fvisibility=default|internal|hidden|protected
Set the default ELF image symbol visibility to the specified
option---all symbols will be marked with this unless overridden
within the code. Using this feature can very substantially improve
linking and load times of shared object libraries, produce more
optimized code, provide near-perfect API export and prevent symbol
clashes. It is strongly recommended that you use this in any
shared objects you distribute.

Despite the nomenclature, "default" always means public ie;
available to be linked against from outside the shared object.
"protected" and "internal" are pretty useless in real-world usage
so the only other commonly used option will be "hidden". The
default if -fvisibility isn't specified is "default", i.e., make
every symbol public---this causes the same behavior as previous
versions of GCC.

A good explanation of the benefits offered by ensuring ELF symbols
have the correct visibility is given by "How To Write Shared
Libraries" by Ulrich Drepper (which can be found at
<http://people.redhat.com/~drepper/>)---however a superior solution
made possible by this option to marking things hidden when the
default is public is to make the default hidden and mark things
public. This is the norm with DLL's on Windows and with
-fvisibility=hidden and "__attribute__ ((visibility("default")))"
instead of "__declspec(dllexport)" you get almost identical
semantics with identical syntax. This is a great boon to those
working with cross-platform projects.

For those adding visibility support to existing code, you may find
#pragma GCC visibility of use. This works by you enclosing the
declarations you wish to set visibility for with (for example)
#pragma GCC visibility push(hidden) and #pragma GCC visibility pop.
Bear in mind that symbol visibility should be viewed as part of the
API interface contract and thus all new code should always specify
visibility when it is not the default ie; declarations only for use
within the local DSO should always be marked explicitly as hidden
as so to avoid PLT indirection overheads---making this abundantly
clear also aids readability and self-documentation of the code.
Note that due to ISO C++ specification requirements, operator new
and operator delete must always be of default visibility.

An overview of these techniques, their benefits and how to use them
is at <http://gcc.gnu.org/wiki/Visibility>.

could this contain the solution to the problem?
 
could this contain the solution to the problem?

Vtable alignment optimizations in C++ classes (as passed through a DLL interface) is the main issue, specifically when a mod adds methods or members to the C++ classes that pass through the DLL interface which the main app does not know about.

We do also have changes to deal with visibility, but those are more or less resolved at this point. (Never say never until we're bug-free, of course.)
 
Back
Top Bottom