1. We have added a Gift Upgrades feature that allows you to gift an account upgrade to another member, just in time for the holiday season. You can see the gift option when going to the Account Upgrades screen, or on any user profile screen.
    Dismiss Notice

Plan to get the source code for the exe

Discussion in 'Civ4 - Creation & Customization' started by Nightinggale, Dec 23, 2016.

  1. Nightinggale

    Nightinggale Deity

    Joined:
    Feb 2, 2009
    Messages:
    4,576
    A question came up in the quick modding questions thread and since the code in question happens to be in the exe, this line showed up:
    This sparked multiple posts about the topic, which in turn resulted in this thread.

    Why do we want the source code?

    Being able to mode everything
    There are some code in the exe, which we would like to mod. Most noteworthy is the Main Menu, including all the "submenus" (new game, custom game etc) and the diplomacy screen.

    Bugfree and stable mods
    Accessing everything will not only allow modding, but also bugfixing. This is not only bugs in the exe itself, but also bugs in the DLL caused by assuming something about the exe, which seems correct at the time, but isn't. Also debugging anything where the exe is involved will also be a whole lot easier.

    There are also some undocumented limitations, such as some enum values are hardcoded in the exe. More than once have modders added something, which results in an offset of one, hence broken code.

    Better performance
    The exe has inefficient code when communicating with the DLL, like it loops and calls a DLL function to get a bool to tell if it's a certain enum of not, meaning we often run into unneeded loops.
    Compiling the exe and dll into one file will not only allow just using the enums, it will also allow some compiler/linker optimization, which is not possible today because the dll can't optimize for something in the exe.

    Modern tools
    Right now we are using a compiler from 2003 with boost and python from 2004. This is problematic because there is no support for those. For instance I looked at boost threads and it turns out it was changed significantly in 2007 (for the much better), meaning getting info on how to do stuff prior to 2007 is near impossible today. Also the dll is unavailable today meaning you have to compile yourself, which turned into a nightmare.

    Using a newer compiler also means more advanced algorithms for optimization and ability to optimize for modern hardware. Modern CPUs have more registers and more instructions, which sits idle when running old code. Just compiling something, which is intended precisely for the CPU in question will make the code execute faster.

    Modern C++
    When looking up how to do stuff in C++, what you find online today is C++14 or perhaps C++11. We use C++03, which doesn't support all the cool stuff people talk about today. Using a modern compiler will allow usage of C++14.

    64 bit application
    32 bit applications are limited to 4 GB of memory. 64 bit has no realistically reachable limit in address space. This allows bigger mods.

    IPv6
    The game uses IPv4 for multiplayer, which is in itself ok. The problem is that we have run out of IPv4 addresses and more and more people can't get a public IPv4. IPv6 has more addresses than there are grains of sand on the planet, meaning everybody should be able to get one and will eventually. Adding support for this will ensure that people will not lose multiplayer. IPv4 will not be dropped, though mixing IPv4 and IPv6 in one game will likely never be supported

    Speaking of multiplayer, all players act sort of like a server. Say A hosts a game and B and C joins, C can then be refused because A can be contacted, but C can't connect directly to B. With firewalls and stuff, which can become problematic. Access to the source of the exe can allow the host to be relay between B and C if that is the case.

    Modern graphics engine
    This seems to be the biggest concern and rightfully so. Graphics engines are big and complex to develop, meaning companies who have done so keep them close to themselves. We might not be able to get the graphics engine at all.

    Luckily there is a plan B. Rewriting the graphics interface to use a freely available engine. Unity has great features for this and AFAIK should be free to use until you sell a certain amount of copies. Since we won't be selling copies at all, it should be free to use for us (must verify to be 100% sure. Didn't read everything in their terms). Using a different engine allows modern graphical engine with modern features.

    A new graphics engine will also get rid of engine limitations, such as limited to 128 (or was it 127?) rows of plots being drawn when zoomed out.

    Cross platform support
    With a switch to a cross platform engine like unity, it might be possible to compile for different platforms. We might finally be able to make mac/linux mods, which mods the DLL. This isn't top priority though, more like a possible bonus.

    Now what?
    The question is who is interested in doing anything for this and how do we do it?
     
    Last edited: Dec 24, 2016
    merijn_v1, Leoreth, 1SDAN and 2 others like this.
  2. Thunderbrd

    Thunderbrd C2C War Dog

    Joined:
    Jan 2, 2010
    Messages:
    28,363
    Gender:
    Male
    Location:
    Las Vegas
    Some of the points you make are interesting. I have learned programming purely for the modding here and therefore have NO advanced knowledge beyond what I'm limited to understanding by having examples in the codes here and left from other modders who have worked on the C2C code set. So a lot of what you're bringing up is flying way over my head but it sounds good.

    It's the limitations that drive us nuts.

    To me, if you were to:
    then you might as well just initiate a whole new game.

    I found a free to use graphic engine (both 2d and 3d platforms provided) at Garage Games and it looks pretty good. I've also heard of MOgre or something along those lines. I'm terrifically hindered by the lack of experience with setting up a game from scratch and no formal programming education.

    Koshling was passively working on a project to setup a new game engine but I'm not sure it's going to be the game I wanted to work on. But if he had the ability to work directly on every layer of the game's architecture, he'd never have had to consider taking this step. My point is that Firaxis needs to realize that there are efforts that WILL eventually pop up to circumvent the brick wall that having no EXE access is keeping us behind.

    As far as design and game theory goes, if I can work from a platform of furthering development anywhere near what CivIV provided, I can work on continual innovations for this game to my heart's content... but then if it's going to be of completely original design then we should have every (and I mean EVERY) right to make some money from it when we decide to monetize from the project.

    Never being able to have this potential in C2C is a cruelty because I just KNOW that if we ever reach the point where the full vision for the game is reached, it would become one of the most played games in history. So yeah, I see why the .exe is limited and why they'll never release it. But maybe we could write a contract with Firaxis to give a small team of us access for a shared money making opportunity in it. If we create the game Civ IV could have become and really develop out the other branch of the Civilization family tree that they stole from us by moving in the design directions chosen for Civ V+, then it would only be fair that they both Firaxis and the team that develops the mod, both share from any potential profit that might arise, and I don't see why we wouldn't cut in Civ Fanatics as well for the opportunity to have an exception granted to the 'nobody gets to make money here' policy they have as well. Just seems like it would be more beneficial for all to have that eventual end goal potential working from a complete code access platform.

    Who to approach and how to beseech them and what lawyers would be needed I have no idea.
     
  3. Nightinggale

    Nightinggale Deity

    Joined:
    Feb 2, 2009
    Messages:
    4,576
    Actually I was thinking something like releasing the source code under conditions like Call to Power 2.
    • just the source code, not commercial libraries
    • it's not allowed to distribute anything, which can make the modded game work without a vanilla game installed
    • no commercial releases
    Getting the exe source, but not the graphics engine means it will fail to compile due to missing headers. We would then have to rewrite each call to a new engine. Precisely which one is an open question at this point.

    Sure we could make a brand new game, but leaving the civ4 license behind, we would have to make something, which is clean of civ4 code and that includes the vanilla DLL code, which is the basis for all mods. In other words all mods would have to be rewritten from scratch and then what's the point? Also writing some sort of wrapper for some libraries is way less work than writing a new game engine.

    I'm not sure I support the idea of making C2C commercial. I get that you would like it to be on steam or whatever, but it's a legal nightmare. You would need every contributor to sign that they agree (good luck even figuring out who those people are) and you would need some sort of contract for distribution, which is likely much harder to get because their lawyers will need to investigate their original contracts and stuff for their distribution. I suspect this will only make the legal issue so complex that we can fail everything.
     
  4. Thunderbrd

    Thunderbrd C2C War Dog

    Joined:
    Jan 2, 2010
    Messages:
    28,363
    Gender:
    Male
    Location:
    Las Vegas
    You're probably right. One cannot help but dream, when they find true opportunities scarce in this world, that the things they've put thousands of hours of effort into that many others have spent thousands of hours enjoying, that there could be SOME means of providing a manner in which one can be a little financially supported by it all. But you make very good points about how much added complexity it presents to reach for that. I believe we'd have to admit that C2C has been built on platform after platform after platform of modding work that preceded us so there could be hundreds of modding contributors that would need to be tracked down. We have a list on our SVN access but that's for just C2C alone. Counting AND, RoM, Rev DCM, Better BtS AI, all the hundreds of individual art contributions and animations and then contributions from other games and even some copywritten material we may not even realize IS copywritten... it would only open the door to a legal nightmare.

    If we could obtain some deeper access as you suggest, with the limitations you considered, would we be answering to all the benefits you mentioned in the first portion of your proposal?
     
  5. Nightinggale

    Nightinggale Deity

    Joined:
    Feb 2, 2009
    Messages:
    4,576
    You can scratch probably. It's not the first time I have dealt with issues like this. If you want to make something for profit, you start with a small group who agrees on it in writing and closed source. I'm not aware of any open source projects, which managed to close the source and sell for profit.

    Yes, but I'm quite concerned that it will never happen. If it was to be done, we would need a strong community pressure as well as a team of skilled people to take on the task. Looks like we lack both.
     
  6. Thunderbrd

    Thunderbrd C2C War Dog

    Joined:
    Jan 2, 2010
    Messages:
    28,363
    Gender:
    Male
    Location:
    Las Vegas
    The community pressure would have been stronger a little bit ago before many modders just accepted things and moved on to Civ V and VI. I'm sure if they were still paying attention to posts in the general Civ IV forums as much, they would be chiming in. But truly the frustration of the status quo has already made the best and brightest move on to other goals so far as I've seen. This discussion topic has been brought up many many times before in the C2C threads.

    I've used modding as a means to gain some coding understanding and I've done pretty well with that I think, but I've seen the added depth professionals bring. And I think it ends up being too demotivating to work on the mods when there are so many ways one can trigger problems in the EXE and have little clue how those problems were triggered. In many ways, it becomes easier to start thinking of starting from scratch. And there's so little, outside of love for the game, to motivate.

    The best programmer I saw work on C2C gave up in irritation with the blind spots lost to the EXE when trying to implement Multi-maps.

    You said you've worked in the industry and have seen a lot... did you have a hand in the original CivIV project or know anyone that did?
     
  7. Zeta Nexus

    Zeta Nexus Deity

    Joined:
    Jan 23, 2014
    Messages:
    4,223
    Gender:
    Male
    Location:
    In a constant brainstorm...
    Very interesting topic.
    Looking forward to see how it all develops...

    BTW: Is there the slightest chance of rewrite the EXE at least partially to get som of those benefits mentioned in the opening post?
     
  8. Nightinggale

    Nightinggale Deity

    Joined:
    Feb 2, 2009
    Messages:
    4,576
    If I had that kind of connections, I would have use those rather than writing on a public forum.

    It could be done and we would get all the benefits. The problem is that it would be significantly more time consuming to follow that path. Also I can't say anything about the legal issues from doing so, at least not offhand.
     
  9. Leoreth

    Leoreth Prince of Blood Moderator

    Joined:
    Aug 23, 2009
    Messages:
    34,959
    Gender:
    Male
    Location:
    House of Hades
    Interesting discussion. Allow me to be pessimistic for a moment. This endeavour faces two separate challenges: making Firaxis care about this problem, and figuring out a way to open access to the exe source without legal issues for Firaxis.

    As it stands right now, Firaxis neither seems to care a lot about Civ4 nor about modding in general. Considering the attention and inherent support modding received in Civ4, it really hasn't been given nearly as much attention with Civ5 or Civ6. I think that reflects a different inherent attitude to modding in the current Firaxis team/leadership as compared to back when Civ4 was developed.

    Even worse, I don't think anyone there has any stake in Civ4 at all right now. Given the turn over rate in the industry, I wouldn't be surprised if no one from the original team is still employed at Firaxis. For the leadership, they have just released a new current game in the series, why would they be interested in furthering modding in an older version of it? Better wait for these modders to be frustrated and maybe move on to mod for Civ6, to increase its attractiveness and longevity.

    I find it hard to find a point of attack here. Trying to reach Firaxis through official channels will probably either be ignored or at best replied to with a statement that they are unable to release the source because of legal limitations. Honestly, I don't even think we could muster an impressive enough number of signatures for some sort of community petition to bolster this request.


    For the second point, Firaxis seems to have two major reasons not to release the exe sources:
    - licensing constraints with the Gamebryo engine and maybe other proprietary libraries used
    - restricting the ability of third parties to build an instance of Civ4 that can run without having to purchase a copy

    I don't actually think an agreement like Nightinggale mentioned from Call to Power 2 would work here. Gamebryo will hardly map one to one to an open source game engine, so replacing it will be a huge undertaking. Not sure actually if there is anyone in the Civ4 modding community who is familiar enough with graphics engines to produce a presentable result, and even then it would take a long time until that is complete and we could see the actual desired benefits of a custom exe (while a more modern graphics engine would of course be a good thing, I think we all agree it is secondary to the concerns of performance and ageing Nightinggale mentioned in the OP).

    So what is left to hope is that the license agreement Firaxis had with Gamebryo has expired or will expire at some point, as Thunderbrd suggested. That might be a first point of enquiry.

    Other than that, I think we should instead focus on getting a better exe over having the exe open sourced. Keeping the exe in closed source could alleviate many reservations Firaxis may have if they are otherwise sympathetic to our problems. Instead a select few community modders (if we're honest, there are probably only half a dozen willing and skilled enough) could be given access to that closed source under the condition of signing an NDA. This team could produce a new exe, or a series of new exes, that Firaxis could release as patches or even a Civ4 Remastered Edition or the like.

    With such an arrangement, possibilities would of course be more limited; mods wouldn't be given free reign over the exe, and more specific desires would probably remain unaddressed. However, the most pressing concerns with the exe are affecting all game mods: transitioning to 64 bit and IPv6 (didn't even consider that until now), outdated C++ and dependencies, underperformant implementation; and could be fixed generally for everyone.

    This may be unsatisfying for mods like C2C that intend to push even further against the constraints of the exe, but I think here we should consider the needs of the whole community: a more modern and more performant exe would not only benefit the whole Civ4 modding community, but all Civ4 players overall.


    Despite all these concerns, I really hope this does not fizzle out, and am willing to lend my support to whatever initiatives may come out of this. Maybe we should start with drafting a petition to have some backing to approach Firaxis with, at least to get an initial feeling as to what their stance on this is.
     
    Last edited: Jan 10, 2017
  10. Midnight-Blue766

    Midnight-Blue766 The filidh that cam frae Skye

    Joined:
    Sep 24, 2007
    Messages:
    3,452
    Location:
    Northern Equestria
    As much as I support this idea, I still remember the last time we tried petitioning releaseing Civ3's source code... :rolleyes:
     
  11. AbsintheRed

    AbsintheRed Deity

    Joined:
    Jul 27, 2009
    Messages:
    8,288
    Location:
    Szeged, Hungary
    I would be all for this!
    Having the exe open sourced seems very unlikely to me, but this might be achieved with Firaxis...
    This alone would already do wonders to the longevity of Civ IV.
     
  12. mourndraken

    mourndraken King

    Joined:
    Nov 11, 2003
    Messages:
    921
    Gender:
    Male
    Location:
    54°30′59″N, 128°35'59"W
    See if they would sell the required sources

    then try and crowd-fund . start a gofundme campaign !
     
  13. Leoreth

    Leoreth Prince of Blood Moderator

    Joined:
    Aug 23, 2009
    Messages:
    34,959
    Gender:
    Male
    Location:
    House of Hades
    I don't think they are in a legal position to do this, and even then, selling the source is clearly not in their interest.
     
  14. mourndraken

    mourndraken King

    Joined:
    Nov 11, 2003
    Messages:
    921
    Gender:
    Male
    Location:
    54°30′59″N, 128°35'59"W
    why do you think that ? While putting together a deal, never let fear of rejection stop the pitch..

    I think we should ask them and let them respond for themselves. The worst thing that could happen is they say NO/
     
  15. Leoreth

    Leoreth Prince of Blood Moderator

    Joined:
    Aug 23, 2009
    Messages:
    34,959
    Gender:
    Male
    Location:
    House of Hades
    Because of everything that has been said above, no need to repeat it.

    We are not in a negotiation position with Firaxis. The hardest part here is to get them to listen to us at all, and any agreement we might be able to reach will be basically a show of support of the community on their part. When you are dealing with a corporation, everything becomes more complicated: what we are asking may cost them lawyer hours, and maybe even developer hours. If we open up the discussion with an overblown request, we show that we are not aware of the situation Firaxis is in, which won't help us in convincing them that the proposal is constructive and will have a positive outcome for them. It will make it more likely that they will reject the request with a boilerplate PR statement or not even reply at all.

    This is not going to be a conversation. If the worst thing happens and they say no, that was it.
     
  16. mourndraken

    mourndraken King

    Joined:
    Nov 11, 2003
    Messages:
    921
    Gender:
    Male
    Location:
    54°30′59″N, 128°35'59"W

    Good luck with that.
    I doubt that you have ever prospected and initiated a proposal to a large client before. I always start the process with a courtesy call.
     
  17. Leoreth

    Leoreth Prince of Blood Moderator

    Joined:
    Aug 23, 2009
    Messages:
    34,959
    Gender:
    Male
    Location:
    House of Hades
    You are probably talking to people who are there to receive your calls.
     
  18. mourndraken

    mourndraken King

    Joined:
    Nov 11, 2003
    Messages:
    921
    Gender:
    Male
    Location:
    54°30′59″N, 128°35'59"W
    They are a company and easy to reach by phone, fax, or flight.

    now what ? I look at your project as a B2B recycle of an old product.

    turn that old clutter into new revenue. They would still retain the ownership

    I would like to list information points that support an open source CIV4
    and why they should trust you with their legacy.
     
  19. Leoreth

    Leoreth Prince of Blood Moderator

    Joined:
    Aug 23, 2009
    Messages:
    34,959
    Gender:
    Male
    Location:
    House of Hades
    I honestly have no clue what you are talking about or are trying to say.
     
  20. mourndraken

    mourndraken King

    Joined:
    Nov 11, 2003
    Messages:
    921
    Gender:
    Male
    Location:
    54°30′59″N, 128°35'59"W
    Leoreth. You don't think Firaxis should be contacted. That there is no way to get their support.

    I get it. I just don't agree.
     

Share This Page