Moding and Editor Question Thread:

[Balthasar]

I have a question; what language is all this written in? How hard is it to look deeply into - for instance, how do editor modders like Quintillus and Steph determine that they can or cannot include certain functions in the editor? Is the exe file in the same language, and does it follow the organization of the Biq in any way?

 

[Gojira54]

ELIMINATE THE 512 CITY LIMIT HERE

-Oz

I've yet to have anyone say that they have seen the crack work properly. AI does not build cities after 512 are placed, even with the crack. The human player can, and apparently you can preplace more than 512 cities on the map for a scenario, but once at least 512 cities are on the map, the AI simply will not build any more (but you can).

I've tested this out thoroughly, and have asked others to do the same, and nobody has reported anything differently.

Now, it is possible there are different builds out there and the one I have is bad, so if anyone has a version they think works I would love to have it and play with it, but until then I do not think this works at all.

 

[Quintillus]

ELIMINATE THE 512 CITY LIMIT HERE

-Oz

Gojira54 pretty much covered what I was going to say, but I had the disclaimer because in this thread (which I believe is the thread for the DL database download you linked to) several people, including Knuckles himself, mentioned having issues with the AI building cities beyond 512. But it sounds like Gojira54 knows more about this than I do.

I have a question; what language is all this written in? How hard is it to look deeply into - for instance, how do editor modders like Quintillus and Steph determine that they can or cannot include certain functions in the editor? Is the exe file in the same language, and does it follow the organization of the Biq in any way?

Well, the original Civ3 editor is written with Microsoft Visual C++ (screenshot source), so I would assume that Civ3 is also written in C++. The BIQ is largely documented at Apolyton, so determining whether something can be included in the editor is mostly a matter of figuring out if it's one of the things in that thread at Apolyton. There are a few unknowns in the BIQ format as documented at 'Poly, so occasionally someone comes up with an idea that might possibly be what one of the unknowns are. But these days it's rare to figure out one of the unknowns. Charm is an example of something that took awhile to figure out, and scientific leader toggling for scenarios (at least with 1.22) is something that was somewhat broken for awhile after I joined.

There's also a decent amount of figuring out whether something is likely a hardcoded or an arbitrary restriction. For example, upper limits that are powers of two (such as 512 cities, 8192 units, 32 civs, etc.) are likely hardcoded in - Firaxis probably used some array that of length 9 for cities, 13 for units, 5 for civs, etc. in at least some places, so these would require .exe changes (and as seen, that can be difficult even for wizards such as syn and Knuckles). But some limits in the Firaxis editor appear to be rather arbitrary. For example, it works to set maintenance negative in practice (although I'm not so sure that the AI recognizes this - I haven't tested that enough). It looks like Firaxis probably set zero maintenance as the minimum as a "common sense" limit, rather than for a technical reason.

By and large, I can't actually make head nor tails of the .exe. I studied a bit of assembly language, but it wasn't Intel x86, and even if it had been, it would likely take a long time to figure out the structure. I took a security class as well, which looked at some aspects of deconstructing .exe's, but those were fairly simple cases.

Since Civ3 was likely written in C++, there's also the factor that the .exe code, when converted back into assembly, likely would still be confusing, since the C++ compiler wouldn't always convert it into the type of assembly a human would write. RollerCoaster Tycoon, by contrast, since it was written in assembly, likely has relatively legible assembly. So, even for someone versed in assembly, Civ3 would likely be more challenging and time-consuming than RollerCoaster Tycoon or other games written in assembly language.

The .exe likely does specify the full BIQ in code at some point (after all, it has to load and save BIQs, and it has BIQ/SAV-like structures in memory while it's running), but since it's been compiled, it's easier to figure out the BIQ format by simply changing the .biq (or .sav) in a hex editor than trying to deconstruct the code.

 

[Balthasar]

Wow. Thank you Quintillus for such a detailed reply. I've wanted to know all of that stuff for a long time, and I appreciate the time you spent writing it.

Have you worked any more at adding the alternative terrains that embryodead discovered to the terrain options in your editor? (I remember that you were at one point.) As terrain modding is my favorite thing, I'm very interested in the possibilities. Does the BIQ tell you at all which folders and file names it's looking for?

 

[Quintillus]

Wow. Thank you Quintillus for such a detailed reply. I've wanted to know all of that stuff for a long time, and I appreciate the time you spent writing it.

Have you worked any more at adding the alternative terrains that embryodead discovered to the terrain options in your editor? (I remember that you were at one point.) As terrain modding is my favorite thing, I'm very interested in the possibilities. Does the BIQ tell you at all which folders and file names it's looking for?

No problem. I didn't realize it would be such a lengthy reply at first, but I guess I kept thinking of details.

I haven't worked on that yet, and to be honest, I haven't been working a whole lot on the editor recently, in part due to changing life events, and in part due to shifting interests. That is amongst the many features I'd like to add, however. When/whether it will happen... I'm not sure. There will be a version 0.82, but I can't promise a whole lot beyond that.

 

[Balthasar]

I haven't worked on that yet....in part due to shifting interests. . . . There will be a version 0.82, but I can't promise a whole lot beyond that.

I'm glad to hear that, but alarmed that your interest is waning. Please don't take it off the stovetop just yet. You're my last hope, Obi Wan. I have an immense interest in the result. If you've ever been looking for the perfect tool to do a particular task, you know what I mean.

The last question I asked, 'does a BIQ reference specific file names that it's looking for?' is important because I always had the feeling that embryodead's 'extra terrain' biq worked sporadically because the biq was looking for files that weren't there: something like LMxtgc, for instance. The same could be true for overlays, but naming conventions in the terrain folder are erratic (Dessert, for example), so it occurs to me that perhaps the biq is looking for specific file names, which would have to be written in there among the code. Tell me, is my hunch correct?

 

[Quintillus]

I'm glad to hear that, but alarmed that your interest is waning. Please don't take it off the stovetop just yet. You're my last hope, Obi Wan. I have an immense interest in the result. If you've ever been looking for the perfect tool to do a particular task, you know what I mean.

The last question I asked, 'does a BIQ reference specific file names that it's looking for?' is important because I always had the feeling that embryodead's 'extra terrain' biq worked sporadically because the biq was looking for files that weren't there: something like LMxtgc, for instance. The same could be true for overlays, but naming conventions in the terrain folder are erratic (Dessert, for example), so it occurs to me that perhaps the biq is looking for specific file names, which would have to be written in there among the code. Tell me, is my hunch correct?

The good news is, there are still times when I become highly interested in specific aspects of Civ3. But it's different from the roughly 2003-2005 and 2007-2010 periods when Civ3 was nearly always in the top 3 games I'd spend time with in any quarter (both playing and modding-related stuff). Now it's more of a case of instead of it being one of my default activities, if I'm on CFC, read something interesting, and have time to work on it, things get done in small bursts. But, I see what you mean. We're a smaller bunch here than we used to be. And I think I can smooth out the terrain features some more.

I'd thought about the last question, but simply forgot to add in a reply before hitting submit. But indeed, the BIQ does look for specific file names, and in specific locations, in a specific order. Are you specifically referring to jungle/marsh/volcano terrains that embroydead mentions used the base graphics (except with jungle volcanoes)? My guess is that probably isn't possible, and I suspect embryodead would have found out how if it were (although if it is some weird typo thing, I might be able to figure that out).

Of note, on page 2 of that thread, Virote Considon mentioned tundra jungles and such. I remember playing around with that a few months ago and having mixed results - I think I might have posted about it on CFC, but I can't remember if I did or not. Regardless, that sort of thing is something that I feel should theoretically be possible, but will require more research.

Now I kind of wish I had my main machine available so I could play around with some of this more easily - still might be able to on my laptop, though...

Edit: Took a look around the .exe, and I can only find one references to the marsh.pcx file, and for volcanoes, there are Volcanos-snow.pcx, Volcanos jungles.pcx, Volcanoes forest.pcx, and Volcanoes.pcx. I can't recall off the top of my head how/if snow volcanoes work? But it looks like this is probably all the options... LXTGC.pcx, which embryodead calls out for LM Tundra, is specifically mentioned.

Edit 2: I could perhaps make a list of all the terrain-related PCX files mentioned? That's likely to be all the ones that Civ3 can use, and there might be a couple of unexpected ones. Aside from ones related to the city view (after clicking the eye, when in a non-scenario), there appear to be a relatively limited amount of them.

 

[Balthasar]

Are you specifically referring to jungle/marsh/volcano terrains that embroydead mentions used the base graphics (except with jungle volcanoes)? Of note, on page 2 of that thread, Virote Considon mentioned tundra jungles and such. I remember playing around with that a few months ago and having mixed results - I think I might have posted about it on CFC, but I can't remember if I did or not. Regardless, that sort of thing is something that I feel should theoretically be possible, but will require more research.

Embryodead's Extra Terrain BIQ, which I used to make the Lost Worlds map, added LMTundra, LMFlood Plain, LMJungle, LMMarsh, LMVolcano, LMCoast, and LM Ocean. The names all appeared in the editor list, and some worked, some didn't. I know that I could never get LMVolcanoes to appear. I think I used LMTundra, LMMarsh (made a nice file for that), and LMCoast on the map without much trouble.

I can't recall off the top of my head how/if snow volcanoes work?

I think they have to be in the midst of snow mountains.

I could perhaps make a list of all the terrain-related PCX files mentioned? That's likely to be all the ones that Civ3 can use, and there might be a couple of unexpected ones. Aside from ones related to the city view (after clicking the eye, when in a non-scenario), there appear to be a relatively limited amount of them.

That's an excellent idea. I'd like to see that list!

 

[Blue Monkey]

To expand a little on Quintillus' excellent & thorough explanation for the uninitiated. Forgive me, programmers, if I oversimplify a bit. Just trying to give people the 5 cent tour.

The questions Balthasar raised and comments by everyone in the last few posts are precisely why so much effort has been made to get access to the source code. Source code is human-readable language such as the image Quintillus linked to. An .exe such as the biq is compiled from the source code into a lower level language. The actual program run by the computer is a binary code but human beings can access that via a hexadecimal editor such as the one enbryodead linked to in his thread. Assembly language looks like this image from embryodead's thread. People like Quintillus & Steph are looking at that, trying to translate it into something as understandable as Quintillus' example, then create an editor that lets the rest of us manipulate the assembly / hexadecimal code by clicking on some buttons and typing a couple of words.* It's as if they are reverse engineering a 747 so that the rest of us can buy a first class ticket. IMHO what they are doing is such a herculean task that it's amazing anyone even attempts it.**

The people who mod newer versions of civ don't face the same issues because they have been given tools like SDKs (software developers kits) that let them read the programming language that was used and edit or add new code. That's the equivalent of the C3 source code being downloadable almost as soon as the game itself was released. What they're writing may be incomprehensible to most of us. But it's a lot more like editing the civilopedia text file than doing hexadecimal math to figure out how to properly change some assembly language.* **


--------------

*To get some idea how difficult that is try adding AF0357DB to 00C918EE .

** Andy Hertzfeld is legendary among programmers. To quote wikipedia: "Hertzfeld's business card at Apple listed his title as Software Wizard. He wrote large portions of the Macintosh's original system software including much of the burned-in ROM code, the User Interface Toolbox, and a number of innovative components now standard in many graphic user interfaces ..." Andy Hertzfeld could do hexadecimal math in his head. He could do that with properly functioning assembly code. He could verbally offer suggestions for changes and additions in assembly code while he was working on something else. That means he could write and edit functioning programs from memory - he knew where in the long strings of hexadecimal numbers the commands were that needed changes. He could also change other others' code if he could get them to read it to him without any errors or omissions.

Every once in a while Quintillus feels up to probing around in someone else's 10 year old code - without access to the programmer or anything humanly readable - and try to figure out what's usable, what's incomplete but fixable and what's deadwood. Then not just make it editable via a nice gui, but editable across platforms with biqs from versions of the game whose assembly language differs from each other. While doing his day job & having a life. He's good at it and so is Steph. Wish we could pay them 6 or 7 figures a piece to do it. & give them some place as nice as Apple's campus to work at. All we can do is give them time & play-test the heck out their creations so they know what works and what doesn't. Know all that is preaching to the choir - but at least now it's been said publicly.

 

[Civinator]

Blue Monkey, in my eyes this is one of your best posts.

Other civers who must be named here by achieving great "hexediting improvements" to Civ 3 are skyer2 (especially the no-raze-patches), SYn and Shay Roberts.

 

[Ozymandias]

Blue Monkey, in my eyes this is one of your best posts. :

I most humbly concur

Other civers who must be named here by achieving great "hexediting improvements" to Civ 3 are skyer2 (especially the no-raze-patches), SYn and Shay Roberts.

-Oz

 

[Blue Monkey]

Blue Monkey, in my eyes this is one of your best posts.

I most humbly concur

Thank you both. I'll take that as encouragement to dust off the notes/files & get back to some work on tutorial articles for the modiki.

 

[Quintillus]

You are too kind, Blue Monkey, but I agree with Civinator, very good post and explanation. Gramphos and others who worked on the BIC/BIX/BIQ format also deserve mention for discoveries made with a hex editor - it's only by standing on their shoulders that I was able to get the editor to a useable state in an amount of time that made it possible to begin with.

Thanks for the additional information, Balthasar. I'll see if I can put together that list, and will be looking into landmark terrains in the editor.

 

[Blue Monkey]

... and will be looking into landmark terrains in the editor.

Ability to see more of the map (larger window/zoom) and access to even some of the other lm terrains are the two things that would make it possible for me to use your editor in preference to the standard one.

 

[Quintillus]

Embryodead's Extra Terrain BIQ, which I used to make the Lost Worlds map, added LMTundra, LMFlood Plain, LMJungle, LMMarsh, LMVolcano, LMCoast, and LM Ocean. The names all appeared in the editor list, and some worked, some didn't. I know that I could never get LMVolcanoes to appear. I think I used LMTundra, LMMarsh (made a nice file for that), and LMCoast on the map without much trouble.

Just took a look at the Lost Worlds map, and wow! That is quite the map! I spent quite a bit of time just examining the map, and more reading up on Hollow Earth, Edgar Rice Burroughs, and more. Looks like I've been missing out on a lot of cool stuff in the Project and Mod Development forums. I'm impressed.

Do you recall what the particular problem with the LM Volcano is? I'm able to get them in-game (on a random map with customizations), although I don't think there will be a way to get custom graphics for them. But what I'm seeing is that I can get landmark terrain for anything (including custom stats - testing with food/shields/gold), although for certain terrains such as coast and ocean the landmark symbol does not appear in-game when you have "Show Food and Shields on Map" enabled in Preferences.

Ability to see more of the map (larger window/zoom) and access to even some of the other lm terrains are the two things that would make it possible for me to use your editor in preference to the standard one.

Looks like access to other landmark terrains will be happening fairly soon. I've now added the ability to add any sort of landmark terrain, as well as toggle landmark-ness for a particular tile. I still need to polish it up a bit, but was distracted by the Lost Worlds mod.

Being able to zoom out is something that I've experimented with, but have not been able to get working that well. I have an ability to zoom out, but keeping the interactive parts of the map functioning as expected when zoomed out hasn't been reliable. Map display stuff in general is difficult. I suspect I'll be able to work out the kinks, but I'm not sure when. Aside from the ill-advised attempt to figure out how Firaxis determines which mountain and forest graphics to use on a tile, improving map display has been among the slowest areas for return on investment.

 

[Blue Monkey]

Zooming wouldn't be so important if the window (viewable area of map) was larger. Last time I used the editor it was about 10 tiles across iirc. But it's been a while. Maybe you changed that in a more recent version?

Edit: Would always showing the map at 50% be feasible? That would help. Single colored tiles such as maptweaker uses would be adequate for the kind of work needed to place lm terrain. Actual hame graphics could always be doublechecked in the Firaxis editor. But maybe all that falls into the "reverse-engineering a 747" category.

 

[Quintillus]

Zooming wouldn't be so important if the window (viewable area of map) was larger. Last time I used the editor it was about 10 tiles across iirc. But it's been a while. Maybe you changed that in a more recent version?

Edit: Would always showing the map at 50% be feasible? That would help. Single colored tiles such as maptweaker uses would be adequate for the kind of work needed to place lm terrain. Actual hame graphics could always be doublechecked in the Firaxis editor. But maybe all that falls into the "reverse-engineering a 747" category.

You can expand the window. While many of the tabs stay the same size, you'll be able to see more of the map with a bigger window. Since version 0.78, it's supported resolutions up to infinity pixels wide by 1200 pixels high. In practice, 1920x1200 is the largest I've been able to test at.

There was a point when the window wasn't expandable, but at some point I realized that didn't make sense any more since there was now a map.

The zoomed-out work I've done has largely been with 50% as a primary goal. I haven't looked into the possibility of simplified graphics. I'll have to consider that and whether it would make it more feasible.

 

[Blue Monkey]

You can expand the window. While many of the tabs stay the same size, you'll be able to see more of the map with a bigger window. Since version 0.78, it's supported resolutions up to infinity pixels wide by 1200 pixels high. In practice, 1920x1200 is the largest I've been able to test at.

There was a point when the window wasn't expandable, but at some point I realized that didn't make sense any more since there was now a map.

The zoomed-out work I've done has largely been with 50% as a primary goal. I haven't looked into the possibility of simplified graphics. I'll have to consider that and whether it would make it more feasible.

Sounds like I need to download a more recent version then. Since my monitor is at 1680 x 1050 that should work alright. Adding a 50% zoom would be close to perfect. If you did want to do something with simplified graphics I'm sure one of the 2d graphics artists here would be happy to make something for you.

 

[Balthasar]

Just took a look at the Lost Worlds map, and wow! That is quite the map! I spent quite a bit of time just examining the map, and more reading up on Hollow Earth, Edgar Rice Burroughs, and more. Looks like I've been missing out on a lot of cool stuff in the Project and Mod Development forums. I'm impressed.

Also: check out some of the projects in my Workshop (click the link in my signature). Glad you like the stuff we've done; I love the creativity in the Steampunk Forum.

Do you recall what the particular problem with the LM Volcano is? I'm able to get them in-game (on a random map with customizations), although I don't think there will be a way to get custom graphics for them.

The problem was that I couldn't use custom graphics for the Volcanos. Oh well.

But what I'm seeing is that I can get landmark terrain for anything (including custom stats - testing with food/shields/gold), although for certain terrains such as coast and ocean the landmark symbol does not appear in-game when you have "Show Food and Shields on Map" enabled in Preferences. Looks like access to other landmark terrains will be happening fairly soon. I've now added the ability to add any sort of landmark terrain, as well as toggle landmark-ness for a particular tile. I still need to polish it up a bit, but was distracted by the Lost Worlds mod.

Arrgh! Don't get lost in Lost Worlds! The rest is Great News! If you think LW is a good map, wait til you see what I can do with this!

Have you seen any way to make LM Roads or Rails, or to make roads era specific?

Any progress on that list of file addresses from the biq?

 

[Quintillus]

Sounds like I need to download a more recent version then. Since my monitor is at 1680 x 1050 that should work alright. Adding a 50% zoom would be close to perfect. If you did want to do something with simplified graphics I'm sure one of the 2d graphics artists here would be happy to make something for you.

The new versions are nicer! It isn't usually necessary to have the very latest release... but by a 0.10 difference between versions, there's usually a pretty significant difference in functionality, even though the functionality does come a little bit at a time.

I feel like I shouldn't need to get simplified graphics, but we'll see...

Also: check out some of the projects in my Workshop (click the link in my signature). Glad you like the stuff we've done; I love the creativity in the Steampunk Forum.

The problem was that I couldn't use custom graphics for the Volcanos. Oh well.

Arrgh! Don't get lost in Lost Worlds! The rest is Great News! If you think LW is a good map, wait til you see what I can do with this!

Have you seen any way to make LM Roads or Rails, or to make roads era specific?

Any progress on that list of file addresses from the biq?

I think I have taken a look at the workshop before - a few of those, such as the K-Street and Goldrush! - look familiar. Might've come across it while browsing SOC, or by your sig - not sure which. But, other than perhaps the interface, I'd missed the boat on the Steampunk Forum.

I just posted a new thread for the PCX's. It isn't complete yet - there's probably close to a thousand PCX's all told - but I believe I have finished the terrain ones. Unfortunately, it appears that there likely is no specific landmark terrain for volcanoes. Or marsh, for that matter. So, my guess is that Firaxis wasn't just lazy in the graphics department - it looks like didn't put code in for those graphics, either. But there might be something new and interesting in there already - I haven't examined the status of every PCX file that I've recorded.

Roads and rails look like another area where there will be no luck. The only mentions of roads are the road/railroad PCX's, four road PCX's for the City View, and one more than I assume is for the road button. Once more, it looks like these road ideas are simply not possible in Civ3. With the source code, it would probably be relatively easy to make era-specific roads. But without it... Roads will probably remain like they always have...