Cross-Platform Civ3 Editor

Cross-Platform Editor for Conquests now available! 1.48

So, what have I been working with on the editor? It hasn't been quiet, just under-the-covers.

The first thing was a proof-of-concept that a new, more scalable UI would work. By and large, it's been successful. Redoing the whole interface would take approximately forever, but new additions will likely use the newfangled UI.

Then came a detour to do some evaluations of what would be affected by the AtomicGamer shutdown.

More recently, I've been experimenting with modernizing and speeding up the loading code, in particular for PCX files. This has been interesting. In 2013, I'd sped up PCX loading on multicore computers by allowing multiple PCX's to be loaded at once. But I'd done so using a very old-school way, using methods available since the mid-'90s. It did speed it up, but was by no means state-of-the-art.

So I updated it to use the latest technology of 2004. At worst, I figured, this would be equally fast. And there was reason to believe it might lead to a bit of an improvement. And it didn't take too long to get it working correctly. The problem? Not only did it not speed things up, it was slower. Even in the best case, it took 20% longer to load the PCXs for a scenario, and at the worst case it was much slower. It didn't matter whether I was testing on the latest Windows 8.1 (a few days before Windows 10 came out), or old Windows XP, nor whether I was loading from a hard drive or solid-state driver - the new method simply wasn't as fast. All I can figure is the overhead outweighed any theoretical benefits.

Thus, I've gone back to my mid-'90s method of opening multiple PCX files at once, and looked at the algorithms to see if they can be improved. The good news is that after a couple more years experience, I spotted several areas that could be improved, and importing PCXs now takes 14-43% less time than before (average 25%). And this should be seen even on old single-core systems.

All in all, it reinforces two software development principles:
- Always make sure that theoretical performance benefits actually benefit performance.
- Most of the time, a better algorithm is the best way to improve performance.

And arguably a third, that developers love optimizing code. The cumulative time saved by faster PCX loading might never equal the amount of time I've spent optimizing it, but it's an enjoyable area to work on.

For those curious, a solid-state drive has essentially no impact on how fast the editor loads scenarios or works. When importing a BIQ or PCX, over 90% of the time is actually processing the file, not reading it from disk. I haven't tested Civ3 as much, but my inclination is it's probably a pretty minimal impact for Civ3 as well.
 
[...] Thus, I've gone back to my mid-'90s method of opening multiple PCX files at once, and looked at the algorithms to see if they can be improved. The good news is that after a couple more years experience, I spotted several areas that could be improved, and importing PCXs now takes 14-43% less time than before (average 25%). And this should be seen even on old single-core systems.

All in all, it reinforces two software development principles:
- Always make sure that theoretical performance benefits actually benefit performance.
- Most of the time, a better algorithm is the best way to improve performance.

And arguably a third, that developers love optimizing code. The cumulative time saved by faster PCX loading might never equal the amount of time I've spent optimizing it, but it's an enjoyable area to work on.

As ever, thank you for your extraordinary efforts - and, as IT guy from the '80s ( :old: ) I very much appreciate (1) "that developers love optimizing code" and (2) that "newer" isn't necessarily better.

:thumbsup: ,

;)z
 
As ever, thank you for your extraordinary efforts - and, as IT guy from the '80s ( :old: ) I very much appreciate (1) "that developers love optimizing code" and (2) that "newer" isn't necessarily better.

:thumbsup: ,

;)z

Glad to be able to contribute :).

Second that emotion.

& (3) The people doing the work are too busy doing it to give updates as often as clients want them

:goodjob:

Ah yes. I've experienced that at work, too. And when, at my old company, we started giving updates more frequently last fall, we also ran into the corollary that once you start giving updates more frequently, the clients expect their requested changes to be available more quickly as well, even though it's only the frequency of updates that has changed and not the amount of time to work on things. Thank goodness for project managers with experience at setting expectations.

I did forget to mention the actual feature that I've been looking into - being able to add or remove tiles from an existing map. Most of the work to do this is still to be done since I started with the user interface part of it rather than actually adding/removing tiles, but the first part of the newfangled UI will likely correspond with that feature. In the meantime, if you're looking to be able to do this, it's currently available as part of Steph's editor.
 
Unlike I wrote in my previous post, I do have now a question and/or suggestion. :blush:

Your editor opens every time in 3/4 screen window which I enlarge to full screen (1280x1024) to avoid side scrolling.

While everything is completly on the screen, only the unit selection (framed red) is still a very narrow slot and shows most of the names only incomplete.



Would it be possible to make unit selection broader, so that the names are fully visible (except for extraordinary long ones)?

Is their a way, that your editor remembers to go on full screen when starting?

And yes, I have to upgrade my editor to the latest version yet. ;)

Thanks again!
 
Does anyone know why when I'm in the unit tab, and I scroll through different units I get their values randomized?

For example if I'm on the settler, and I skip to the medieval infantry, the med infantry will acquire the combat values of the settler and have its own values deleted, then act as if it was a settler.

I dont know if it has anything to do with me owning a steam version of civ. I think there is a work around but it makes my work very slow. I'd have to make the unit back to the way it was, then add changes I was gonna do with Quint's editor.

Any help will be appreciated.
 
Unlike I wrote in my previous post, I do have now a question and/or suggestion. :blush:

Your editor opens every time in 3/4 screen window which I enlarge to full screen (1280x1024) to avoid side scrolling.

While everything is completly on the screen, only the unit selection (framed red) is still a very narrow slot and shows most of the names only incomplete.



Would it be possible to make unit selection broader, so that the names are fully visible (except for extraordinary long ones)?

Is their a way, that your editor remembers to go on full screen when starting?

And yes, I have to upgrade my editor to the latest version yet. ;)

Thanks again!

That the editor doesn't take up the whole screen, except on the map tab, when you have more than 1024x768 pixels is a current limitation. When I've mentioned in posts the new UI that I'm looking at, that's one of the primary problems it's designed to solve. Unfortunately, it requires rewriting substantial parts of the program, so it's slow progress. So in summary, it is possible to make the unit selection broader when there are extra pixels, but it's not something that could be done quickly, particularly on complex tabs such as units.

There isn't currently a way to have the editor remember to start in full screen, but that's something that could potentially be added fairly easily. I've added it to the list of features to add (https://trello.com/b/VoC07SVK/conquests-editor).

Does anyone know why when I'm in the unit tab, and I scroll through different units I get their values randomized?

For example if I'm on the settler, and I skip to the medieval infantry, the med infantry will acquire the combat values of the settler and have its own values deleted, then act as if it was a settler.

I dont know if it has anything to do with me owning a steam version of civ. I think there is a work around but it makes my work very slow. I'd have to make the unit back to the way it was, then add changes I was gonna do with Quint's editor.

Any help will be appreciated.

To clarify, which editor does the randomization happen with? I haven't seen it with either my editor or Firaxis's, and was unable to cause it with either tonight when trying to do so. And by "scrolling through", do you mean with the arrow keys, clicking on different ones, mouse wheel, or something else? It's possible that the way I'm trying to make it happen is different than the way you're doing it. Unfortunately, I probably won't be able to help with it unless I'm able to reproduce how it happens, or be sure that I'm testing it in the correct way.

Tonight I was testing with the CD version of Civ3 Conquests; I can test with the Steam one as well on another computer if it's the Firaxis editor that's causing it (though I may or may not wind up being able to reproduce it). If it's my editor, details such as which operating system (including version) may also help, although unfortunately I don't have all versions available to test, particularly with recent versions of Mac OS X.
 
Hi Quintillus

Im playing on Windows 8.1, 64 bit with the steam civ 3 complete version 1.22. I'm using your 0.96 editor to load my civ3 complete ver 1.22 scenario.

I load up my scenario on your 0.96 editor, turn to the units tab and:

1- I click on the settler, then with my keyboard's arrow keys on the left of the num pad I hold the down arrow, and the unit selector begins to scroll. Eventually the blue shade selector will duplicate itself, then I click it to see whats going on, and I notice that the values of that unit have been changed. I click on the surrounding units above and below the duplicated blue selector and notice they also have the wrong stats, and they all have the same wrong stats (Tank 3/3/3, Mech 3/3/3, horseman 3/3/3)

2- I click on the settler, but instead of scrolling with the keyboard arrows I proceed to clicking random units as I use the side bar to scroll down a little at a time.

There seems to be a pattern. Everything seems ok until I stop clicking on random units. The changes seem to happen in clusters of 3-5 units. When I click 1 unit, then click the next unit following the list more likely than not it will obtain the previous unit's stats, and they both will be the same. Its almost a cloning issue, but instead of cloning into a new unit it clones onto an already existing unit, and keeps the original graphics I had for that unit.
 
So, I tried to reproduce it on Windows 8.1 64-bit, but was not able to. However, since there isn't any functionality associated with having multiple units selected at once (and some tabs such as building only allow one at a time already), I plan to make the lists single-select, and that should probably solve the issue you're experiencing.

Another thing to check would be the log.txt file in the folder where the editor is located. It's possible that it's running into an error of some sort that's causing the issue (one possibility is there's certain data for a unit that's throwing it off, but isn't in the BIQ I've been trying to reproduce it in), and if there is error information there it could help identify the problem much more precisely, including why you can reproduce it but I haven't been able to. Any error should be identifiable by a lot of consecutive lines in the log about technical programming things, and most of them will be indented.

On an unrelated note, I've been making decent progress on allowing the editor to import maps from Vanilla Civ3 maps, a suggestion that came up in another thread. So far that's working so long as the BIC doesn't have custom rules. Haven't tried with Play the World maps so far.
 
The other day I decided to do some forum archaeology, and find as many Vanilla BIC files of different versions as I could. With Conquests, I'd only ever seen 12.06 and 12.08, and only in some of the Conquests scenarios, 12.07 (which didn't appear to have any unique characteristics). After perusing the dustiest pages of the Maps forum for a couple hours, I found a greater variety of Vanilla BICs - likely to be expected, since the editor was evolving fairly rapidly in those days compared to the Conquests days. In all, I found these versions:

2.05 = Vanilla 1.00
2.07 = Vanilla 1.07f? So far I've only found this with the BICs that shipped with my Vanilla disc, which was version 1.07f
2.10 = Vanilla 1.16f (and 1.17f. Confirmed at Apolyton; 1.17f initially shipped with an editor with 3.00 BICs, but the game didn't support them so 2.10 was put back)
3.08 = Vanilla 1.21f
4.01 = Vanilla 1.29f (most recent patch. This is what the editor could open as of post #551)

Still figuring out the differences between them, and there's at least one difference I've found - between 2.07 and 4.01, one of the undocumented TILE bytes was added - I'm hoping patch notes might shed some light on that, although it's possible it's a feature that never made it into an actual release, but was left in the BIC so as to not break compatibility with Firaxis-developed pre-release BICs. Nominations for potential TILE features added in that timeframe are encouraged.

As for the BIC -> Vanilla version matchups, I've deduced them by comparing the versions found in BICs released at particular dates with the patch release dates. It's somewhat inexact since not everyone patched right away, and also challenged a bit by the occasional broken link for old downloads (although the ones uploaded to CFC are still there, and that's a lot of them - good job Thunderfall!). But largely they add up - the biggest question is whether there's a different 1.17f version that I haven't found, and if there's another patch that wasn't in the Info Center list and I missed - 1.07f was missing, but the 2.07 BIQs and having paid enough attention to my Vanilla disc allowed me to deduce that link.

Ultimately the goal is to be compatible with Vanilla BIC files of all versions. PTW will likely follow that, but one step at a time! For now, the editor will still only save as Conquests 12.08, and only be open-compatible with old files.

It does look like the old Apolyton threads on the BIC/BIQ formats will be helpful in determining the differences in versions with less trial-and-errors, but the BICs here will be very useful in actual testing. It turns out that the thread I didn't download also has useful information in this case - thankfully it's still up as well.

What's the ultimate benefit of it? Probably not a ton, but it would be cool to have equal compatibility to the Firaxis editor, and it'd also let Mac users load up Vanilla BICs without translating them via the Windows editor. It's also slightly cool that it'll let the editor open a version 2.05 BIC uploaded to CFC shortly after the game's release by the author of Firaxis's editor. Since the BIQ/BIC code is also available on the project's Bitbucket page (which I will add a link to in the OP), it will also make it easier for others to add BIC support to their own utilities.
 
Hi Quintillus,

I have upgraded to the last version 0.96 and I have a little problem.

There have been now three times, that the KI does not build settlers in my game. I have checked my main BIQ and several scenarios with both Firaxis and your editor.

It seems to me that your editor removes the "join city" order.

All scenarios have got both "build city" and "join city" order active. If I save a scenario with your editor (and thereby remove "join city") and then open and save the same in the Firaxis editor, the Firaxis editor "helpfully" removes the the strategy checkbox, making the KI unable to use the settler.

Thanks for your help!
 
Yes Kirejara I was about to post this lol. Remember the issue I talked about the editor cloning? It seems that a half ass cloned version of the worker is being posted on the settler. It changes the unit_32 to the workers, and it disables the join city. This is what I noticed when I opened the editor to double check.

In game its still a settler but it wont settle. I can confirm this.
 
Hi Quintillus,

I have upgraded to the last version 0.96 and I have a little problem.

There have been now three times, that the KI does not build settlers in my game. I have checked my main BIQ and several scenarios with both Firaxis and your editor.

It seems to me that your editor removes the "join city" order.

All scenarios have got both "build city" and "join city" order active. If I save a scenario with your editor (and thereby remove "join city") and then open and save the same in the Firaxis editor, the Firaxis editor "helpfully" removes the the strategy checkbox, making the KI unable to use the settler.

Thanks for your help!

Yes Kirejara I was about to post this lol. Remember the issue I talked about the editor cloning? It seems that a half ass cloned version of the worker is being posted on the settler. It changes the unit_32 to the workers, and it disables the join city. This is what I noticed when I opened the editor to double check.

In game its still a settler but it wont settle. I can confirm this.

You are correct, I have been able to verify there is an issue with the Join City option. It traces from the 0.90 version, where I was making some changes to improve the long-term maintainability of the editor, and made a mistake that introduced this issue. Actually, there were two separate mistakes, but both were with the Join City option. I've fixed both now, and they will be in version 0.97. Combined with the Vanilla changes, I think it's about time to release that one, so it will likely be going up later today (North America time).

I still haven't been able to reproduce a separate cloning issue, but I did find an area where the editor will save a unit's data twice when switching units, and there's a chance that it could get thrown off with multi-select. Although I did disable multi-select a couple weeks ago, I changed this so it won't try to save twice as an extra precaution (there wasn't any benefit to it saving a unit twice anyway). I still haven't been able to reproduce it so it's a "should fix it" change (whereas I could reproduce the Join City issue), but I believe the code changes should eliminate the chance of that happening.
 
Version 0.97 is now available. Changes are:

  • Added initial Vanilla scenario support. Currently, it is recommended that you only use this for importing Vanilla maps.
  • Optimizations to PCX importing, resulting in 14-43% less time spent on PCX importing (average 25%).
  • Fix the issue with Join City not functioning properly.
  • You can no longer select multiple units at once. Also removed a duplicate save of data when changing units.
  • The editor no longer tries to align resources between BIQs when importing a map, if either BIQ does not have custom rules (and thus resources).
  • Mountains and hills now use Grassland as their base terrain, rather than mountains/hills.
  • Fix a bug where if you imported a map via BMP to a scenario without an existing map, saving may not work due to a communication issue with the GAME (scenario properties) tab.
  • Fix a rare case where if you added a river to one side of a tile, it wouldn't be added to the tile opposite the river.

For the mountains/hills change, previously, the behavior would result in the desert terrain showing up under them, which is different than the Firaxis editor and not visually advantageous (unless you had a lot of hills/mountains on deserts and had modified your graphics for hills/mountains to fit in with deserts). Now grassland shows up under them.

For the river fix, the rare case was if you opened a BIQ where it already was the case that there was a river indicated on one of the tiles bordering the river, but not the other. Previously, adding/removing a river would tell the other tile to change whether it had a river on that border - so if only one of the two tiles did, you could never get both to agree on having a river. Now, if you add a river, the opposite tile will always have a river, and if you remove one the opposite one never will. I've only seen this rare case on one Firaxis-shipped map so far, but it may be present in a few others.

The unit select/save changes should fix the cloning issue observed by ZergMazter.

For Vanilla support, importing maps should work. For opening scenarios, particularly scenarios with custom rules, support is still somewhat of a work in progress. Additional work on dealing with the difference between PTW actions and pre-PTW actions is needed before custom rules will really work well. So for now, go ahead and import maps from BICs, but I'd recommend waiting until 0.98 before modifying BICs themselves. The Open dialog will thus not show BICs by default.
 
I'm back doing some map work again. I've installed the latest version of the editor (0.97). When I try to open a new fille :

File>New

I get the error message :

Could not find the file: /Volumes/Hulot/Civ III/Conquests/conquests.biq (Not a directory)



in the editor options the default opening directory is:

/Volumes/Hulot/Civ III/Conquests Game Data

which matches the actual path on my computer

/Volumes/Hulot/Civ III/Conquests Game Data/conquests.biq


running OSX 10.9.5
 
Hmm, is this the File --> New option? Or adding custom rules to a BIQ that doesn't have custom rules? Either way, it sounds like it's time for me to fire up the old PowerBook and see what isn't working as expected. At first glance it looks like I probably forgot to account for part of the different directory structure on OS X Civ3 versus Windows Civ3.

Edit: Have been able to replicate it, looks like it was indeed not remembering that the directory structure differs.
 
Top Bottom