What I'm Up To

[Puppeteer]

Hey, looks like we have the board!

So I've clearly been running off in a particular direction. It was wholly inspired by WildWeazel's proposal but clearly a pretty lone-wolf, uncoordinated effort.

My aim is to largely recreate my previous SAV/BIQ/map tools in C#, a language much more appropriate to the purpose and something others in the mod community have experience in. And I'm trying to do it in such a way that the code can be reused either via copy/paste, inclusion, or dll linking.

I'm thinking that no matter which direction the project goes in, this code can be useful. I hope.

If it's not clear, I personally favor and am most energized by a new clone game using modern tech. But I'm not a very reliably consistent contributor to the community, so I certainly shouldn't be the driving force or big influencer behind the project.

In the public thread on the C# SAV/BIQ libraries I'm developing in public, hopefully without prematurely advertising what we're discussing. My history of projects has been all over the place enough that another change of direction and language should be wholly unsurprising from me. I think I've coded stuff in at least four different languages so far.

I'm currently working on an intermediate idea "Civ Parade" where users will be able to use Lua scripts to spawn and animate units on screen. Ideally it will read from the BIQ so you could showcase all units of a mod, all animations. As amusing is that is for me, it should go a long way to making pieces useful for a game engine, or if not then my next generation save monitor(s).

 

[Puppeteer]

Some random thoughts and questions I have. They don't necessarily need answers as I haven't even tried researching them yet, but I won't discourage answers, either.

- How is landmark terrain handled? Is it actually a base or overlay terrain, or is it something laid on top of one like a resource or improvement graphic?
- In e.g. all-grass tiles between pcx tile sets, are the all-grass tiles different or the same? (My short term plan is to only use one of each all-* terrain, but if e.g. the all-grass tile of plains-grass-coast is different than the one in tundra-grass-<whatever>, how would one choose which all-grass tile to place?)
- Are *all* FLC files 10 frames (11 with ring frame) in each direction, or are there differing frame counts?
- Is the frame time / speed always the same in all unit files? Is the duration of the whole animation constant? (Some careful timing may be needed in Godot/Unity to prevent looping which is probably not wanted for e.g. death animations.)
- Assuming landmark terrain does not count, is 256 base terrain types enough (16 is the current limit)

Assorted things I need/want to do sooner or later:

- Read the speed/frame-time from FLCs
- See if any FLCs change palettes mid-animation; I haven't seen this yet, but the Flicster guy I think said it happens
- Figure out how the palettes of leaderheads work (although I've recently discovered and fixed a color problem with my reader; I should try a leaderhead again)
- Have a reusable unit object with animation/movement levers
- Read in and display an entire map (base terrain first, then overlay terrain; can worry about other items later)
- Peek at Freeciv graphics as a possible stand-in as all my current ideas require Civ3 graphics be handy

 

[Puppeteer]

If anyone's looking at my code, it's a bit all over the place at the moment. Mostly because I'm trying to keep different modules separate: SAV reading, media conversion, unit control object, and front end graphics framework.

On the other hand, when experimenting it's easy to do it all in one place, so the Godot code is getting heavier. I want to pull some of that out into a Godot module that could be used for multiple Godot projects, but for the moment I'll pile a little more into the Godot-specific code rather than break out modules for a Godot-specific library and embedded scripting.

I still intend for the code to be usable with Unity, but each time I think about it, I can think of 10 interesting things I want to do instead of start from scratch with Unity. Also, even with Unity, there are surprisingly few tutorials aimed at the direction I'm approaching from, which is to create nearly everything from code because it's all in 20-year-old formats that modern tools aren't ready to use as-is, and there are hundreds of graphics files that I don't want to pre-convert.

I'm kind of counting on keeping logic in non-UI modules and having Unity be similar enough to Godot in that I can build 32-bit images from data and make tiles or animated sprites out of them. I think that's a reasonable assumption.

 

[Ozymandias]

I've already done the mea culpa of being many years removed from hands-on coding.

That being said, rather than going crazy with the Google machine, might you point me to reasonably sophisticated, yet concise, descriptions of each of -

• Godot
• Unity
• Lua
• - And, yes, even C#

Lastly, @WildWeazel - A matter of simple curiosity: why is Python so categorically undesirable?

 

[Puppeteer]

• Godot and Unity are game engines; for our purposes they would handle the game and user interface graphics. They will work on Windows, Mac, Linux, mobile devices, and even web. Both allow for advanced scripting with C#.
  • Godot - "is a 2D and 3D, cross-platform, free and open-source game engine released under the MIT license."
  • Unity - "is a cross-platform game engine developed by Unity Technologies." Paid licensing, but personal and organizations making less than $100k can use it for free.
• Lua - is a scripting language designed to be embedded into applications made in other languages. Its intent is not to make apps but to allow scripting within an app.
• C# - "C Sharp" - An object-oriented C/C++ adaptation in a managed environment. Originally a Microsoft-only product but now is open source and quite popular.

More on C# and .NET / dotnet : C# and a number of other "#" and ".NET" languages run on the CLR (Common Language Runtime) that manages memory. All data are class object instances, and memory is created for them upon creation, and automatically released by the CLR when there are no references to them. (e.g. the variable falls out of scope, replaced by a new object value, or is explicitly destroyed). So the programmer doesn't have to worry about managing memory, buffer overruns, and the like. This is quite similar to Java and it's JVM (Java Virtual Machine), and I've always believed that C# and .NET were Microsoft's direct challenge to Java.

You wouldn't write firmware or an operating system in C# or Java, but they're great for applications of all sorts and each have a lot of open source libraries available.


I'll throw in my echo that Python doesn't make a lot of sense as a game language. Because it requires an interpreter and isn't really intended to deliver a standalone application. And it can be awkward to use on Windows; it excels on Unix-like systems. I was going toå say there aren't really graphics or game libraries for it, but I guess there are a few after all: Pygame, Panda3d, OGRE...but I have no familiarity with any of them.

 

[Quintillus]

Some random thoughts and questions I have. They don't necessarily need answers as I haven't even tried researching them yet, but I won't discourage answers, either.

- How is landmark terrain handled? Is it actually a base or overlay terrain, or is it something laid on top of one like a resource or improvement graphic?

The last six bytes of a TILE are divided into:

4 bytes - C3C bonuses (binary)
2 bytes - unknown

(If the data length of a TILE is 49 instead of the expected 45, there are 4 more additional unknown bytes)

Whether a tile is landmark is part of the C3C bonuses. My code notes the following on them:

    public int C3CBonuses;
    //8192 or 0x2000 = landmark terrain
    //2048 = standalone river to south
    //1024 = standalone river to east
    //512 = standalone river to west
    //256 = standalone river to north
    public static final byte PINE_FOREST = 32;
    public static final byte SNOW_CAPPED_MOUNTAIN = 16;
    public static final byte BONUS_GRASSLAND = 1;

So if the byte at the 0x2000 location of C3CBonuses is set for a tile, it is a landmark terrain tile.

- In e.g. all-grass tiles between pcx tile sets, are the all-grass tiles different or the same? (My short term plan is to only use one of each all-* terrain, but if e.g. the all-grass tile of plains-grass-coast is different than the one in tundra-grass-<whatever>, how would one choose which all-grass tile to place?)

Not necessarily. Per the Apolyton documentation, the TILE structure begins:

    4    long        length of tile data 

    1    byte        river connection info  (binary)

                    00000000

            river in north    .......1

            river in west    ......1.

            river in east    .....1..

            river in south    ....1...

    1    byte        border

    4    long        resource

    1    byte        image

    1    byte        file:    0 = xtgc.pcx, 1 = xgpc.pcx, 2 = xdgc.pcx,

                3 = xdpc.pcx, 4 = xdgp.pcx, 5 = xggc.pcx,

                6 = wcso.pcx, 7 = wsss.pcx, 8 = wooo.pcx

The "image" byte is the offset within the "file". So for example there are all-grassland tiles in both xgpc.pcx and xggc.pcx. A tile could use either one, but you can look at the file and image bytes to know which ones to use.

Now if you're writing an editor, there are additional questions about how to know which one to save. IIRC, I have a hard-coded order for that; I have no idea what Firaxis's editor does.

In practice, as far as I'm aware most terrain sets will use consistent graphics for the all-grasslands, all-plains, etc. graphics, however, as it could become unpredictable otherwise. But there's nothing to prevent them from doing otherwise, and one of my wishlist features for my editor is allowing the user to choose which all-ocean graphics they wish to use from wooo.pcx... so you could have a humpback whale in one, a pelican in another, and a typhoon in a third if you wished, and gain variety on the map simply through choosing which all-ocean graphics you used, without needing to add lots of resources (and while still being able to add resources to those tiles if you wished).

- Are *all* FLC files 10 frames (11 with ring frame) in each direction, or are there differing frame counts?
- Is the frame time / speed always the same in all unit files? Is the duration of the whole animation constant? (Some careful timing may be needed in Godot/Unity to prevent looping which is probably not wanted for e.g. death animations.)

I can't answer these; hopefully someone else knows the answer.

- Assuming landmark terrain does not count, is 256 base terrain types enough (16 is the current limit)

For C7? I would imagine so, although we should probably have it as an arbitrary-length list rather than a hardcoded length. But if we did need to hard code it for some reason, 256 sounds like enough to me.

 

[Puppeteer]

Thanks Quintillus!

---

GodotCon notes (will edit and add):

- In case this still works after it's done
https://chat.godotengine.org/

Godot in the Enterprise is worthwhile

Shader Shenanigans is awesome! He uses UV light to create hidden text/objects, makes outlines of 3d objects, and replicates a Portal effect where you can see through a portal to a camera in another location.

Authoritative multiplayer - good, has a longer YT series: https://www.youtube.com/playlist?list=PLZ-54sd-DMAKU8Neo5KsVmq8KtoDkfi4s

WebXR was interesting to me, as was the IoT.

Flash talks: Garden Path - not my thing. ECS for Godot - interesting, and I want to check into this concept more myself (not for C7). Dialog system for Hauma VN - the talk drags but introduces some interesting concepts and design including communicating with another engine (Ink),

 

[WildWeazel]

A matter of simple curiosity: why is Python so categorically undesirable?

I didn't mean to say that, just simply that the game engines we've discussed are .NET based. (Godot's GDScript is similar to Python though) It makes sense to choose the engine and let that dictate the language(s), provided it's one we're comfortable with. There are some Python game libraries* as Puppeteer mentioned. I don't know much about them either, only that they're not nearly as popular which I'm guessing is for a reason.

*ETA: specifically libraries, which leave a lot more of the architecture and grunt work to you, as opposed to full blown game engines.

 

[Puppeteer]

@Quintillus , do you know how map tile rendering is handled at the top and bottom? For now, I'm just making "north" equal to "west" when y=0 and "south" = (east or west, I forget) when y=max. But I'm guessing the game does something different.

 

[Quintillus]

Not precisely. It depends somewhat on whether polar ice is enabled. In-game, if you have a map with polar ice and both sea and tundra at the bottom of the world, you can see it's not the best visually - there are sharp edges between terrain types. For example, this is an in-game screenshot of northern Antarctica (south of South America) in The Old Empires mod (which uses the default terrain graphics), with the grid enabled to show where tiles are actually located:

Perhaps due to the game's imperfections, I never bothered trying to replicate it exactly.

 

[Ozymandias]

• Godot and Unity are game engines; for our purposes they would handle the game and user interface graphics. They will work on Windows, Mac, Linux, mobile devices, and even web. Both allow for advanced scripting with C#.
  • Godot - "is a 2D and 3D, cross-platform, free and open-source game engine released under the MIT license."
  • Unity - "is a cross-platform game engine developed by Unity Technologies." Paid licensing, but personal and organizations making less than $100k can use it for free.
• Lua - is a scripting language designed to be embedded into applications made in other languages. Its intent is not to make apps but to allow scripting within an app.
• C# - "C Sharp" - An object-oriented C/C++ adaptation in a managed environment. Originally a Microsoft-only product but now is open source and quite popular.

More on C# and .NET / dotnet : C# and a number of other "#" and ".NET" languages run on the CLR (Common Language Runtime) that manages memory. All data are class object instances, and memory is created for them upon creation, and automatically released by the CLR when there are no references to them. (e.g. the variable falls out of scope, replaced by a new object value, or is explicitly destroyed). So the programmer doesn't have to worry about managing memory, buffer overruns, and the like. This is quite similar to Java and it's JVM (Java Virtual Machine), and I've always believed that C# and .NET were Microsoft's direct challenge to Java.

You wouldn't write firmware or an operating system in C# or Java, but they're great for applications of all sorts and each have a lot of open source libraries available.

I'll throw in my echo that Python doesn't make a lot of sense as a game language. Because it requires an interpreter and isn't really intended to deliver a standalone application. And it can be awkward to use on Windows; it excels on Unix-like systems. I was going to say there aren't really graphics or game libraries for it, but I guess there are a few after all: Pygame, Panda3d, OGRE...but I have no familiarity with any of them.

TY!

Within the next few days, I'll be releasing a comprehensive outline of how we might build a new game engine which could:

• Allow us to utilize every game asset at hand:
  • Every Tile type and associated attribute (movement cost; how much food; etc.)
    • Readily add new Tile types, including, for example, "Land + Water."
  • Every Unit type and attribute (Attack Strength; movement; etc.)
    • Readily add new Unit types (like "hybrids:" a helicopter which can move over Land and Water Tile, but can only end its move on either a Land Tile or a Carrier.)
• Much of the game's logic would involve little more than simple math and Booleans.

I had a "Eureka!" moment a while ago, when I realized that imposing simple, old fashioned, cardboard-and-paper game rules could (as I've already mentioned) produce sophisticated AI Civ pay, from what to build, to how to deploy and use its Units - and, no, I've not taken any psychotropic drugs recently ( ).

I've already mentioned that part of the "Ah-ha" moment was realizing that the following terms can be interchangeable:

• Rules
• Constraints (on what the AI can do)
• Logic
• (Simple) Algorithms
• AI behavior.

If nothing else, I believe that being able to define those words (and others) for a common terminology and goal, would make it much easier to have a conversation between those of us who know the game well (and aren't coders) and those who know the game well (and are coders.)

You might have already guessed that the approach OO, meets all of our goals - and we won't need nobody else's stinkin' game engine!

I know that this post is, in part, a repetition, but I'm actually working on Object diagrams now, and I confess to being a tad excited.

 

[Vuldacon]

I believe those of you who have and are looking into Existing Game Engines or Making a New Game Engine along with the work involved to work with or make them have some differences of opinion.

Personally, I am simply continuing to work as usual, helping (more behind the Scene) Civfanatics Members with various procedures for Graphics work such as Making Units from Creation in an Animation Program to the finished work in Game.

Other than helping where I can, I have several projects on my "to do list" and I am watching here to see what collective decisions are made to actually begin work using what is decided for our Group Endeavor.

From what I have gathered, it seems we are still in an "Experimental Stage" concerning a Game Engine and I hope we will be able to make an informed decision regarding it sometime soon.

Life, especially now, has added individual problems and difficulties and no doubt will continue. Despite our problems and difficulties, this meritorious endeavor is, at the very least, a worthy distraction and I am excited about where it will lead.

All things considered, this "experimental stage" is necessary to be able to decide on the direction that will be taken regarding the Game Engine that will be Made or Used.

Eric... I am Looking foreword to seeing your presentation regarding a New Game Engine.

 

[Ozymandias]

All things considered, this "experimental stage" is necessary to be able to decide on the direction that will be taken regarding the Game Engine that will be Made or Used.

Speaking of which - What are the benefits and constraints which of any of thee game engines might (accordingly) help us (best case : plug compatibility) and hinder us (worst case: long learning curve)?

 

[Puppeteer]

Speaking of which - What are the benefits and constraints which of any of thee game engines might (accordingly) help us (best case : plug compatibility) and hinder us (worst case: long learning curve)?

I'm not sure plug compatibility will help us any; I've long suspected it's possible to define data structures that could natively read in the SAV file (after decompression), but I'm less sure of that today, and even if that were so, the SAV file and in-memory data structures are a huge part of the limits we want to remove, especially the 31-civ limit and the strategic resource limit. I think Qunitillus' work, mine, and previous SAV file monitors (CA II, MapStat) show that it's possible to get the data out and use it in a new format without the limitations.

That said, C++ is what Civ III was written in (if memory serves), and in theory it might perform better as would C or Rust. Heck, being an older language, maybe there are older libraries to deal with FLC and 8-bit PCX files. And C++ still appears near the top of popular programming languages, so in the event we all disappear, others might have an interest and knowledge to carry on with what we leave behind. But I've yet to hear any of us express the slightest interest or experience in C++, so that may introduce a learning curve. Being a systems language, it's far easier to severely and/or subtly break things with it.

I would suggest that there are fewer modern graphics/game engines made for it, but several are made *with it* and are open source, but there is zero dev or community support for making a game in C++ and directly linking the engine code. (3rd edit: I may be wrong here; Godot and GDNative may allow for using C++ libraries, although it's not a thing lots of folks do to my knowledge. I very briefly tried using GDNative with Go and quickly ran into problems, but I think that had more to do with passing 'unsafe' data to/from a C/C++ environment and a managed code environment.)

Java, C#, and Python all are up there with C++ in popularity, although I'd argue data science has a LOT to do with Python's resurgence, and we've already covered some reasons why Python may not be the best for our purpose (although I still am interested to try out using it as an in-game script engine with IronPython).

Java and C# are almost peers in general language features. C# would appear to have better game engine support and I think portability, although I'm not certain on the last point, and also I'm not sure we're even imagining a clone game would be playable on mobile or web, for example. Just Win, Mac, and Linux with keyboard and mouse controls likely.

WildWeazel has expressed interest and has experience with C# and Unity and has an interest in Godot. I've toyed with Godot a bit and have recently implemented a basic SAV reader, FLC reader, and PCX reader in C#, and demonstrated decompression using a C# library. Quintillus has done a lot of BIQ and SAV reading in Java.

For me a biggie is that Unity and Godot are modern still-active game engines with sizable communities and C# as their language interface.

A possible downside of C# and/or Java is version creep. As popular evolving languages we're seeing CA II (made in .NET / presumably C#) and Qunitillus' editor having problems with older features being deprecated in newer iterations of the runtime. I'll counter that by saying that version creep will be an issue, anyway, as long as we use a 3rd-party graphics/game engine, and that using a very open license it will be possible for future folks to update it if we're no longer present and available to do it ourselves.

And while C++ may not have that problem per se, there is a similar problem we're seeing with Civ III itself in that newer operating systems aren't dealing with 20-year-old games very well because they were programmed with a rather fixed display size in mind and semi-tied to an older version of DirectX which I guess counts as a graphics and sound engine in this comparison.

Edit: Side note: I haven't done any C++ in probably 26+ years but gave it a quick try as the compilers should be included on my Mac and Linux boxes. I failed, but I only Googled twice and gave up.

#include <iostream>

cout << "Hello, world"

It says it can't find iostream. I originally had <stdio.h> then remembered it's <iostream.h>, then Google said that's the old way and the new hip way is <iostream>. Then I tried some env vars to point to the include paths, then I gave up.

Edit 2: Ok, being stubborn I installed c++ compiler on a different Linux box, Googled again and got it working. I now recall the main() function and returning an int but don't recall using std:: 26+ years ago.

#include <iostream>

int main() {
        std::cout << "Hello, world";
        return 0;
}

 

[Quintillus]

I would agree that C# has better game support than Java. Java has better enterprise support, but relatively little desktop (and thus game) support - unless you are talking developing games for Android, in which case Java (and Kotlin, which runs on the JVM and I like to describe as, "what Java would be, if it were developed with 15 years' knowledge of how Java would be used") is the way to go. But I agree that target should be Windows, Mac, and Linux, in that order, and anything else is icing on the cake.

C++ has changed a lot in the past decade or two. We've gone from regular pointers to auto pointers, now to smart pointers. Various smaller improvements, too (although I'd find leaving out the sys:: more readable, personally). I never learned any C++ until 2008 or so and my knowledge is already antiquated. There's also the issue of small differences between compilers (Visual Studio, GCC, etc.) that we'd have to reconcile, and which C# and Java don't have. It would definitely add a learning curve, except perhaps for WildWeazel whom I believe uses C++ at work. I think without compelling library advantages, we'd be making our task more difficult by going with it, and the libraries seem to favor C#.

 

[Puppeteer]

Quick personal update: I've moved 1400 or so miles the past couple of weeks, and I'm still settling into a new apartment. I haven't been particularly active on the project or the forums, but I have been playing Civ3 lately.

So I haven't disappeared; I've just been busy, and I still need to finish all the moving stuff like new drivers license and all that fun stuff.

 

[Ozymandias]

C++ has changed a lot in the past decade or two. We've gone from regular pointers to auto pointers, now to smart pointers.

Look at the bright side: You never had to threaten junior "C" coders with public execution for suggesting having arrays of pointers pointing to other arrays of pointers.

Quick personal update: I've moved 1400 or so miles the past couple of weeks, and I'm still settling into a new apartment.

And in a Time Of Plague as well. I'm impressed!

 

[Puppeteer]

This isn't on my public repo yet as I'm trying to refactor for possible future use in a new C# game using Godot. But I am able to:

• Read civ install location from registry (alternately from CIV3_HOME environment variable)
• Browse saves starting in the Conquests saves folder
• Auto-decompress save if needed
• Display base terrain where all surrounding tiles are plains, grassland, or coast

 

[Puppeteer]

I changed how I'm drawing the map. I'm using the file byte and image byte as mentioned by Quintillus earlier in the thread. It's much easier!

I can also print tile values on the map; this will help figure out what unknown values correlate with. That's near the end of the video. This should come in handy to help find the overlay terrain references.

While recording this I crashed the program several times; at first I thought I was misclicking, but no one of my save game series errors out. I think I don't have all the tilesets accounted for yet.

Edit: If you can freeze it at 0:22 in the video you can see debug text that seems to correlate with the map...that's the river data I'm pretty sure. Mostly 0s everywhere but bigger numbers on parts of land most likely bordering rivers.

Edit 2: Oh, the crashes were because I had the map size hard coded at 100x100, and I was trying to open a bigger map. It's now allocating the right map size based on the save.

 

[Ozymandias]

At whenever is a good point, you might want to have a glance at Plotinus' "Undiscovered Worlds" project.