What I'm Up To

[Puppeteer]

That's pretty far out for me. I'm still trying to figure out how to assemble Ikea furniture with partial secondhand instructions and missing all the connectors. I'm working from the bottom up on trying to simply understand the data from the save, logically what it means, and how to remove restrictions without knowing the data structure of our future tile(s). Or rather how to code the visual representation without putting constraints on a future tile dat format.

But we'll have to at least think about the implications of procedural generation when formulating the new tile structure.

Clearly there is demand to remove the tile type limit, but the way the current graphics are implemented, even adding a single tile type will require a lot. It has to have tilesets for every tile type it could be next to. Heck, there isn't even graphics for putting desert next to tundra. The number of tile combinations to draw and even the file count grows exponentially, and/or you have to prevent tiles from being placed next to each other if there is no graphics for it.

My early thought is that we'll have to abandon the media-tiles-as-quarter-of-four-different-tiles paradigm and adopt a logical-tile-centered media with procedural blending between types. But I also imagine you could just have a bigger field of grass texture rather than go tile-by-tile, so instead of drawing a single tile you're drawing a big area that masks, wraps, and blends appropriately when the game draws it.

Editing to add: Oh, I guess I didn't make it clear here: This map drawing effort is to replicate what Civ3 does; it's not meant to be the new primary graphics. It's what we can use to draw pretty pictures for any new tile, map, or unit data formats we might come up with. I'm using programming interfaces so the new tile type can have its own design but just implement a few getters to use the legacy Civ3 graphics until a new map graphics format evolves.

I originally thought we might mix old and new graphics, but it's quickly becoming clear that makes no sense; it will make more sense for a new map format to pre-convert the media to the new format. What I'm doing now writes no new data but just reads the SAV into memory, reads the appropriate legacy terrain pcx files and displays them.

Editing again to add: The goal is still to be able to use preexisting mods, and this map viewer should be able to do it. The goal is also to remove limits on things like tile types. But from where I am now, I see no path to being able to add more than 16 tile types to old mod formats. Instead I imagine the old mod can be one-click converted to the new format then be edited with new tile types (17 or 1700 or 1.7 million more), or perhaps a new mod can load multiple old mods and pick and choose unlimited tile types from them.

 

[Vuldacon]

Complex and Extreme indeed! I think more research and thought required here.but that said Good "path understanding". "Brain Storm" possibilities and Sleep on it.

 

[Puppeteer]

Well, I've hit a brick wall. I thought finding and adding the overlay terrain would be easy, but the more I look at it, the more confused I am.

After using the image file ID and image offset Quintillus mentioned earlier, I figured there would be another such pair for the overlay terrain. But I have yet to find anything promising in documentation or in my debug value viewer.

I know how to get the overlay terrain type ID, and I'm sure I could grab "a mountain" or "a hill" or whatnot, but I have no idea how to pick which one.

Hills and mountains are tile-centered graphics as opposed to the base terrain 4 quarters graphics. And there are forested and/or snow-capped variant tile files. Each tile file has 16 items, with the SE & SW edges varying between flat and raised. But I can't see any pattern to how it decides whether to connect mountains and/or hills and why it might choose any particular of the 4 models for each edge combo.

Rivers aren't overlay terrain, but they seem to go back to the 4-quarters layout, and I presume flood plains must, too. I haven't really tried to figure out jungles and forests yet, but all of the above has variant tile graphics.

@Quintillus , are you drawing overlay terrain? How?

 

[Puppeteer]

Needing better control of the map screen and running into a brick wall with map decoding has led me to look into Godot UI some more. It's pretty nice I guess. I don't have much to compare to as DHTML and console are my only previous UI experience. Generally there is a hierarchy of containers and controls all descended from a Control node much like everything else we're doing is derived from Node2D. Control nodes can scale with the window size, so toolbars across the top of the window are possible. (Or bottom, side, center, etc..) Here are a couple of videos I watched and found helpful, but the second one starts dragging after a bit.

Coming from a programmer point of view, sometimes I need to remind myself that Godot is primarily from a designer perspective even if it has incorporated scripting wonderfully. Rather, the most common use of Godot is to use the scene editor to arrange nodes, set their property in the inspector, use signals for inter-node communications, and use just a little scripting as glue where needed.

It works fine code-first, but there are two paradigms at play that raise questions on which way to code: Do I want to place a node in the scene, or do I want to instantiate it from code?

As a programmer it's much more natural for me to define and instantiate from code. I have the most control, and it's easy to see changes in version control. The scene editor makes it easy to order the nodes which is important to draw order, and my heavy-lifting map code gets dynamically added to the node tree while the UI is in the scene editor, so the map draws over the controls.

When defining and instantiating from code I always have a reference to the node. When the node is placed in the scene editor, I have to GetNode<T>("NodeName") to assign a reference to it to access it programmatically. And if the name is changed in the scene or maybe if it's moved, the code may break.

On the other hand, my workaround is GetNode'ing the control node, detaching it and reattaching it to get it to draw over the map, but I just realized that's not really cleaner than just having a node for the map and GetNode'ing it.

I think it will eventually boil down to a combination of ease of use and possible future contributors. e.g. I think coding UI controls will be a bad idea in case a future Godot UI expert/designer who doesn't code much contributes, having the UI in the scene node tree is what they're used to.

Another thing is that the Godot signals are basically an observer pattern, and we can code an observer pattern in C#, too. I think that choice will probably boil down to a definition of what Godot's purpose is, and I think it is the UI/graphics/audio engine which is separate in our use from game/business logic and file I/O.

Another observation is that although we're coding in C#, I don't think all our data is managed. I think when we instantiate Godot nodes we're creating unmanaged data in the Godot runtime, and simply removing the node from the tree and dropping its reference in C# will not free that memory, although I'm not yet sure of this. Objects not in the Godot namespace should be pure Mono and managed, though.

Edit: Huh. I wonder if it's possible to have the Godot editor itself be a new-game mod editor? I don't imagine that being the primary editor, but Oz's "dll for a civ" concept may be doable like this. It's possible to make DLC PCKs with Godot. I haven't looked into how or whether it's something a non-developer can be guided or sandboxed to do, but it's a thought. While decoding the map I'm imagining refactoring it into reusable pieces as it seems like one class could handle 4-quarters offset tiles (terrain, roads, rivers) and another handle tile-centered tiles (cities, mountains, hills, resources). Those might then be turned into custom Godot nodes that could be placed in the scene editor, say, to add a new layer to the map; perhaps a sky layer or underground layer. Just crazy thoughts for now. I'm not sure if it's easy to either offer custom nodes or to restrict which nodes are available, or to restrict which properties the user can set.

Edit: Oh, I learned the Godot way to ensure the UI gets drawn over other stuff: put the Control nodes under a CanvasLayer. That fixed the drawing order and the UI responsiveness opposed to placing a Control under a Node2D.

 

[Puppeteer]

Here's a short demo of the much-less-hideous toolbar UI and my first attempt at map scrolling. Well, my first working attempt.

I tried ScrollContainer, but that only works for Controls, not Node2Ds. This is using an invisible KinematicBody2D "player" with a Camera attached. The left/right buttons jump the "player" to the left or right, and the Camera scrolls the scene to keep the player centered.

When I resize the window you can see both the responsive toolbar resize and the map scroll to keep the invisible "player" centered.

I think endless loop scrolling is possible by drawing the map 2 or more times and using two Cameras where the non-current Camera jumps its offset as needed to seamlessly take over when the current camera nears an edge. That means drawing visible units multiple times, too, but oh well. Then again, the fog of war will hide most units, anyway. Ok that should work, but really drawing each unit 2-5 times (for really wide screens) is probably doable, anyway.

The KinematicBody2D and Camera are intended for side-scroller platformers, but the functionality for wraparound maps should work just fine at no discernable cost.

Edit: Got smooth scrolling and zooming with mouse drag and mouse wheel working. I also have a slider control for zoom.

Edit 2: Huh...I wonder if instead of a fake player and cameras if I could just have duplicate maps across two Node2Ds and just scroll & teleport the Node2Ds as needed.

 

[Blue Monkey]

Here are some quick comments on designing Terrain PCX in response to some remarks by Puppeteer. I apologize in advance if these have been already posted. It's difficult to keep up with the rapid flow of posts to find - betwixt and between all the programming discussions that are relatively esoteric for me - where I can contribute. Anyway ...

My experience working with terrain pcx has involved a lot of experimentation. I'm sure Ares knows far more than I do, but some of my observations:

• Flood Plains as displayed in-game are Desert Terrain with a graphic overlay designed to follow the pattern of the rivers. A one tile Flood Plain is basically a blank overlay on a desert tile.
• Overlay terrains such as forests, marshes & jungles are centered to the underlying tile, and can exceed the tile boundaries, up to the limit of the oblong boundaries in the overlay's pcx. Mountains & Hills pcx otoh use the same diamond-shaped boundaries as the base terrain so I suspect that their placement follows the same rules as the base terrains although I have not tested this thoroughly.
• I made a set of annotated tile pcx based on Wild Weazel's templates that I found useful for experimentation & development. Something similar may be of use in this part of the project:

• i made a similar set for the forest/jungle/marsh overlays - with those it worked better to have the labels offset toward a corner rather than centered.
• It's important to note that the terrain pcx are not aligned with the tiles as seen ingame. This graphic illustrates the pattern, with the lighter grey representing the pcx boundaries and the black lines the ingame tile borders.

Hopefully this info is of some use.

EDIT:
I believe I saw a reference to 16 terrains in an earlier post in this thread.

Based on working with Quintillus' Editor it's known that every base terrain has a landmark version. Landmark terrains are separate graphics and can - along with variations in stats - have completely different appearances from the standard ones. Grass, Plains, Desert, Tundra, Coast, Sea, Ocean + the landmarks = 14.

While the overlay variations are not each in separate pcx files each type has a standard & a landmark version. And there are separate versions based on the underlying terrain. For example, there is Plains Forest, Plains Pine Forest, LM Plains Forest, & LM Plains Pine Forest.

For purposes of converting bmp to C3 maps within his editor Quintillus combined some types of terrain. Notably the 6 water terrains are combined into 2 colors. Even so, if I counted correctly there are 36 colors used by the conversion process. So I believe that - counting the overlays as distinct terrains - there is a total of 40 terrains.

 

[Blue Monkey]

The post above has been edited to add additional information. If you "liked" it you might want to glance at it again.

 

[Puppeteer]

Every bit helps, thanks!

I was able to mostly lay out base terrain from just the PCX files and terrain IDs but couldn't figure out how to choose between the 4 or so all-grass tiles available. And then there are entire tilesets for OOO and SSS.

Then I realized what Quintillus was saying and that there were bytes telling you which file and index to use and got that working.

But I'm not yet seeing any candidate data for other file/index pickers. It seems only base terrain has that specific info. Which the more I think about it, the weirder that sounds. Why would they code tile selection differently? Actually that question may be helpful...I have a few vague ideas why now. (Like you'll definitely see huge swaths of grassland and ocean which is a smooth texture where patterns will stand out to the eye, so more effort taken to avoid distracting patterns.)

So for other terrain features I'm back to "how to pick which of many equivalent tiles" for mountains, hills, etc.. But I guess it's a problem that needs to be solved, anyway. If I recall you can place mountains in the editor but the editor picks the graphic.

I'll attack this three ways:

• I'll punt and just pick the first or a random tile of the right features to get a helpful and pretty if not pixel-exact replica
• I'll better map out the data I do know what it means to leave less candidate data for the unknowns
• I'll try to discern how the designers were thinking as far as patterns and data formatting which may offer clues

I think one of my bigger unknowns is where is it coded if a tile edge is raised? Mountains, hills, and borders have tiles with the SW and/or SE edge raised up. It doesn't always occur between two raised tiles (mountains or hills); sometimes there is a valley. That must be in the data somewhere, and I'll try to figure that out.

Edit: I had been thinking I may want to either painstakingly identify individual tiles to compare to data or code an image classifier. It just suddenly hit me I can mod the stupid tilesets with numbers or colors or something to make the process obvious. Duh.

Edit 2: And I can change the debug display to hex and/or code a bitmask image to more easily identify bits for terrain features.

Edit 3: Assuming similar design patterns, we know the bit mask for river placement, so I can code that bit pattern as dots at the corners and edges to see if other features use the same bit-to-direction map.

 

[Quintillus]

Writing somewhat quickly as I've moved recently and haven't been on CFC for awhile and there are cats to keep me busy now.

For rivers, the code I use is here. It shows how I calculate which river graphics to draw, based on neighboring tiles and some constants. It isn't 100% exact with Firaxis, as some of the edge cases (e.g. a river that's only at one vertex) don't match, but for rivers that follow a path along land, it winds up being highly accurate.

For hills/mountains, I never did figure out exactly how Firaxis does it; they have some algorithm that is consistent but I never reverse engineered. Instead, I came up with my own that makes sense based on the built-in graphics, but has less non-random randomness when you have multiple mountain chains in close proximity. The main code is here, with the code for determining which of the graphics within a hill/mountain/volcano/etc. PCX file to use being here.

I think one of my bigger unknowns is where is it coded if a tile edge is raised? Mountains, hills, and borders have tiles with the SW and/or SE edge raised up. It doesn't always occur between two raised tiles (mountains or hills); sometimes there is a valley. That must be in the data somewhere, and I'll try to figure that out.

Best of luck! I tried to figure this out for a long time back in 2010, and thought I had something kind of matching, but with more test data realized it didn't match. What you see linked above is what I wound up with after I gave up on matching the "It doesn't always occur between two raised tiles (mountains or hills); sometimes there is a valley" part. Perhaps someone with a different algorithms background would recognize what Firaxis is doing better than I was able to.

 

[Puppeteer]

I think I just had a "Eureka" moment and want to note it somewhere before I forget: For map wrap scrolling I want/need to duplicate graphics so I can swap the off-screen version's position to 'catch' the edge of the other node.

I've been puzzling how to do it, but I *think* if I just create a new Node2D and assign it the value of the other node I'll have one set of objects drawn twice. I'll have to try it and be sure, and I'll have to see if I can test if there are performance hits.


I haven't done any coding since last post. It's a busy week: new drivers license, license plates, birthday, and getting my first COVID vaccine shot tomorrow morning. So lots of running to the copy shop for prints and copies and generally taking care of business.

I'm also mentally refactoring how I want the map to work. I think I can simplify it and separate responsibilities where they should be. (e.g. Which part of code needs to know the hard-coded filename and which needs to be more broadly usable.) And I think the tile draw routine will use consistent coordinates while the view is shifted 64 pixels left for the offset terrain tiles; more reusability that way and a door for replacing, adding, or removing terrain and feature layers later.

Edit: Looks like I can't quite do my idea. Nodes can't be in more than one place in the scene hierarchy. It's their properties that are not unique objects when duplicated. But I don't think changing the tile ID will propagate to the copied node. Maybe I can split the map into vertical pieces and wrap without duplicating, but that would put a max view width. Or we could just duplicate one section as needed and perhaps refresh it frequently which is probably better than refreshing a whole duplicate map. Or maybe observables can help here.

Edit 2: I may be able to use ViewPort nodes; place the may layer Node2Ds under one, make a peer one and copy the texture. Huh, a ViewPort may also work as a mini map using a ViewPortContainer?

 

[WildWeazel]

My coworker tried to explain one way of doing something like this in Unity with offset cameras rendering different layers to make it look like the opposite sides are adjacent. I didn't quite get it but maybe that will spark something for you.

 

[Puppeteer]

rendering different layers to make it look like the opposite sides are adjacent.

Huh. My initial reaction is that only one Camera2D can be active at a time, but this made me think about that some more. Can there be two active cameras but in different Node2Ds that can be on screen at the same time? I'll find out.

I still haven't poked at the code in a while, but I'm doing more Civ3 YouTube videos and thinking of ways to use my stuff so far to make custom graphics and animations to add interest to the videos.

I'm also increasingly eager to completely port CIA3 over to Godot. It could easily be as useful as CAII used to be, but it's really ugly and clunky to use as-is.

 

[Puppeteer]

I'm coding a bit again. At the moment I'm focused on reading the save & biq files. Or rather exposing their data in a convenient way.

I forgot I actually had clua–my Go-based Lua Civ3 save reader–parsing as much of the file as it does. It even does cities!

So I'm porting over the object model to C#. The values are just getters that read from the file data (decompressed and in-memory) directly. e.g. SavData.Game.CityCount is the world count of cities from the GAME section.

This should work for all the use cases I expect it to get: as an importer for the new game and for may projects like cia3 or civ parade.

I'm also kicking the tires on MoonSharp, the C# Lua interpreter. That's meshing really well with my object model & getter tactic. I can just register and expose e.g. the GameSection class and then just insert the SavData.Game object into Lua as the global game. So you just instantiate Civ3Script with a path to a save file (which inherits from MoonSharp.Interpreter.Script) and get back a Lua environment with the globals loaded to provide access to the C# getters with the save data. print(game.CityCount) . I don't expect this particular environment to be useful for the new game, but it will help me in decoding unknown parts of the sav and biq files. And it shows how we can expose native C# classes for modding purposes. There are some security concerns, but MoonSharp can be sandboxed, and if you don't expose unsafe methods in registered classes, it should be fine.

 

[WildWeazel]

Can you integrate MoonSharp into Godot such that a compiled game can run external Lua scripts? I don't know how Godot's build pipeline works.

 

[Puppeteer]

Can you integrate MoonSharp into Godot such that a compiled game can run external Lua scripts? I don't know how Godot's build pipeline works.

I don't see why not. I haven't mentally mapped it out, but I'm already using the Godot FileDialog to pick sav and pcx files then my C# code converts the file data to Godot assets.

Lua is a purpose-built embedded language. It's meant to not stand on its own but to integrate with existing programs. So a the very base level:

using MoonSharp.Interpreter;

Script MyLua = new Script();

string LuaCode = @"
  print(""Hello, Lua"")
";

MyLua.DoString(LuaCode);

But once you have that MoonSharp.Interpreter.Script instance (MyLua), you have a Lua environment, and you can set global Lua vars from C#. And all the object/data/function types are first-class citizens in Lua, so that can be strings, tables, numbers, functions...whatever.

So whatever the C# API is can either be proxied or exposed directly inside the Lua environment, and scripts can execute in it.

So far I'm just inheriting the Script class and populating the Lua globals in the constructor.

Spoiler MoonSharp code exposing C# classes :

In SetGlobals() is where I set Lua's "install_path" variable to the Civ3 path (something I did in my Go project but am not sure it's relevant anymore) and expose 3 classes as "game", "wrld", and "sav", the last of which gives direct access to the QueryCiv3Sav.Civ3File instance for the loaded save.

The classes exposed have to be explicitly registered for security reasons.

using System;
using MoonSharp.Interpreter;
using ReadCivData.QueryCiv3Sav;

namespace ReadCivData.LuaCiv3 {
    public class Civ3Script : Script
    {
        private SavData SavFile;
        public Civ3Script(string savPath)
        {
            SavFile = new SavData(savPath);
            RegisterUserData();
            SetGlobals();
        }
        private void RegisterUserData()
        {
            UserData.RegisterType<Civ3File>();
            UserData.RegisterType<SavData.GameSection>();
            UserData.RegisterType<SavData.WrldSection>();
        }
        private void SetGlobals()
        {
            Globals["install_path"] = ReadCivData.UtilsCiv3.Util.GetCiv3Path();
            Globals["sav"] = SavFile.SavFile;
            Globals["game"] = SavFile.Game;
            Globals["wrld"] = SavFile.Wrld;
        }
    }
}

 

[Puppeteer]

I'm gaining some interesting new insights on scripting languages. I originally tried out Lua with my Go-based c3sat and really liked it. But a big part of its then-immediate use to me to mass-analyze sav or biq files, either hitting multiple locations in one file or comparing across all file in a directory.

I had actually tried to use PowerShell to drive my tools, but it was clunky. I even tried making Go-based DLLs, but I ran into huge roadblocks exporting safe functions in a managed environment to unsafe C interfaces, then use a different managed environment to talk to the unsafe C interfaces.

Well, now I'm using C#, and I'm making libraries, anyway. I quickly found that using PowerShell to quickly test new code was much faster than having a dotnet console program import & compile. Because C# and PowerShell are in the same managed ecosystem and can directly interface with each other.

I suddenly realized today that *any* .NET language should easily be able to use our libraries, including IronPython.

And suddenly Lua is a lot less useful on the dev side. But it's still my favorite as an in-game mod script. I don't think embedded PowerShell is even a thing, and I'm almost certain there's no sandboxing for it. But I'm kind of eager to check out IronPython both as a dev-side utility script and if it can be sandboxed and embedded into an app.

Spoiler A PowerShell Test Script :

I just keep changing this script up to make sure the sav reader is producing sane results. I was working on the city section most recently.

# Must first run `dotnet build` in the type/project folders
Add-Type -Path ('../UtilsCiv3/bin/Debug/netstandard2.0/ReadCivData.UtilsCiv3.dll')
Add-Type -Path ('../QueryCiv3Sav/bin/Debug/netstandard2.0/ReadCivData.QueryCiv3Sav.dll')

$Civ3Path = [ReadCivData.UtilsCiv3.Util]::GetCiv3Path()

"4000", "3950" | ForEach-Object {
    $SavPath = "${Civ3Path}/Conquests/Saves/Auto/Conquests Autosave ${PSItem} BC.SAV"
    $DefaultBicPath = "${Civ3Path}/Conquests/conquests.biq"
    $Sav = New-Object ReadCivData.QueryCiv3Sav.SavData($SavPath, $DefaultBicPath)
    
    $Sav.City.Name # RawBytes | Format-Hex
    $Sav.City.StoredFood
    $Sav.City.StoredShields
    $Sav.City.CitizenCount
    $Sav.City.RawBytes | Format-Hex
    "========"
}

 

[WildWeazel]

Have you come across this? https://github.com/perbone/luascript
I hadn't given much thought to third-party languages, but I suppose the advantage of Lua is that it's already so popular for modding other games. (Civ 5 & 6...) Plus it doesn't sound like it would require any dependencies to write mod scripts with a precompiled game.

 

[Vuldacon]

WildWeasel... it was developed using Linux and Mac but he did say he would help solve any problems for Windows if there are any.

 

[Puppeteer]

I feel like we're on different pages here. I'm sure it's my fault as I do things weirdly.

I've been developing on the Mac and touching Linux, but everything I've done works fine on Windows.

The luascript project appears to aim to replace GDScript for those who've used Lua but don't want to learn Python/GDScript or C#.

As I understand our "new game" direction, most game logic is in pure standalone C#, and we're just using Godot for the graphics & UI. This is a bit of a contrast to how most of the Godot community approaches Godot; they do as much as they can in the Godot editor then stitch advanced pieces together with some GDScript or C#. But a 4x game has so much internal state and logic that it doesn't really make sense to me to try to start from Godot and stitch bits together.

As far as Lua, maybe think of it as an advanced user API, possibly sandboxed, to your code API. I don't see us making any game logic with it, but allowing users/modders to call methods, retrieve and submit data without having to compile the project. Just a text file at most.

I don't have a coded example of how I see us using it yet; they way I've been using it is getting me familiar with how to implement it and how to better automate my own dev & test work without recompiling. But for the moment PowerShell seems more suited to that particular use case.

Once we get some actual game logic coded–any logic at all–maybe I can better illustrate what Lua can do for us. So far I'm just working on data & media conversion and no actual game logic.

 

[WildWeazel]

No I get that, I didn't realize Luascript sought to replace GDScript. I thought it was just implementing the API in Lua for external scripts to use. But I haven't looked into it beyond the Readme.