C# SAV/BIQ/media library

Puppeteer

Emperor
Joined
Oct 4, 2003
Messages
1,687
Location
Silverdale, WA, USA
https://github.com/myjimnelson/read-civ-data - A collection of cross-platform C# (.NET Standard 2.0) libraries to read Civ3 SAV & BIQ files (at a very low level so far) and to convert Civ3 PCX and FLC files...to I'm not sure what yet, but maybe Godot or Unity assets.

TL;DR: What's this for? What does it do? Meh, I just made a thing and wanted to share in case anyone finds it useful. Not yet sure what it's for, and it's not in a form usable by a non-programmer yet.

New year, and new tech project inspirations. This one is really just a collection of functions with no coherent end goal for the moment. MIT-licensed but also using Apache-2.0-licensed dll dependencies for decompression and image support.

I've been playing around with reading the save file format for a while now with c3sat/cia3 and briefly toyed around with some civ-like game mock-ups with otda3. I've used Python, C# very briefly, JavaScript, and a lot of Go, with some poking at Godot game engine and its Python-like GDScript.

Well, now I'm playing more with C#. I gravitated to the other languages because they excel where I usually do my thing: system management, devops support, and web utilities. But an actual device app (pc/phone/etc) would usually make a lot more sense in C#, and others have made Civ3 utilities with C#, so maybe others might actually find these libraries useful.

The SAV/BIQ reader is basically a reimplementation of the core of my c3sat/cia3 save-file-reader Go programs. At this point it would be trivial but time-consuming to reach feature parity with the data I've extracted using c3sat/cia3, so I'm probably not doing that until I have a good destination format and use for the data.

The PCX reader is working fine but isn't in a really useful form yet. I have a test function that reads a hard-coded file location and saves the PCX as 'out.png'. An array of transparent palette entries can be passed to the real function for, well, transparency. I think my eventual hope here is to have this create Unity and/or Godot tile sets from Civ3 files.

The FLC reader does pretty well but needs some work as I don't have any Civ3-specific decoding yet. I got something pretty good then realized the colors are off–not just the civ colors–and I don't seem to be reading all the frames yet. I'm getting 80 for units, and I think there are 88, and the very first key frame seems to be wrong or blank. Again I have just a test function that saves each frame as a png, but I think I want to make Unity or Godot animated sprites from the FLCs.

Not that I've ever used Unity. I did toy with Godot a bit, but I am by no means proficient.

Attached is a FLC frame of a running warrior with transparency and transparent shadows working. The lime green stand-in civ color is expected, but I don't recall the warrior's skin being purple?
 

Attachments

  • out-41.png
    out-41.png
    2 KB · Views: 52
Last edited:

Quintillus

Archiving Civ3 Content
Moderator
Supporter
Joined
Mar 17, 2007
Messages
7,483
Location
Ohio
Interesting. I may find the FLC reader quite useful for the possibility of converting it to Java and adding perhaps adding some level of animation support to my editor - something I've thought about over the years but never pursued due to lack of a way to read FLCs or much knowledge about their format. I should probably also study your SAV/BIQ code some... you've been making progress in that area far more quickly than I have over recent years, and I suspect more efficiently as well.

I'll also have to take a look at your decompression work; an MIT-licensed decompression (and ideally compression) function is great to have, as it can be natively integrated into utilities, unlike the solutions Steph and I have used, which are external to our editors.
 

Puppeteer

Emperor
Joined
Oct 4, 2003
Messages
1,687
Location
Silverdale, WA, USA
Thanks! Both C# and Java are object-oriented managed code environments, so I imagine the translation should be fairly direct.

As far as decompression, I do have a Go-based MIT-licensed decoder, but there was an Apache-2.0-licensed C# library for blast, so I'm using that so far and haven't spent the time to port my own implementation over. I haven't really figured out yet what the impact of dll-linking MIT to Apache 2.0 code has on reuse...they're pretty similar, but copyright law is not for the feint of heart. But it can trivially be replaced or omitted.

I think I'm about to shed the Apache 2.0 image library, though; I was just using it to convert a raw pixel array to a png, and now I'm toying with going straight from PCX/FLC to Godot's internal image format. I expect Unity will have a comparable internal image format.
 

Puppeteer

Emperor
Joined
Oct 4, 2003
Messages
1,687
Location
Silverdale, WA, USA
I now have an examples/GodotCiv3 in the repo that can read a terrain file–well one specific terrain file–make a tiled sprite out of it and randomly spew tiles out for a 'map'. It's basically what I did in the otda3 thread with Godot, but this time it's in C# and I also have a SAV reader in C#, so there's a (slow) path to showing a rendered map from a saved game. I'm not sure that's my next destination, though.

Using C# in Godot has been a bit of an adjustment for me. First, Godot pops up a warning that C# support is alpha quality. And while the objects, properties, and methods are mostly the same between GDScript and C#, being C# instead of Pythonic necessitates some changes that either aren't especially well documented or I just haven't found the right places yet. Finally, I've been using dotnet core targeted to .Net Standard 2.0 while Godot uses mono targeted to .NET Framework 4.5.1. The current result of this is my VSCode environment won't give me code hints, completion, or error detection unless I add a reference in the csproj. But if I do that, Godot loses its mind. So I can either have code help or a working program, not both at the same time.

I'm sure the latter is fixable in a way obvious to more experienced dotnet coders, but I'm just rolling with it for the moment.
 

Attachments

  • Screen Shot 2021-01-08 at 3.30.17 PM.png
    Screen Shot 2021-01-08 at 3.30.17 PM.png
    1.4 MB · Views: 60

WildWeazel

Carthago Creanda Est
Joined
Jul 14, 2003
Messages
7,247
Location
%CIV3%\Conquests\Scenarios\
Really great stuff. I've never been much of a graphics guy but IIRC Firaxis put some extra data in the header(?) of the FLCs to account for civ colors which may explain the bad indexing. But maybe you accounted for that, I can't really tell from your code. If you can load an arbitrary FLC into Godot and have it look identical that would really be something.

I'm surprised to hear about the C# issues. I was under the impression it had first class support. I assume you're using the latest 3.2. Also do you know where the black edges are coming from? Is there a misalignment in the source file?
 

Puppeteer

Emperor
Joined
Oct 4, 2003
Messages
1,687
Location
Silverdale, WA, USA
Thanks! The code isn't pretty; I was alt-tabbing between the editor and a generic FLC format description and pulling the minimum info needed, one step at a time. After I got it working as it does is when I started searching the forums, and I see a few discussions on how it's different, so it will just take a bit more trial and error I think. I believe I can get it to look identical; all the data is there. That's just grinding. The challenging part will be figuring out how best to handle civ colors and maybe smoke, but I have some ideas. I just need to play around with Godot and Unity some more–or at all in the case of Unity–to figure out what format makes sense for run-time color changes there.

There's a 128-byte header, and CivIII apparently stuffed some custom values into the 'reserved area' of the header due to it really being 8 animations jammed into one file for units. But I think I'd rather see an animation in an engine to proceed further instead of pooping out 80+ pngs to disk for each FLC.

I'm using v3.2.2. I was a bit surprised by the 'alpha' warning, too. I don't really think I've had any Godot problems aside from confusion which may be inadequate documentation or it may be me and inexperience.

I literally just solved the VSCode issue: the C# plugin was apparently set to use 'internal mono' instead of the system mono which I was surprised to find I had installed. (With Godot, maybe?) So to my surprise my VSCode has apparently been using mono all along, even if I've been compiling other projects with dotnet core. It was just using a segregated version that didn't have the 4.5.1 assemblies. I switched it to 'always' use system mono, and the code helpers work great now, and Godot is happy.

I had to figure that out on my own; Google was no help, but I finally realized the problem must be inside VSCode or an extension; I thought I had to switch it from dotnet core to mono, but apparently I had to switch it from internal mono to system mono.

I'm presuming full Visual Studio and the mono IDE (Mono Developer?) wouldn't have this problem.

Edit: I'm presuming the gaps are because those tiles weren't supposed to be next to each other. At some point I'll try to lay them out 'correctly', and I'm hoping the gaps disappear.
 

WildWeazel

Carthago Creanda Est
Joined
Jul 14, 2003
Messages
7,247
Location
%CIV3%\Conquests\Scenarios\
@Quintillus have you seen this code? It has a FLC reader in Java https://forums.civfanatics.com/threads/civilization-game-toolkit-in-java.116293/

Ah that explains it. All of these cross platform tools use Mono since Core was not fully compatible. I'm hoping they'll start to adopt .NET 5 this year.

Sprite sheets allow a whole animation in one image, and I thought multiple directions but I'm not immediately seeing how to do that in Godot. Civ colors are an interesting problem. Quintillus had the idea of listing the replaceable colors in the unit metadata (ini or equivalent) though I'm not sure how that would work out with (original) full color images.
 

Puppeteer

Emperor
Joined
Oct 4, 2003
Messages
1,687
Location
Silverdale, WA, USA
My early ideas on civ colors are a separate layer, perhaps grayscale that would be used as a multiply or similar mask...it's a vague idea, and I'll need to figure out how that works in the display engine framework, if it works at all.

And I actually used a shader for civ color when I first toyed with Godot & GDScript. But it was a bit hackish as I had to set the pixel alpha to 0.9 as something the shader could key on. And it's been a while, but I don't think having more than one civ color via shader seemed reasonably likely. Here's an odd demo movie I made then where I was live- changing the civ color with a shader: https://lib.bigmoneyjim.com/civfan/popheaddemo.mp4

I have yet to play with sprite animation, but soon I think.
 

Quintillus

Archiving Civ3 Content
Moderator
Supporter
Joined
Mar 17, 2007
Messages
7,483
Location
Ohio
I hadn't seen that FLC reader in Java. It's unfortunate that it doesn't appear to have gained much traction, as based on what the thread says it could have accelerated utility development back when C3C was the latest thing.

Alas, for my purposes, I'm afraid it won't help as it's GPL-licensed, and my editor is MIT and MPL licensed (Mozilla, not Microsoft). I prefer not to GPL license my work due to the restrictions that come with doing so.

Apache and MIT should mix well, however. While I can't recall the precise difference, I'm about 99% sure that they are compatible with each other.
 

Puppeteer

Emperor
Joined
Oct 4, 2003
Messages
1,687
Location
Silverdale, WA, USA
FYI, I refactored my Flic reader today and added some comments. It now requires instantiating an object and provides access to the palette, width, height, and a jagged array of Images, each image being a byte array. I plan on adding another dimension to the array to split up each direction. The library itself no longer uses the ImageSharp image library as it doesn't make PNGs; I made two example console apps in the repo that use ImageSharp and my PNG & FLC readers to make PNGs, though.

If you're looking at my Flic code, I think some of the variable names are confused, or I was confused while coding. Frames, chunks, subchunks may in places be misnamed, and then words vs packet naming is confusing. At some point I hope to go back and change the variable names to better match the format docs.

I've also worked around the first-frame problem and am reading all 88 frames, although there is a hackish bit in there I don't like. But I'm getting all 88 frames formatted correctly, albeit with no external palette or civ color yet, so the warrior is still purplish.

Attached is a gif of all 88 frames, but the transparency didn't translate from 32-bit png to 8-bit gif. The transparency and shadow is there in the pngs; this is just to show I got the frames extracted. (made with ffmpeg -i out-%02d.png warrior.gif)

Edit: Improved gif w/transparency and solid blue shadow, thanks to Stack Overflow
ffmpeg -i out-%02d.png -vf palettegen=reserve_transparent=1 palette.png
ffmpeg -framerate 15 -i out-%02d.png -i palette.png -lavfi paletteuse=alpha_threshold=1 warrior.gif
 

Attachments

  • warrior.gif
    warrior.gif
    48.8 KB · Views: 34
  • warrior.gif
    warrior.gif
    53.4 KB · Views: 35
Last edited:

Puppeteer

Emperor
Joined
Oct 4, 2003
Messages
1,687
Location
Silverdale, WA, USA
Oh, I was doing PCX conversion wrong, and that's why there were gaps in the random map. I was coming up a pixel short in each direction, and when cutting the terrain file into a 9x9 grid, some of the tiles were too small. Fixed.

I discovered that while trying to use the TileMap functionality of Godot to place tiles instead of placing them as sprites. I'm having a horrible time of that so far. Well, programmatically; I can do it in the editor easily enough.
 

Attachments

  • Screen Shot 2021-01-10 at 12.53.07 PM.png
    Screen Shot 2021-01-10 at 12.53.07 PM.png
    1.4 MB · Views: 45

WildWeazel

Carthago Creanda Est
Joined
Jul 14, 2003
Messages
7,247
Location
%CIV3%\Conquests\Scenarios\
That looks nice (aside from the obvious)
I just came across this: https://michagamedev.wordpress.com/2018/02/24/181/ I'm not familiar with autotiling and I don't know how Civ3 selects tile images, but it seemed relevant. I'm trying to find information on wrapping tilemaps, but that doesn't seem to be something that's supported out of the box or discussed much.
 

Puppeteer

Emperor
Joined
Oct 4, 2003
Messages
1,687
Location
Silverdale, WA, USA
I'll take a look at the article. Quintillus described tile placement in my otda3 thread, but unfortunately the image link is broken now. I think I remember it well enough, though, to help make sense of the description.

As for atlasing, I punted and just looped through and defined each tileset tile for now.

Then I got an AnimatedSprite working in code relatively quickly. It's only one direction of one flc of one unit, but it's a start:


Does anyone know which of the palettes under Art/Units/Palettes/ goes with the warrior? I tried a few at random but haven't seemed to find the right one yet. For that matter, what's with ntp vs otp and the numbering?
 

Puppeteer

Emperor
Joined
Oct 4, 2003
Messages
1,687
Location
Silverdale, WA, USA
Based on the article WW linked, I presume that Civ3 is a 3x3 bitmap...but it's not active vs inactive; there are three adjoining tile types in this one file alone. So I'm slowly getting the idea, but there are more choices for each tile.

Looking at the article and Qunitillus' post, I guess I need to subdivide each visual isotile into four pieces as the center of what seems to be a tile in the pcx file is really the corner where four tiles meet. Or maybe I don't need to subdivide but just need to realize the shapes in the tile...well there is no spoon...I'm confused and tired now.

As for what to do with my map tiles and animated unit sprites, I'm thinking of a Civ Parade / Civ Marching band where a user can use Lua to spawn units and move them around, preferably with a soundtrack. C# has a Lua implementation I think I can use, and somebody even made a text editor in Godot, so it looks like I could have in-app scripting ability, perhaps.
 

Puppeteer

Emperor
Joined
Oct 4, 2003
Messages
1,687
Location
Silverdale, WA, USA
Yeah, probably. I notice the palette in the FLC file has lots of 0,0,0 entries which I'm sure isn't the end product.

As far has how to handle civ colors, when talking about importing Civ3 graphics I'm starting to lean more towards just converting from native every time, and the civ-coloring can be done in that process while we still have the palette and color index info. That would seem inefficient, but on the other hand it seems pretty quick. How many unit sprites will be drawn on the screen at the same time, and at what speeds? Not many, and not fast I'm thinking.

As for map tiles, I was tired and confused last night. The visual isotiles in the pcx file don't need to be further subdivided; they just need to be centered on the corners of where four logical map tiles meet. So it's a 2x2, but it's not a bitmask; it's a 2x2xn where n is the number of base terrain options.

I was thinking that not all tile types can go next to each other, but mostly they can? Not sure if desert and tundra or tundra and plains can be next to each other, and sea and ocean tiles can't meet land, but otherwise I think everything can be adjacent?

So I'm not sure the built-in autotile bitmask will work, but the concept may help me figure out how to organize and place the tile graphics with derived logic.

Oh, I just connected with some math. The terrain pcx I've been messing with has all the plains, grass, and coast combinations, so 3 possibilities. A 2x2 grid with 3 possibilities is 3x3x3x3 which is 81, and the pcx is a 9x9 (=81) grid of tiles. Ok, I'm seeing how the indexing will work if I do it myself, but first I'll make sure a Godot 'bitmask' is truly a 1-bit structure or if there are higher dimensions available.

So many things I want to do, but I may have to slow down. I'm nomadic and am leaving my current place Friday and will be slow-traveling for a few weeks without more than a couple or few days in any one spot.

Edit: Oh, things are starting to click into place! I've mentioned several times that the logical layout of the map isn't isometric but a fake checkerboard-like isometric. The non-map coordinates are the graphics coordinates! So for e.g. 2,1 which is not a 'real' tile, I can look at 1,1, 0,2, 3,1, and 2,2 to get the info needed to choose a tile. I was imagining some sort of convolutional process, but the index is baked into the map design.

And it doesn't have to be perfect from the start; a default tile can be bright red so any layout can be tested while figuring it all out.

Edit 2: Looking at tutorials and documentation, the UI seems to be a 1-bit-only bitmask, but the code calls use a Vector2 which is int, int. And now my brain is broken again. I initially thought that was promising, but now I'm realizing that's not enough values. On the other hand, I can probably game this system by devising my own key coding as it seems to be using that Vector2 as a key.

Edit 3: I think I found an error in the docs. Most of the bitmask values are ints, but the one I was looking at said "void autotile_set_bitmask ( int id, Vector2 bitmask, int flag )", but I'm pretty sure that's a typo. I think the int is the bitmask and the Vector2 is the coordinate.

And yeah, it appears that I can game the 'bitmask' system as it appears to be using the
bitmask' as a key only, so I should be able to assign and select using my own int mapping. 2x2x16 would be 64 bits of info. ... but I feel like I just mathed wrong. 16 choices is 4 bits, so a 2x2 grid of 16 possibilities is 16 bits I think?
 
Last edited:

Puppeteer

Emperor
Joined
Oct 4, 2003
Messages
1,687
Location
Silverdale, WA, USA
Well, I think I have the civ color part sort of working, but I still don't know why the warrior is purple. I even fired up a game to make sure he's not actually purple. He's not.

By the way, having Steam install Civ3 over a wifi network share to an rPi with a usb drive...is a bad idea for performance. (My windows laptop's hard drive was full.)

At first I tried copying the first 64 palette entries from the palette file (ntp01.pcx) and then started at the beginning of the unit FLC palette, but that made the green monster. Next I just used the first 64 palette entries from ntp01.pcx and copied the rest from their same position in the FLC palette. (I've quadrupled the unit scale for better seeing the colors. Also I currently don't have Godot doing anything with the shadow colors.)
 

Attachments

  • Screen Shot 2021-01-11 at 3.04.16 PM.png
    Screen Shot 2021-01-11 at 3.04.16 PM.png
    46.2 KB · Views: 35
  • Screen Shot 2021-01-11 at 3.07.19 PM.png
    Screen Shot 2021-01-11 at 3.07.19 PM.png
    67.5 KB · Views: 36

WildWeazel

Carthago Creanda Est
Joined
Jul 14, 2003
Messages
7,247
Location
%CIV3%\Conquests\Scenarios\
Last edited:

Puppeteer

Emperor
Joined
Oct 4, 2003
Messages
1,687
Location
Silverdale, WA, USA
That helps a lot! I now see the pattern within the file with the hard-edge coloring. As a base 3 number system, the right corner is the lest significant digit, then the bottom, then the left, and the top is most-significant! (Flowing from top-to-bottom, then left-to-right.)

Ok, I guess I know what I'm doing next.
 

Puppeteer

Emperor
Joined
Oct 4, 2003
Messages
1,687
Location
Silverdale, WA, USA
:)

Godot didn't seem to have a function to actually pick a tile by the bitmask int, but by that point I was ready to code the lookup myself with a hash table.

This is a random placement of the 3 tile types, and I can pick the right graphics. The top and bottom edges aren't done, but I think the hard part is figured out.

(Oops, my Mac screenshots are huge)
 

Attachments

  • Screen Shot 2021-01-11 at 8.12.02 PM.png
    Screen Shot 2021-01-11 at 8.12.02 PM.png
    859.9 KB · Views: 51
Top Bottom