C# SAV/BIQ/media library

Puppeteer

Emperor
Joined
Oct 4, 2003
Messages
1,687
Location
Silverdale, WA, USA
I haven't done much on this in the past few hours, but I did fix unit shadows and enable a smoother magnification of the 4x-sized unit. (There is a flags parameter for texture creation; I was passing 0, but passing 7 makes it look better when scaled up or down.)

I'm not sure which direction I'm heading next. There's lots I want to do, but I probably need to generalize the code I have to allow for all terrain types, all units, all animations before running off in other directions. And I kind of want to start with Unity and reach feature parity to see which code I can separate from the display engine and what has to stay specific to Godot or Unity, but I've still done nothing with Unity but open it quite a few times and stare at the window before going and playing with something more familiar.

 

Puppeteer

Emperor
Joined
Oct 4, 2003
Messages
1,687
Location
Silverdale, WA, USA
So many things I want to do, but I may have to slow down. I'm nomadic and am leaving my current place Friday and will be slow-traveling for a few weeks without more than a couple or few days in any one spot.

For reasons unrelated to Civ3, I've extended my stay at my current location for at least a couple of weeks. And then I did nothing with this project most of the day.

But I did work out how I want to move forward with encapsulating the code: I'll have a non-Godot, non-Unity class that uses interfaces to animate and move a unit. The class will read the unit's .ini file, convert the FLC files and ready the palettes and then use interfaces to convert that info into the display framework's implementation. And as I said that or another interface will have animate and move methods to be implemented in the framework.

The map will be similarly decoupled from the specific display framework, although I haven't really thought it out yet.

I did try out the ini-parser NuGet package which is thankfully MIT-licensed. And I was able to use it to parse the Warrior.INI file quite easily.

Spoiler Partial output of ini-parser test script :
Code:
KeyName : BLANK
Value   :

KeyName : DEFAULT
Value   : WarriorDefault.flc

KeyName : WALK
Value   :

KeyName : RUN
Value   : WarriorRun.flc

KeyName : ATTACK1
Value   : WarriorAttackA.flc

KeyName : ATTACK2
Value   : WarriorAttackB.flc

I used the following PowerShell Core code as a quick test, but this is trivially reproducible in C#:
Code:
# Testing the ini-parser NuGet package against unit ini files

$NugetPath = '/Users/jim/.nuget/packages'
$Civ3Root = '/Volumes/minio-storage/civ3'
$IniPath = $Civ3Root + '/Art/Units/warrior/Warrior.INI'

Add-Type -Path ($NugetPath + '/ini-parser/3.4.0/lib/net20/INIFileParser.dll')

$parser = New-Object IniParser.FileIniDataParser
$data = $parser.ReadFile($IniPath)
$data.Sections["Animations"] | Select-Object KeyName, Value | Format-List
 

Ozymandias

In Terra Fantasia
Supporter
Joined
Nov 5, 2001
Messages
10,285
Location
The lone and level sands
I'm decades past hands on programming, but I'm wondering if antal1987's investigations would be helpful. He posted a lot of info and code,both here and on github.To quote: "I obtain C++ code in runtime when I locate a function and use Hex-Rays plugin and it's main feature of decompiling." I know that you, yourself, @Puppeteer, was well acquainted with antal1987's work - yet no one has compiled a "library" of them. If you'd find it helpful, I have somewhere between 15 - 20 bookmarks of his. (And, I start every day by writing, "I don't have OCD," "I don't have OCD," "I don't have OCD," 100 times :yup: )
 

Puppeteer

Emperor
Joined
Oct 4, 2003
Messages
1,687
Location
Silverdale, WA, USA
I have his GitHub repo forked, and I've referred to it a fair bit when decoding SAV files as the on-disk format and in-memory format at least echo each other strongly where they don't match exactly. But sure, I'll stash the bookmarks somewhere and take a look at them.


Nothing to show for today; only code that doesn't do anything visual yet. But the pieces are coming together for a unit display class that should have some nice levers to pull once it's all together. I'm reading the Warrior.INI file and reading all the animation FLCs listed within. And then not storing it or converting it all yet, but I'll get there. And I'll be able to read any unit ini.

I realized the interface didn't make as much sense as an inherited class with overloaded members. (Yes, I'm still talking about C#.) Then all the Godot/Unity script should need to do is instantiate a child, call the base constructor, and then convert the byte array to native sprite formats. And override the methods to pick the animation, direction, play, and stop.

I don't think I'm too far from scattering a bunch of spastic warriors all over the screen, running, idling, attacking, and dying in place repeatedly. Movement itself should be easy, but there's currently no time or event structure to decide to move, how far, and when to change behavior. Yet.

Edit: Oh, I did need to install nuget and do a nuget restore before Godot could see the ini parser library. I still don't really have the hang of how dotnet and mono behave differently.
 
Last edited:

WildWeazel

Carthago Creanda Est
Joined
Jul 14, 2003
Messages
7,247
Location
%CIV3%\Conquests\Scenarios\

Puppeteer

Emperor
Joined
Oct 4, 2003
Messages
1,687
Location
Silverdale, WA, USA
I recall (and can see in the code and thread ) that I tried using the shader and decided it wasn't going to work for lots of units. But I don't really recall why, and apparently there is a shader parameter ability, and each instance has its own shader attached. If I can either use a paramater or programatically copy and color-change the shader it might be doable.

So it's worth another look. I was keying on alpha 0.1 by itself, but I guess doing full magenta plus alpha 0.1 would make it harder. (edit: I mean harder to accidentally change the wrong pixel)

Oh, another possible issue is that in using a shader I need to shade...what, 32-40 civ color shades per unit? Or maybe it's 16-24. Doable for palette conversions...but I'm getting ahead of myself on future potential issues.

Then again, getting ahead of myself is fun. Converting from 256-color palettes, it's easy enough to translate the civ colors by instance or by shader-keying. But if someone were to want to make a 32-bit png animation to bring into CIv Parade, how would they code the civ color portions? I'm thinking maybe a second layer. But I just did a quick Google, and I'm not sure there's a good universal candidate format that supports 32-bit color and layers.

Oh well, no matter; it would be a different class–maybe a child or peer class, or an interface–anyway. So no need to think too hard on that future.

For the moment I'm loading the ntpxx.pcx palette files quickly enough. I can always optimize later.
 
Last edited:

Puppeteer

Emperor
Joined
Oct 4, 2003
Messages
1,687
Location
Silverdale, WA, USA
I made progress on the unit class. It now reads warrior.INI and converts all the FLCs for the actions in the .ini. The Godot object is a child of that class, and it converts the palette and index byte arrays to an AnimatedSprite with a number of SpriteFrames animations. So I can instantiate a bunch of them and animate any action, any direction.

Right now the unit path is hard-coded, but I'll work on that. I tried hard-coding some other units, but for some reason I'm getting access denied errors which makes no sense. (I'm developing on a Mac and accessing Civ III files over a network share at the moment.)

At first I just ran the import for each instance, so 36 times...that was noticeably slow to start, so I'll need to optimize. For now I'm just importing once and then duplicating the AnimatedSprite object.

I can supposedly grab any civ color, but I haven't connected all the parameters for that yet.

I call this warrior calisthenics:

I call this warrior mayhem:

They're synchronized only because I'm starting them all at the same time, and they have the same number of frames and are running at the same speed.

All the code is on the GitHub linked in OP.
 

Puppeteer

Emperor
Joined
Oct 4, 2003
Messages
1,687
Location
Silverdale, WA, USA
Should've gone to sleep an hour or two ago, but it was really bugging me about the file error. Turns out the warrior INI had three animation keys that don't exist in other INI files, and I was testing for zero-length string, but the nonexistent key returned null. So it was trying to read a directory as a FLC file, but the error seemed to indicate the INI file. Weird.


The purple tinge is very noticeable in the next two.

Interesting that most of the smoke is red, but in the upper-left there is some correctly-colored smoke. Weird. (edit 2: Oh, the smoke over the ship/unit is unit-colored, and that's why it's 'right'. If it were smoke-colored, you'd see through the smoke and ship in-game.)

Edit: Why should combatants have all the fun? At first I thought one of the actions was a victory action, but no that's the capture animation.

 
Last edited:

Puppeteer

Emperor
Joined
Oct 4, 2003
Messages
1,687
Location
Silverdale, WA, USA
I've been half expecting to find some secret trick to the unit color palettes, either by reading old threads or having someone pop in this one and say "oh yeah, there's this thing...."

But it suddenly dawned on me that I must be doing something wrong, and given the consistency of the color shifts across units and even in my test scripts spitting out PNGs, the wrong code must be in the Flic converter.

So I tried just swapping around red, green, and blue values in the color palette, and this is what I get when I pick what I think should be blue, red, green (attached pic). That looks much more like it, although there are some stray red pixels in the units where there should probably be highlights.

So I'll go digging into the palette code in the Flic conversion and see what's going on there. It's tempting to guess that the color ordering is just different, but nobody mentions that anywhere, and that doesn't explain these red pixels.

Edit: Just realized that brg is a rotation of rgb which probably means I'm reading one byte ahead behind in the palette; this could also explain the red highlights as any color may be straddling two other color entries.
 

Attachments

  • Screen Shot 2021-01-15 at 1.16.52 PM.png
    Screen Shot 2021-01-15 at 1.16.52 PM.png
    1.1 MB · Views: 57
  • Screen Shot 2021-01-15 at 1.06.49 PM.png
    Screen Shot 2021-01-15 at 1.06.49 PM.png
    1.2 MB · Views: 43
Last edited:

Puppeteer

Emperor
Joined
Oct 4, 2003
Messages
1,687
Location
Silverdale, WA, USA
Just realized that brg is a rotation of rgb which probably means I'm reading one byte ahead behind in the palette; this could also explain the red highlights as any color may be straddling two other color entries.

Okay, it was two bytes behind which is very similar to one ahead. So that's what the samurai is supposed to look like...

The other units look right, too, aside from smoke effects, but I won't redo my calisthenics and mayhem videos.

Edit: I've used ahead and behind from different perspectives in the same post giving opposite meanings. In any case, I had the wrong offset by two bytes for the start of the palette, and it's fixed now.
 

Attachments

  • Screen Shot 2021-01-15 at 3.18.08 PM.png
    Screen Shot 2021-01-15 at 3.18.08 PM.png
    1.1 MB · Views: 48
Last edited:

Puppeteer

Emperor
Joined
Oct 4, 2003
Messages
1,687
Location
Silverdale, WA, USA
I was intending to do some parade beginnings with units running across the screen but didn't get around to it.

I did start poking into audio files. Apparently the .amb format has been mysterious for a while. Old threads here suggest it is a timing pointer/director to other wav files to coordinate the timing of audio to sync with animation, but nobody seems to have known how to decode the files.

Modern-day Internet suggests these are Ambisonic files which are based off of MS's WAVE EX format.

http://www.rwdobson.com/bformat.html
https://docs.microsoft.com/en-us/pr.../design/dn653308(v=vs.85)?redirectedfrom=MSDN

And sure enough, when size and hex dumps are viewed, the amb files don't appear to have audio themselves. It's not immediately clear to me that my linked amb and wave ex links describe the same thing as we see in Civ 3, but it's a starting point.

Hmm, the more I read, the more I can argue both that this isn't and is relevant to Civ3's amb files.:dunno: There doesn't seem to be enough hex-format-specific info to compare to the file.

Oh, I just found this thread here where Tom2050 and others seem to agree it's a modified MIDI file: https://forums.civfanatics.com/threads/modifying-amb-files.326218/ . That actually makes more sense for the apparent use of the files.

Edit: This thread suggests they're also similar to MOD files, and I was sort of thinking that but couldn't remember the name. MODs were like midi with wav samples instead of synthesized notes which was a huge deal back in the day. https://forums.civfanatics.com/threads/amb-unit-sounds.407047/
 
Last edited:

Puppeteer

Emperor
Joined
Oct 4, 2003
Messages
1,687
Location
Silverdale, WA, USA
Oh hey, my palette fix makes the leaderheads look right. They were totally messed up before.

Also I had modified the Flic reader to read the Civ3-specific animation count (8) and frames per animation (10), but the leaderheads seem to use straight Flics, so I had to add some logic to handle either.

(Made by turning FLC into a series of PNGs with my program, then converted to mp4 with ffmpeg, then to YouTube since the forum won't let me attach mp4.)

Edit: @Quintillus it may be possible to link my C# code from Java. I did some very brief Google research, and there seem to be a paid solution or two, but the main way seems to be to use a C++ wrapper. It would be passing a path one way which should be easy enough, but in my brief attempts to pass byte arrays from Go to C a while back I quickly gave up. Maybe C# & C++ behave better.

Edit 2: I've vaguely toyed with the idea of writing FLCs. I've always avoided compression algorithms for lack of knowledge and experience in deciding when and how to compress, but I realized that RLE format doesn't leave many if any decisions to make. I've also wondered about using a modern pixel art editor to make FLCs. It turns out somebody made an MIT-licensed pixel editor in Godot (in-browser version). So there's a crazy idea. I'm thinking I'd probably have to bypass a bunch of its functionality, force a certain palette size, enforce color changes when the palette changes, and somehow account for civ colors, smoke, and shadows. I'm not about to take a stab at this; it's just a back-burner idea.
 
Last edited:

Puppeteer

Emperor
Joined
Oct 4, 2003
Messages
1,687
Location
Silverdale, WA, USA
I did a little work on encapsulation today. A singe object instance can read a unit's files, has a parameter for civ color, and can animate and move in any direction. I also added a way to duplicate an instance without re-converting the original files.

There's currently no correlation in animation speed versus movement speed, but for the moment I have some hard-coded values that look about right. There's also no scaling or positioning levers yet; those are also hard-coded for the moment.

So here are a bunch of samurai running in different directions:
 

WildWeazel

Carthago Creanda Est
Joined
Jul 14, 2003
Messages
7,247
Location
%CIV3%\Conquests\Scenarios\

Puppeteer

Emperor
Joined
Oct 4, 2003
Messages
1,687
Location
Silverdale, WA, USA
Interesting, and that's this Saturday. Some of the talks look interesting: multiplayer, shaders, and..."Godot for the Enterprise"? I want to see that one out of morbid curiosity. On second thought it could make a really cool free charting library....

I think commit ca37ce0 is the one with unit scaling. 20329d1 is the latest for the moment with the running samurai. Of course the code is very similar with values changed or alternate lines commented out, but those commits are the media you saw.

I haven't tried starting from a clean clone yet, so let me know if there are any things needed doing (like nuget restore or similar) before building. I know you'll want to change the Civ 3 path, but that's a script parameter in the Godot inspector (and/or in the RootNode2D.tcsn file).
 

Puppeteer

Emperor
Joined
Oct 4, 2003
Messages
1,687
Location
Silverdale, WA, USA
I've been developing this on a Mac with Godot (w/mono) v3.2.2 as installed by Homebrew (`brew cask install godot-mono`). I just did a fresh clone on Windows and got it going.

On Windows I have Godot w/mono v3.2.3 as installed by Chocolatey (`choco install -y godot-mono`). I also had to install mono and .NET 4.7.2 SDK before it would build. (`choco install -y mono` && `choco install netfx-4.7-devpack`). I had to manually add C:\Program Files\Mono\bin to the path, but I'm not sure if that's needed to build.

I also already had dotnet core cli installed, but I don't think that's needed.

I updated the Godot example readme with build requirements as I currently understand them.

Side note: v3.2.3 greatly simplifies the .csproj file which I appreciate because I was wondering if all that complexity was needed. Not sure if that has more to do with Godot or with the .NET version it targets. I haven't committed the simpler .csproj yet; I want to see about getting my Mac Godot up to 3.2.3 first.
 

Puppeteer

Emperor
Joined
Oct 4, 2003
Messages
1,687
Location
Silverdale, WA, USA
Well, I'm half-settled in a new locale and thinking about this project again. Besides getting busy, I had too many potential directions in which to take this and a lack of ideas on how to future-proof it for later reuse.

I think (programming) interfaces are the answer. This library is graphical in nature, so it's just a visual representation and not the data itself. So if I wanted to use this library for a future version of CIA3, the data structures of CIA3 could then just use this interface and provide getters for the data needed for the graphics to draw. (Unit, civ color, coordinates, etc..) That should decouple the media conversion and graphic display from any future logic use or data structure.

I also learned how to export a project so that others can download and use it without needing to know how to use Godot. But I'll need a different kind of interface–a user interface–for that.

I'm thinking some early target user interfaces may be to load a SAV file to display the map, and a way to load units, either by file browsing or via the SAV or BIQ data. And then I can worry about panning/zooming/moving/animating after that.

But before loading a real map I'll have to do some additional work on the map tile file reading; so far I've just been reading the grass-plains-coast image set, and I need them all to display a real map.
 

Puppeteer

Emperor
Joined
Oct 4, 2003
Messages
1,687
Location
Silverdale, WA, USA
TL;DR: Terrain questions:
  • Are file names for terrain (e.g. xpgc.pcx) hard-coded or in the BIQ editor or config file somewhere?
  • Are bonus grassland bonuses hard-coded or configurable in a BIQ editor? (I know you can place them manually on a map; I just mean can you remove BGs from terrain in the BIQ or give randomly-distributed bonus shields to other terrain types/array-offsets?)

I'm refactoring stuff so don't have the newer code public yet, but I can read in a save file and display parts of maps, the base terrain where all 4 surrounding tiles are either plains, grass, or coast.

To expand to the rest of the terrain tiles I need to understand a bit more about terrain. At first glance it looks like file names are hard-coded as I can't find them in the BIQ. So I presume every mod using new terrain graphics must use the same filenames for terrain? Even if "grassland" (or terrain[2]) is modded to something completely different?

Also, I see no way to configure bonus grasslands in the BIQ. I conclude that bonus grasslands are hard-coded into the game engine to randomly apply to grassland/terrain[2] type terrain. But I'm not sure yet.
 

Attachments

  • Screen Shot 2021-03-27 at 7.51.30 PM.png
    Screen Shot 2021-03-27 at 7.51.30 PM.png
    190.5 KB · Views: 31

Puppeteer

Emperor
Joined
Oct 4, 2003
Messages
1,687
Location
Silverdale, WA, USA
Oh dear. Um, there are multiple versions of some tiles. In fact, there is an entire 81-tile file that's all ocean tiles. Not a mix of coast, sea, and ocean...just all ocean. I don't know how to choose the 'right' one.

I seem to recall talking about this before when realizing there were duplicates of 'all grass' tiles between sets like plains-grass-coast and desert-grass-coast and others, and I think Quintillus gave me a tip...I'll have to go find that. I didn't really expect to see a literal ocean of ocean tile choices. And there are more all-grassland tiles than I realized because they can border several tile types, and there is even a grass-grass-coast tile set which has even more more full-grass tiles.

Edit: Ok, there is a byte in the tile to indicate which tileset to use. I don't think we know how to choose e.g. the right ocean tile from xOOO.pcx or which same-orientation tile of xggc.pcx, though. Maybe something to do with the landmark terrain byte?

Edit 2: Oh, looks like there is an image byte to choose which tile. The confusion was on how to pick one when creating a map and saving. Actually, having a file and image byte may save me a LOT of trouble for just displaying a map.
 
Last edited:

Puppeteer

Emperor
Joined
Oct 4, 2003
Messages
1,687
Location
Silverdale, WA, USA
That's more like it. It's the base terrain only; now I need to find the overlay terrain image references.

Interesting (to me) side note: the image index begins at the top-left and moves left-to-right, then down. Earlier I concluded that for a 2x2x3 tile grid layout there was a pattern going top-to-bottom, then left to right.

There is, but since it's a 3-step pattern and a 9x9 grid, there is also a left-to-right then top-to-bottom pattern, and it goes like this:

  • Top is least significant digit
  • Left is next least
  • Bottom is next least
  • Right is most significant digit
So basically a clockwise pattern starting from the top instead of a couterclockwise pattern starting from the right. As long as the conversion is consistently applied it doesn't matter, but it's pretty clear left-to-right then top-to-bottom is how the designers were thinking.
 

Attachments

  • Screen Shot 2021-03-28 at 11.43.07 AM.png
    Screen Shot 2021-03-28 at 11.43.07 AM.png
    828.9 KB · Views: 758
Top Bottom