Civ3 Show-And-Tell

Puppeteer · May 20, 2014

Um, wow, I've been banging on this a few hours tonight. Let's see if I can remember what I've accomplished. (Git repo updated.)

Can read any uncompressed game file now!
Layout change with Bootstrap
Added a "Redraw" button as it turns out Webkit browsers (Chrome/Safari et al) don't always redraw when you expect! et tu, Chrome? It's a hack, but it redraws.
Fixed IE map height...was that today? Seems like longer ago.
Now searching for first WRLD section and then reading the next two WRLD sections, the second of which has width and height so I don't have to hard code the size anymore. Seeking to the first WRLD is still a bit of a hack, but it works!
Changed "svg" front end to take a filename or stdin, but it will choke on an compressed SAV. I'm currently using dynamite (in most Linux distros) to decompress files or blast (zlib contrib) to decompress in the pipe to stdin. (However, autosaves are not compressed!)

So it's still not very close to a Windows executable, but I'm getting very close to being able to chain some commands together and perhaps make a web service out of it. Somebody with Python on Windows might be able to make a map from an uncompressed save like an autosave with something like "python svg.py path\to\autosavefile.SAV".

Some random notes because I'm very disorganized at the moment and need to put this stuff somewhere:

Overlay terrain 0x04 I think is flood plain
Overlay terrain 0x0a is a volcano
2nd WRLD section 2nd integer (offset 0x04) is map width
2nd WRLD section 6th integer (offset 0x18) is map width

So it's inching closer to being useful to others. I have an idea of how I want it to work as a web service for showing maps for online saves (e.g. posted to a SG and the players aren't posting many screenshots). There is interest in this; what do you want if you could run this on your PC? You want a web page, an SVG?

Currently it makes an SVG that appears to be a big black rectangle until it's put on a web server and opened with the web page which applies the colors via CSS. That has some advantages, but it makes the SVG a bit useless as a standalone graphic. I could put the colors inside the SVG to make it useful standalone, but then it's more difficult to style it after-the-fact.

Also, now that I can make it open arbitrary saves (compressed ones by piping through blast) I am not even sure what to put up and what not to. I probably need to code it so it doesn't always use the same name for the map, and maybe make the browser where it can pick from more than one. For now I'll post some screenshots. (will post and then edit lest I lose all this typing)

Afterthought: I haven't thought about rivers in a while. That's pretty important to have on the map. I may want to start looking into where the river info is stored.

Quintillus · May 20, 2014

Wow, that's a nice big jump from the isometbrick of last year! It's working great for me in Opera 11.64 (which works slightly better on CFC than 12.x). It's pretty cool that zooming and panning really is working, and it really does look like a lower-graphics version of a Civ map.

And yeah, IE tends to be a pain. On my job, while it's not as bad now that we don't support IE8, IE still is responsible for most of our browser-specific issues. Every so often we'll get the odd Firefox one or something like that, but usually it's IE, and more often than not, it affects all versions of IE, not just IE9. Although for my own personal use, I don't care whether a website supports IE or not - I only use it if it's the only browser a website supports.

Puppeteer · May 21, 2014

Quintillus said:
Wow, that's a nice big jump from the isometbrick of last year! It's working great for me in Opera 11.64 (which works slightly better on CFC than 12.x). It's pretty cool that zooming and panning really is working, and it really does look like a lower-graphics version of a Civ map.

Thanks! I keep thinking I may add an "isometbrick" option to the SVG output for nostalgic reasons. I think it will be pretty easy, and I may even be able to animate the transformation back and forth with little or no trouble.Plus it will serve as a template for someone to build their civ map out of building blocks.

I took a nap today. Not very productive to the mapping cause, but I needed it. I decided to install Inkscape and try drawing some overlay terrain. This is my first attempt at a tree and a test where I just plopped it over the map as-is to ensure I could copy the Inkscape SVG output into my map as I intended. I am amused.

This is actually a success. I can use a transform command to scale and position the tree and xlink the tree a couple of times and suddenly I have a forest tile. At least that's what I think.

Edit: For the hell of it, here is my masterpiece in its most basic SVG form, meaning this is what I pasted inside my map SVG just before the last </svg> tag:

Spoiler :

Edit 2: I've been saying "xlink" when I really mean <defs> and <use />. xlink is what I used to show the half-tiles at the edges of the map.

Puppeteer · May 21, 2014

Getting the tree sized, positioned and duplicated was pretty easy, but I kept tweaking the positioning.

The hill turned out to be a lot more trouble than I imagined. This time I started with a correctly-sized tile diamond shape, positioned it at 0,0 and drew the hill on top of it.

Well, apparently Inkscape's 0,0 is different than what's in the browser. It took me a while to figure that out and to get the hill shape copied. The hill is way, way better than the unicode circle I've been using. (The circle was supposed to get overdrawn so it was a semicircle hill, but with IE's different text positioning and the pan & zoom code all the hills turned into full circles...d'oh.) But I'm not sure if this is what I want.

Today's interactive, scrollable map with new forest and hill terrain

Minor note: The new forest and hill have inline styling and therefore I don't think will be easy to style. Maybe that will get fixed eventually.

Screenshots show the evolution of the forest tile.

Edit: In the linked map in the upper-left corner you can see a stray hill (well, half or a quarter of it). That's the original definition which I forgot to put in the <defs> section so it got drawn. It's fixed in the code, but I'll leave my mistake at that link.

Edit 2: Idea: Flip the forest tile on even/odd rows so stacked forest tiles aren't visually aligned.

Edit 3: I'm itching to make a new mountain tile, but I really need to get to sleep. I am not sure what I want to do for jungle, marsh or flood plain tiles. I want to be able to tell jungles from forests, and I can't really think of how to icon-ize marsh or flood plain. (I've never been particularly happy with Civ3's rendering of such, either, but I'm not trying to re-tile the game, I'm trying to make an easy-to-read-for-the-default-epic-player map.)

Puppeteer · May 21, 2014

Thoughts from today:

Thinking about physically-impossible overlay trick for adjacent overlay terrain tiles...think I can come up with a way for forests and hills to appear joined if two are next to each other without additional logic.

Was reminded that 0xffff is -1 and realized runs of 0xff in the map are likely unused/disabled values...perhaps civ-specific tile info for e.g. the 23rd civ in an 8-civ game. Maybe by-civ visible flags or stale tile improvement/overlay info.

Just realized that barbs must have fog of war info, too.

Puppeteer · May 21, 2014

Hmm...hills three ways. Comments?

Puppeteer · May 21, 2014

Volcano and mountains.

I've decided I can't obsess over each terrain element at this time. I want to get the terrain overlays done. Later I'm sure I'll go back through and tweak everything.

I had originally intended for the map to be mod-able via CSS, but it's looking more like it will be very difficult to do that in a sane manner. I think I'll eventually need to have SVG templates and a way for a user to substitute his own tiles when generating the map. The upside of this is that I can put all the styling back into the SVG, so the standalone SVG file is usable as-is.

Ooh, that just gave me an idea: I can have one SVG template file with the tiles spaced so that they match up with their terrain ID nybbles. e.g. a hill is 0x05, so it's position on the template will be n*5, probably 1000. The trick will be ensuring any 3rd-party edits remain in the group for that tile graphic. I'll think of something.

And mega-DUH, I've been adding these SVG snippets in a huge string in the Python code. This is meant to be an external template that is included. Hmm, time to go try a few things...wondering if I can put a whole <svg> inside a <defs> tag....

Edit: I'm going to add a link momentarily to a world map with the new forests, hills, mountains ~~and volcanoes~~. (Wow, no volcanoes on this WM.)

Edit 2: Sorry, no I'm not. The big WM is crashing IE right now. Well, I was about to attempt to speed things up and save memory, anyway.

Spoiler :

Edit 3: Some random ideas going through my head:

The tiles don't have to be perfectly straight. I can experiment with slightly different shapes for a less harsh look. But may cause problems with fog of war overlap and false fog gazing
I want to put the rivers on the map but don't know yet where they are coded
I want to refactor the code to pull a definition SVG in and use lots of linking; thinking this may save memory and speed up the big maps zooming
I want to finish overlays but am not sure what I want to do with jungle, flood plain or marsh
I really need more sleep
It would be nice if I could get this where the average CivFan person can download and do something at all with it

Puppeteer · May 22, 2014

Hmmm, making everything xlinks didn't seem to help. I think large maps are a problem. I believe gradients are a problem, but I temporarily replaced hills with mountains, so there should only be filled shapes, and it was a little better but still pretty slow.

I may have to figure out a different way to organize the SVG info, if that helps.

That thing I said about runs of 0xff's? I think I'm right. Well, this was in the WRLD big section: the world map had many fewer 0xffff's than the 4-civ small map.

I'll end the day with a bit of nostaliga: Zoomable SVG Isometbrick map

Puppeteer · May 22, 2014

I've been poking at it tonight trying to figure out how to speed it up. At first I thought it was the gradients, then I thought it was the metric ****-ton of trees. But nope...if I load the SVG as an external image it is fast, even faster in IE for a change:

Huge pan/zoom world map as external image, way faster in IE

So I think the problem has something to do with loading the SVG in the DOM, although I don't know why IE suddenly flies and Chrome lags when I use the SVG as an image tag.

I may have to rethink how I want to use SVG. Or maybe I can figure out what's happening in the DOM to slow it down.

Edit: Random thoughts from today:

I can probably eliminate the tile gaps in IE by stroking the tile with the same base color. Come to think of it, that plus antialiasing or some intentional feathering might make the tile tranitions softer
I may try getting rid of my groupings and see if that speeds things up
Maybe jQuery is the problem or mixing Bootstrap and the pan'n'zoom. Thinking maybe jQuery is traversing all those tags I stuffed in the DOM and that is slowing it down
I don't get why Chrome and IE switch which is fast and which is slow when I load the SVG differently
Now that I know gradients aren't slowing things down and ****-tons of trees aren't slowing things down I can add shadows to my mountains and stuff tons of other shapes in the map.
I want to show BGs
I still want rivers

warmwaffles · May 23, 2014

This looks great. Keep it coming man!

Just so you know on my Mac I can't two finger scroll in and out. I'm on FF 29.0.1

WildWeazel · May 23, 2014

I must have missed this the first time around. Impressive work. Repo starred

Puppeteer · May 23, 2014

warmwaffles said:
This looks great. Keep it coming man!

Just so you know on my Mac I can't two finger scroll in and out. I'm on FF 29.0.1

Thanks! I don't have a Mac to test on. Can you try a couple of things and tell me the results? Please try:

Does two-finger scrolling work in Safari?
In Firefox, does this tiger at the bottom zoom or not?
In FF can you two-finger zoom the black square in the lower-right frame here? This last one is supposed to be a fix for a known FF issue.

WildWeazel said:
I must have missed this the first time around. Impressive work. Repo starred

Thanks! I'm not sure what "Repo starred" means, but I can tell it's a good thing. I was trying to figure out how to mega-star Antal1987's threads but couldn't figure out how to do it.

Edit: OHhhhhhh...GitHub repo starred. Thanks!

Puppeteer · May 23, 2014

The past couple of days I have been bugged about how I want to or need to rearrange the SVG structure, but actually it doesn't matter. That is, it is pretty trivial to change how the SVG file is structured, and it doesn't make a difference visually. And really it's only a problem on the huge maps, and I can always figure that part out later.

The takeaway is that number of shapes and gradients does not seem to be a significant factor, so I can continue making shaded and multi-shape terrain tiles.

My goals continue to evolve. I enjoyed the buttons that could show and hide elements, but really there is no practical need to show/hide the stuff I was showing and hiding. It was really a proof of concept and debug helper, but I think I will want to be able to hide tile improvements, units and cities (if I ever get that far). But again, I can proceed now and easily adapt later whether I want to try to load all the info in the SVG into the DOM, layer separate SVG files or combine the two tactics.

At this time my aim is to have an SVG file that is styled and colored as-is, and an HTML viewer. I need to finish the overlay terrain and figure out rivers before I go looking at other map data, anyway. The SVG file might even be useful to someone to import into Inkscape, Illustrator or something else to play with.

One thing that I think could be easy if I were a real programmer would be to have my program handle compressed game files. Civ3 compresses save game files with PKWARE Data Compression library which for some reason has nothing to do with PKWARE's zip format and is not widely supported. I can find a standalone windows tool and linux tool that can decompress the saves, but I don't think I can leverage them from my program. I was able to find contributed code "blast" in zlib, and I even compiled it into another standalone program that can decompress a file and stream it directly into my program if I use pipes on the command line, but I don't immediately have the knowledge of how to call it as a library to decompress on the fly, and I think if I were a real programmer that would be simple to do. But my true colors are shown. I'm a tinker-toy programmer.

Edit: As you may have noticed, I am easily distracted and tend to over-think things. i really like how my volcano turned out. I am wondering where the game marks that a volcano is active? I wonder how hard it would be to capture live volcanoes and show that on my map? I'm also thinking about snow caps for mountains and trees; that's got to be coded in there somewhere.

PatriotNorwood · May 23, 2014

So where is this all headed? What are your goals for this, is this just a map editor, online? Could you explain this in simple terms? It looks really good, in what I understand.

Puppeteer · May 23, 2014

PatriotNorwood said:
So where is this all headed? What are your goals for this, is this just a map editor, online? Could you explain this in simple terms? It looks really good, in what I understand.

The original motivating factor was when people in this forum post save games but not screenshots. How many times have you read "I'll look at the save when I get home tonight"? I had this crazy idea that there could be a website that will show you the map for that posted save game in lieu of screenshots and without having to download the save game and open it in Civ to discuss it.

So, in a nutshell, accessory viewer, "screenshot anywhere", or "Google Maps for Conquests" for discussion and/or for people who find it difficult or are otherwise too bothered to post screenshots.

That's still what I want to do. I get other crazy ideas, but what I want to happen is for you to be able to paste a link to an uploaded save game into a web page and see the map.

Since there is interest, and since I want to share, I also want you to be able to download a file and be able to turn any of your own saves into SVG files and/or HTML pages. What use is that to you? I don't know.

One "crazy idea" feature I think I can implement is dot mapping. If you read the succession games, they often make dot maps to discuss and plan city placement. I think I can make that doable online as a dynamic overlay to these maps. But then again hardly anyone is doing SGs anymore.

Other crazy ideas include duplicating some of the features of CivAssist II or MapStat, but I really don't think I'm following through on these because, well, we already have CivAssist II and MapStat.

So mainly I'm just entertaining myself.

I do not aim to edit the game file or game art. This is an accessory viewer that's completely independent of the game code.

PatriotNorwood · May 23, 2014

That sounds pretty cool, I could see a being able to view maps on tablet/smartphone being useful. It would also make sense to use it, on this site as a way to preview the map creations of other people. You could just upload a map! Is there anything I can do to help? (Even though I don't program)

warmwaffles · May 24, 2014

Puppeteer said:
Does two-finger scrolling work in Safari?

Unfortunately not. I tried in Chrome and Safari. I looked at the JS console in both FF and Chrome, and nothing is erroring out.

Puppeteer said:
In Firefox, does this tiger at the bottom zoom or not?

Yes the scrolling works

Puppeteer said:
In FF can you two-finger zoom the black square in the lower-right frame here? This last one is supposed to be a fix for a known FF issue.

This doesn't work for me.

Quintillus · May 24, 2014

I think this would be really awesome for multiplayer PBEM)games. It's not uncommon for me to see a PBEM thread and think, "That sounds cool, I wonder who's winning?" but to have no effective way to see the map, since there are generally very few maps posted in the thread. Being able to have this tied in with a multiplayer game tracker would make it so much easier to follow. It's also the same basic reason that when I was in an EU3 multiplayer game on CFC, I made sure to upload the map and an update on the action after each session - so the lurkers could follow along. And there were indeed some lurkers, more than I see in most PBEM threads.

Puppeteer said:
One thing that I think could be easy if I were a real programmer would be to have my program handle compressed game files. Civ3 compresses save game files with PKWARE Data Compression library which for some reason has nothing to do with PKWARE's zip format and is not widely supported. I can find a standalone windows tool and linux tool that can decompress the saves, but I don't think I can leverage them from my program. I was able to find contributed code "blast" in zlib, and I even compiled it into another standalone program that can decompress a file and stream it directly into my program if I use pipes on the command line, but I don't immediately have the knowledge of how to call it as a library to decompress on the fly, and I think if I were a real programmer that would be simple to do. But my true colors are shown. I'm a tinker-toy programmer.

The way I call the external decompression program I use in my editor is like this:

Code:

               //BIQ input routine, where it detects if it's compressed
               if (comrpessedBIQ) {
                        logger.info("Detected compressed file");
                        String[] decomQuery = {"java", "-jar", "./bin/BIQDecompressor.jar", file.getCanonicalPath(), "._tmp.biq"};
                        Process dcp = Runtime.getRuntime().exec(decomQuery);
                        //wait until it's decompressed before continuing or we'll blow up
                        dcp.waitFor();
                        return inputBIQFromScenario(new File("._tmp.biq"));
                }

I have to use an array of strings when executing the external process on the command line, so it works even if BIQ has a space in its name. Then I wait for the process to die, and open the resulting uncompressed file (which I then later delete after it's opened).

You'll likely have to call down to your server from the JavaScript/HTML in the web browser to do the decompression, and then again call out to the command line to do the decompression (and hopefully the decompressor you're using allows you to output the file - if not, I can find where I uploaded the decompression JAR I'm using on CFC). Then the server can wait for the decompression to finish, and when it's done, open up the decompressed file, similarly to what the code I have above did.

I'd go and look in more detail, but it's late after a night of playing Civ, and likewise,

I really need more sleep

So I'm going to address that before another half-hour has passed looking for an exact solution to this.

Puppeteer · May 24, 2014

PatriotNorwood said:
That sounds pretty cool, I could see a being able to view maps on tablet/smartphone being useful. It would also make sense to use it, on this site as a way to preview the map creations of other people. You could just upload a map! Is there anything I can do to help? (Even though I don't program)

Thanks! Ooh, I hadn't thought of map creations. You mean like this Game of Thrones Scenario?

Game of Thrones Scenario first turn (IE much faster this link)
Game of Thrones Scenario first turn (Toggle-able features this link)

(For these latest links I am not constraining the SVG to the window. There is some voodoo between SVG size, div size, viewBox, panzoom controls and the browser I'm trying to slowly figure out.)

My program doesn't read BIQ files (yet), so I started a game, and this is the starting map. In-game the whole map is revealed, but my map reader only had fat-X-sized bits shown here and there, so I implemented a spoiler flag I can turn on to show the whole map. I want to be careful not to make it too easy to accidentally post spoiler map info, but clearly there is a use for the capability.

warmwaffles said:
Yes the scrolling works

Ok, please try these. I made a slight change in the tag structure to more closely mirror the tiger.

Mac zoom test, svg as img
Mac zoom test, svg in DOM

When making this test I realized it's not just the big map that behaves differently depending on how I include the SVG. IE is faster than anything when I include the SVG in an img tag. Chrome seems to do better with the SVG in the DOM while IE pretty much chokes on it. I'll be doing more experimentation to see what the root cause is, but I'm more focused on other things so I'll just offer two viewers for now. Please also try two-finger-scrolling the Game of Thrones links above. They are mostly the same, but I removed a couple of constraints and wonder if it makes a difference.

Quintillus said:
I think this would be really awesome for multiplayer PBEM)games. It's not uncommon for me to see a PBEM thread and think, "That sounds cool, I wonder who's winning?" but to have no effective way to see the map, since there are generally very few maps posted in the thread. Being able to have this tied in with a multiplayer game tracker would make it so much easier to follow. It's also the same basic reason that when I was in an EU3 multiplayer game on CFC, I made sure to upload the map and an update on the action after each session - so the lurkers could follow along. And there were indeed some lurkers, more than I see in most PBEM threads.

I started this project with a fear of spoiler info, but lately I've been thinking I'm pretty sure it's possible to see the known map for each civ, and there might be use cases for that. Is the multiplayer game tracker something that exists, or is it something you want but don't have? What type of map info would be most useful: whole map, known-map-by-civ or aggregate known world?

For the decompression I currently need to solve for Linux and Windows if I want to make an .exe . On my server-side thing I'm pretty sure I can make it work, easily if I can assume all saves are compressed. Actually now that I think about it, reading the first four bytes and if it's not "CIV3" then feeding it through the decompressor should be trivial. I guess for Windows I can distribute the decompressor program and call it directly.

Edit: Screenshots: Old Valyria, I presume?

Puppeteer · May 24, 2014

PatriotNorwood said:
Is there anything I can do to help? (Even though I don't program)

I meant to address this but got distracted. Um, maybe? I had originally imagined people might want to style the tiles via CSS, but now I seem to be pivoting to where changing the tiles would require SVG snippets. Other than that, let me know how you want to use this and I can see if that will work or not.

I notice most of your posts are in the Civ4 section...you know this one is for Civ3, right? The SVG map concept could work for many different applications, but this project currently only reads maps from Civ3 save files.

Civ3 Show-And-Tell

Emperor

Attachments

Restoring Civ3 Content

Emperor

Attachments

Emperor

Attachments

Emperor

Emperor

Attachments

Emperor

Attachments

Emperor

Emperor

Programmer

Carthago Creanda Est

Emperor

Emperor

Howdy!

Emperor

Howdy!

Programmer

Restoring Civ3 Content

Emperor

Attachments

Emperor

Similar threads