1. We have added a Gift Upgrades feature that allows you to gift an account upgrade to another member, just in time for the holiday season. You can see the gift option when going to the Account Upgrades screen, or on any user profile screen.
    Dismiss Notice

SVE file format

Discussion in 'Civ1 - General Discussions' started by darkpanda, Apr 15, 2013.

  1. darkpanda

    darkpanda Dark Prince

    Joined:
    Oct 28, 2007
    Messages:
    601
    You're right, Civ can actually generate maps with more than 14 islands or oceans... It'd be interesting to see how AI players would behave with those, whether they will settle on them at all...

    I am also attaching those 2 maps with continents/oceans overlay, exported from JCivED:

    CIVIL0.SVE_256.islands.gif CIVIL1.MAP_256.seas.gif
     
  2. darkpanda

    darkpanda Dark Prince

    Joined:
    Oct 28, 2007
    Messages:
    601
    A quick word to notify that I updated the SVE data reference table with latest comments form Gowron and Dack...

    Also, we have deciphered 88.6% of the byte content of SVE!!! Congrats everyone, we're almost there :)

    Hereunder is the deciphering map (blue=known, red=unknown):

     

    Attached Files:

  3. darkpanda

    darkpanda Dark Prince

    Joined:
    Oct 28, 2007
    Messages:
    601
    After some quick analysis, this data block seems definitely related to Civ and Continents, e.g. 8 blocks (1 for each Civ) of 16 short integers (1 for each Continent):
    • For an active Civ, all values are 1 by default, except for Continentz 0 and 15, which have value 0
    • For a dead Civ, all values are 0
    • If a Civ possesses a city on a continent the value is 5 (although is some occasions, it seemed to be 5 even if the civ has no city but a lot of units... to be confirmed)
    • Another often witnessed value is 2, but I couldn't correlate it to any special in-game situation

    That leads me to think that this value is actually a flag, where the bit 0x04 means that the civ has a city on the continent... Bit flag 0x01 may mean that the Civ has never set foot on a continent yet, while 0x02 may mean that is *has* actually set foot already... More investigation needed.
     
  4. Gowron

    Gowron Chieftain

    Joined:
    May 21, 2007
    Messages:
    62
    This value describes the AI agenda for the respective continents.
    It can only assume a few values, but it is not a flag.
    Possible agendas:
    0 = Settle
    1 = Attack
    2 = Defend
    5 = Transport
    With the cheat/debug mode activated, you can view the current agendas at the F7 screens. They are only displayed for continents where the "defense strength by continent" value is greater than zero.
     
  5. Gowron

    Gowron Chieftain

    Joined:
    May 21, 2007
    Messages:
    62
    This is the current Civilization Power Ranking, it's used to estimate the power of a civ.

    7 = most powerful civ
    6 = second-most powerful civ
    ...
    1 = least powerful civ
    0 = Barbarians

    (2 bytes per civ)
     
  6. darkpanda

    darkpanda Dark Prince

    Joined:
    Oct 28, 2007
    Messages:
    601
    During my journey inside the map generation code, I figured out this code was also setting up the values of the data block unknown25, which is apparently some kind of high-level path-finding data, i.e. data meant to help the Civ AI moving units around the map in an optimal way.

    The funny thing is that the Civ code setting up this data seems to wrong - although it may not matter in the end...

    Let me try to explain:

    1. The first thing that can be remarked, is that this data is almost always the same, whatever gamesave you take form whichever version of Civ... Actually, I noticed this after having gone through everything explained below, but anyone can see it for themselves by opening their SVE save in JCivED or any hex editor.
    In my case, this data block is nearly always as below (the rare differences happen on 1 or 2 bytes, usually at the end of the block):

    Code:
    00 16 13 11 01 10 01 02 00 00 0A 02 00 
    30 61 00 00 02 10 21 00 12 29 20 8A 00 
    00 00 00 24 00 0A 18 25 08 02 A4 08 80 
    14 03 00 46 20 0A 80 C0 20 80 48 08 80 
    68 00 38 45 20 00 80 02 00 04 00 8E 80 
    00 90 01 C0 00 00 22 10 01 40 34 5B 85 
    00 00 00 00 00 22 00 10 01 00 68 00 C6 
    00 10 01 00 20 04 10 01 00 00 00 B4 43 
    00 00 00 04 00 40 00 00 00 14 03 64 00 
    00 00 00 40 00 00 00 10 01 64 00 40 00 
    00 04 00 10 01 00 00 00 00 40 00 00 04 
    00 44 00 02 02 00 06 10 11 01 00 00 40 
    00 40 20 20 00 3C 4F 10 01 10 01 00 00 
    00 00 00 00 00 76 DB 95 13 11 01 00 00 
    10 01 00 06 38 65 04 E4 00 00 00 00 04 
    08 00 30 41 00 C2 44 40 0A 00 00 00 40 
    00 80 00 00 24 00 40 22 00 94 03 0C 00 
    00 04 00 02 44 10 21 00 00 62 04 58 85 
    00 40 24 00 42 12 01 00 28 00 40 08 C0 
    00 00 44 20 20 00 16 0B 00 8C 00 00 84 
    
    Note that I arranged the bytes in a 13x20 table format (=260 bytes) , you will understand by reading further below.

    2. Second thing: how Civ generates this data was very interesting to walk through.

    Here is basically how it goes:
    • Civ divides the map area into 4x4 tiles: 80/4 = 20 and 50/4 ~= 13, which makes a set of 20*13 = 260 tiles; this explains the 20x13 data size, and we can then expect 1 byte of data per 4x4 map tile
    • The iteration loop is by columns: first tile to be processed is in the top-left corner, then the next tile is the one below, then below, etc... until the column is fully processed; then the next column (right of the top-left tile) is processed, etc.
    • For each tile:
      • Civ checks if any of the 4 central squares is a land square:
        • If not, the tile is skipped and the next tile is processed
        • Else Civ retrieves the corresponding Continent ID
      • Next, Civ iterates through the 4 neighbour tiles at the top (North), top-right (North-East), right (East) and bottom-right (South-East) as follows:
        • If one of the tile's 4 central squares is also land, and has the same Continent ID as retrieved above, and the walking distance(*) from the processed tile to the neighbour tile is less than 20:
          • Update the unknown25's byte corresponding to the processed tile: set the bit 0x1 (for N), 0x2 (for NE), 0x4 (for E) or 0x8 (for SE) to value 1
          • Update the unknown25's byte corresponding to the neighbour tile: set the bit 0x10 (for S), 0x20 (for SW), 0x40 (for W) or 0x80 (for NW) to value 1; note: this is a smart trick to cross-link tiles, so only 4 neighbours need be processed for each tile...
      • Civ goes on to process the next tile

    About the walking distance mentioned above: I didn't decipher exactly how it's done, but at a glance, Civ seems to instantiate a militia unit (alternatively a trireme for sailing distance) and lets it GoTo from the original square to the destination square, counting up steps on the way... I may be wrong, but at the moment it does not really matter.

    So the bottom line is that this generated data is a byte flag for each tile indicating which of the 8 neighbour tiles are within close walking distance. It is like a high-level roadmap that AI land units (or player's GoTo-units for that matter) could use as a guideline to find directions when moving around.

    3. Third, what does it look like?

    Intuition made me think that such a roadmap should follow the shape of continents, somehow like a skeleton of the continents. To confirm this, I wanted to compare the generated unknown25 data with the actual map data, so I wrote a quick routine that displays the continents in the background, then overlays square-level and tile-level grids (in dark- and light-green), and then draws the unknown25 as a graph linking the tiles (pink for N,NE,E,SE and yellow for S,SW,W,NW).

    Hereunder is the result for different 2 gamesaves:




    As anyone can see, the path-graphs do not make any sense compared to the actual continents, and furthermore (this is how I discovered it) they are almost identical! That is to say, the unkown25 data is actually independent form the map geography...

    4. But how can that be wrong?

    Scratching my head and double-checking whether I was misguided in my interpretation of the code, I followed the execution of Civ step-by-step, looking at the memory in parallel (thanks a lot to DOSBox in debug mode for that!).

    Doing this, I could confirmed that I understood it right, but more importantly, I understood what Civ was doing wrong, and I believe it is a programmer's mistake, in the way that the map data is accessed in memory.
    To put it simply:
    • the map data (which is eventually stored in a gamesave's .MAP file) is stored in the "graphics" memory, i.e. a memory region which is managed by the graphics code contained in MGRAPHIC.EXE, EGRPAHIC.EXE or TGRAPHIC.EXE
    • this region of memory actually contains multiple images, and the graphics code maintains an array of "image pointers" to convert an image ID (0, 1, 2 ...) into an image pointer (424F:0000, 51F0:0000, ...)
    • whenever something needs to be read from or written to the map data, Civ calls specific routines from the graphics code, with image parameters (X, Y, color value, etc...) and more importantly with the image ID from which the data must be read from/written to.
    • for most of the map generation code, Civ uses image ID = 1 to store the map data (stored at 424F:0000 in Civ EN 474.01)
    • however, in our specific example of the unknown25 data, Civ uses image ID = 2 (stored at 51F0:0000)

    I was curious whether Image ID = 2 was containing something useful for the generation of map data, so I made a memory dump and converted it to a picture, which is shown below in false colors:



    As you will surely have discovered by yourself, this is actually the first image of the new game introduction sequence, BIRTH1.PIC, which happens to be the current screen content at the time when the map generation code is being executed:



    5. Can it be fixed? Should it be fixed?

    Having discovered that, I tried - unsuccessfully so far - to patch CIV.EXE so that it goes and get the data from the actual map image.

    I am actually very interested in discovering what Civ would do if accessing the data properly...

    But then I am also wondering: if this bug has always been here, in all versions of Civ (to be confirmed) is this data actually used at all by the Civ's AI code? In oterh words, even if it is corrected, is it going to change anything in the game?

    I guess answers to those questions will come in the next episode...

    At least I think that's one more down for the deciphering of SVE file format! :)

    Cheers
     
  7. darkpanda

    darkpanda Dark Prince

    Joined:
    Oct 28, 2007
    Messages:
    601
    After further investigations, I discovered multiple things:
    • Right after generating this (wrong) path-finding data for continents, Civ does the same for oceans; but this time it does it right!!
    • Indeed, when retrieving a square's terrain type or its ocean ID, Civ uses the correct Image ID (1) contrary to what it does for Continents...
    • I also discovered that Civ does use this data in routines managing unit movement... It makes it even more interesting to know what Civ would do if this data was correct!
    • Sadly, the next thing I realized is that fixing this bug in CIV.EXE is very hard - if not impossible at the moment...
    • Next discovery: while the continent path-finding data is stored in a gamesave's SVE, the ocean path-finding data is not saved at all; so this could mean that Civ either re-generates it when loading a saved game, or it is just completely discarded, and Civ can manage without it...

    More investigation to do...

    Hereunder are screenshots of the ocean path-finding data generated by Civ for some random new games:



     

    Attached Files:

  8. darkpanda

    darkpanda Dark Prince

    Joined:
    Oct 28, 2007
    Messages:
    601
    I spoke too fast: I actually found a rather simple fix that corrects this bug in Civ EN 474.01 (should work with 475.01 as well, but untested).

    WARNING: hereunder are details about how to modify CIV.EXE; do NOT forget to make backup copies of any original files prior to modification!

    Basically, the idea is to find the calls made by the "continent pathfinding" routine to the "getTerrainType" and "getContinentID" functions, and replace them by calls to the other "getTerrainType" and "getContinentID" functions, i.e. the ones that are used by the "ocean pathfinding" routine.

    The original (wrong) calls are the 2 following bytes sequences:

    Code:
    9A 3C 13 05 1B ; // call to [B][COLOR="Red"]wrong[/COLOR][/B] GetTerrainType
    9A 2F 19 05 1B ; // call to [B][COLOR="Red"]wrong[/COLOR][/B] GetContinentOrOceanID
    
    Hereunder are the byte sequences to the correct functions:

    Code:
    9A 0C 17 88 23 ; // call to [B][COLOR="Blue"]correct[/COLOR][/B] GetTerrainType
    9A 2D 17 88 23 ; // call to [B][COLOR="Blue"]correct[/COLOR][/B] GetContinentOrOceanID
    
    The "continent pathfinding" subroutine makes totally 16 calls to the above functions, at the following offsets, given as absolute offset in the original CIV.EXE (WARNING: there may be other occurences of those bytes; DO NOT modify them!):

    Code:
    [b]GetTerrainType[/b]
      [0] offset in file: 218477 [0x3556D]
      [1] offset in file: 218519 [0x35597]
      [2] offset in file: 218608 [0x355F0]
      [3] offset in file: 218665 [0x35629]
      [4] offset in file: 218715 [0x3565B]
      [5] offset in file: 218757 [0x35685]
      [6] offset in file: 218862 [0x356EE]
      [7] offset in file: 218919 [0x35727]
    [b]GetContinentOrOceanID[/b]
      [0] offset in file: 218494 [0x3557E]
      [1] offset in file: 218536 [0x355A8]
      [2] offset in file: 218630 [0x35606]
      [3] offset in file: 218680 [0x35638]
      [4] offset in file: 218732 [0x3566C]
      [5] offset in file: 218777 [0x35699]
      [6] offset in file: 218884 [0x35704]
      [7] offset in file: 218937 [0x35739]
    By replacing the wrong function calls bytes with the correct function calls bytes, CIV can now generate the proper continent path-finding values, as illustrated hereunder:





    Now has anyone time to fix their CIV.EXE and experience a game to see whether it makes any difference ?
     

    Attached Files:

  9. Renergy

    Renergy Warlord

    Joined:
    Nov 2, 2010
    Messages:
    197
    Location:
    Czech Republic
    darkpanda, your devotion to deciphering civ.exe and .sav format is quite impressive. :goodjob:

    Did a quick play-through of the patched version. Civ is stable, no problems, but I have not seen any noticeable changes. Save attached.

    Also, attached is the binary patch for easy changing 474.01 original to "474.01dp" - patch.pcx
    (had to give it the .pcx suffix to be able to upload it as an attachment; usual suffix is .bdf; however, the patching works fine even with .pcx suffix)

    tools for binary patching here:
    http://sites.inka.de/tesla/others.html#bsdiff - Windows port
    http://www.daemonology.net/bsdiff/ - source

    usage:
    bspatch original.file new.file patch.bdf

    i.e.
    1) copy original civ.exe to civ01.exe
    2) bspatch civ01.exe civ.exe patch.pcx

    and you should have the patched civ.exe

    checksums (CRC-32) civ474.01
    original: 35C3108F
    patched: 4C44470C

    [darkpanda, could you please confirm? I hope I patched it right]

    CRC-32 can be calculated e.g. in Total Commander - Files->Create Checksum File(s)
    or with https://code.google.com/p/crc32/ (requires dotnet framework 3.5)


    As a sidenote, I'd have some suggestions/wishes (if that would be possible):

    1) fix the goto so that it respect railroads (currently it behaves as if railroads are just roads)
    2) get rid of fadein-fadeout - generally in the game (e.g. when exiting a city screen) - In the later stages, the screen is blinking too much to my liking
     

    Attached Files:

  10. darkpanda

    darkpanda Dark Prince

    Joined:
    Oct 28, 2007
    Messages:
    601
    Thanks for the heads-up :) If there's anything to expect, I guess it's better, more optimal movements from the AI, but honestly, even if this occurs, I don't really know how easily it can be detected...

    Unfortunately, my CRCs are both different:
    original: 9489A0E8
    patched: ED0EF76B

    I downloaded total commander 8.01 just for this purpose, running on WinXP SP2 32-bit...

    But by displaying the path-finding data in your save file file, I guess it worked:

    CIVIL0.SVEpathfind.PNG

    The only way to be sure is for you to start a game with your original CIV.EXE and verify that is fails to generate the pathfind data (just save at 3980 BC, it's enough).

    It would be nice, but I'm afraid taht a side effect could be the AI getting stuck in endless loops while moving their own units on railroads! You can try to set the movement rate of any unit (e.g. militia) to 99, and you'll see that the AI will spend every single move they can, so I suspect that on railroads, they will just carry on moving forever...
    But that's a fun thing to investigate though, I'll let you know if anything can be done (should be the topic of a new thread)

    This one is farther from my reach at the moment, will not progress on it anytime soon.
     
  11. darkpanda

    darkpanda Dark Prince

    Joined:
    Oct 28, 2007
    Messages:
    601
    I can now confirm that Civ re-generates this ocean pathfind data when load a savegame... It is interesting, because there is an explicit call to the very same routine that is used at map generation time, and it's also the one of the only specific routines called when loading a savegame, besides loading the SVE and MAP data itself...
     
  12. darkpanda

    darkpanda Dark Prince

    Joined:
    Oct 28, 2007
    Messages:
    601
    This is actually the next turn when Civ Rankings will be displayed (when a famous anthologist finishes his epics...).

    This value is calculated by Civ as follows:
    - At the start of the game, it is a random value between 0 and 50 (exc.)
    - Whenever the turn is reached, the next turn is computed as: current turn + 20 + a random value between 0 and 40 (exc.)
     
  13. darkpanda

    darkpanda Dark Prince

    Joined:
    Oct 28, 2007
    Messages:
    601
    From the CIV code, it seems those 4 data blocks are closely related:
    • unknown18 and unknown19 are actually 2 sets of coordinates (X = unknown18 and Y = unknown19)
    • unknown16 is an "AI strategy" value similar to the ones in the "Per Continent AI strategy"
    • for those 3 data sets (and presumably for unknown17 as well), there are 16 bytes per Civ, i.e. totally 16*8 = 128 bytes
    • so, it looks like unknown16, unknown18 and unknown19 are used to store an array of up to 16 locations of interest for a computer-controlled Civ, in addition to the per-continent Civ strategy
    • using Civ version 474.01, those locations can be seen by activating the debug mode (Shift+56) and pressing F1, then selecting an army, then pressing F1 again, as illustrated below: (the S seems to mean "Settle")

    I couldn't discover much more so far, but all those data elements are involved in many code parts that seem related to AI handling, so there is still a real lot to investigate.

    To be continued...
     

    Attached Files:

  14. darkpanda

    darkpanda Dark Prince

    Joined:
    Oct 28, 2007
    Messages:
    601
    Just found the routine that handles this code, this data is related to "Civ Epics" (or Anthologies), those random events that happens every century or so, that rank the "Best Civs in the world":
    • it is simply a list of 8 shorts, 1 for each Civ, including Barbarians.
    • For each non-Barbarian Civ, this value is the sum of all ranks in the past (1st rank earns 7 points, 7th rank earns 1 point)
    • The Barbarian short accounts for the number of epics that occurred so far

    That's one more down, we're getting there !
     
  15. Mize

    Mize Warlord

    Joined:
    Jun 17, 2011
    Messages:
    290
    Nice... I'm very interested to know how the 'earthquake' event happens. With all your revelations about how land is grown etc., I wonder if earthquakes have something to do with landmasses, because I've often gotten consistent earthquakes in a single city in some games.

    EDIT:
    I now realize, that by 'random events' you meant something completely different. But still, the question stands. :)
     
  16. darkpanda

    darkpanda Dark Prince

    Joined:
    Oct 28, 2007
    Messages:
    601
    Interesting indeed :)
    I have found the routine for "city disasters", and although there is a good part of randomness, there are also some game rules constraint on them... Typically, earthquakes can only happen if your city has at least on Hills square in its direct 8 neighbour squares.

    As a general rule, it seems disasters cannot happen in cities with less than 5 in population...

    I am still looking at all the details, but will keep you posted - maybe in a separate thread...
     
  17. darkpanda

    darkpanda Dark Prince

    Joined:
    Oct 28, 2007
    Messages:
    601
    This data block lists the IDs of Civs who first discovered a Tech Advance:
    • It contains 1 Civ ID (2-byte word) for each of the 72 techs, totally 2*72 = 144 bytes
    • For each tech, this value is initialized to 0
    • Whenever a Civ is the first to discover a tech, its Civ ID is assigned to the corresponding word entry in this data block
    • The firstly discovered Techs are highlight in bright white in the Civ info screen shown when pressing F7 multiple times in DEBUG mode (Shift+56, Civ v 474.01):

    This data block contains the cross-totals of Units from 1 Civ destroyed by 1 other Civ:
    • It contains totally 1 cross-total (2-byte word value) per attacking Civ (8), per defensive Civ (8), so totally: 2*8*8 = 128 bytes
    • The bytes are arranged by attacking Civ first, i.e. one 16-byte block for each attacking Civ A
    • In each 16-byte block, each 2-byte word represents the number of units belonging to Civ B that were destroyed by Civ A
    • Those values can also be seen in DEBUG mode in Civ 474.01 (Shift+56), by selecting the Advisors Menu, then the Military Advisor, and then pressing a key to show the 2nd screen (this must be done through the menu as the F2 key will show the regular Military status report):
     

    Attached Files:

  18. darkpanda

    darkpanda Dark Prince

    Joined:
    Oct 28, 2007
    Messages:
    601
    Having a second look at how those arrays are handled by Civ's code, it seems like values in unknown16 can only be -1 (0xFF) or 0, while values in unknown17 can only be 0 or 2... In addition, the value of a byte in unknown16 (-1 or 0) acts as a "switch" that controls whether the location of interest is active or not...

    Some other parts of the code are checking whether the value in unknown16 is greater than 2, which seems very strange given there is no assignment anywhere to a value different from -1 or 0...

    It doesn't explain the complete logic behind this data, but at least it is complete from the perspective of analyzing the SVE contents. And this makes me happy :)
     
  19. darkpanda

    darkpanda Dark Prince

    Joined:
    Oct 28, 2007
    Messages:
    601
    This value (2-byte word) is the number representing the maximum number of discovered Techs by a Civ, among all Civs...

    For example, if there are 4 Civs, that have respectively discovered 9, 16, 13, and 7 techs, this value will be 16, which the highest value among all Civs.
     
  20. darkpanda

    darkpanda Dark Prince

    Joined:
    Oct 28, 2007
    Messages:
    601
    Quite logically, the most obscure data is deciphered last, after all the obvious stuff has been dealt with...

    I must say that without looking at the code, it would have been quite complicated to understand the purpose of this 2-byte word, but after searching for it in CIV.EXE, I discovered it is rather powerful...

    To sum up: this byte is a set of bitflags used to enabled/disable several game functionalities.

    If you set it to 0, i.e. all functions disabled, the following happens for all Civs:
    • cities cannot build improvements (except for Wonders)
    • research is stopped
    • civil disorder never happens
    • terrain cannot be mined or irrigated (road, railroad, and de/re-forestation still work)

    I tried to set the bitflags 1 by 1, but I am not exactly sure of each bit's role yet, except for bit 2 (0x2) which enables the selection of City improvements in the build menu.

    At game start, the first 4 switches are set to 1 - in other words, the word is set to value 0x000F (15).

    From a look into CIV.EXE code, it seems that only those four flags are used to control the execution path, meaning that the remaining 12 bits are left unused

    I started a game with this flag set to 0, in Prince level, playing Egyptians, and I must say it was quite an interesting paradigm to only be able to build Settlers and Militia, and try to fight my way through the game that way...
    Eventually, conquering enemy cities gives you access to Phalanx, Triremes, Cavalry and even Catapults (starting Techs of AI Civs), but not before literally hundreds of militia have been thrown against fortified phalanx to gain terrain square by square...

    Anyhow, this makes up for the last officially "unknown" byte in the SVE file format...

    Well, not exactly... There are still some unknowns inside City and Unit data that has to be deciphered, but at least we know their scope is limited to cities or units.
     

Share This Page