Extract GR2 to Blender with mesh AND skeleton and re-export to GR2

Discussion in 'Civ5 - Modding Tutorials & Reference' started by Deliverator, May 6, 2013.

  1. Deliverator

    Deliverator Graphical Hackificator

    Joined:
    Feb 12, 2008
    Messages:
    4,720
    Location:
    London, UK
    This is a tutorial on how to:
    A) Extract 3D models, including their meshes AND skeletons, from the GR2 format used in Civ 5 into Blender.
    B) Get 3D models from Blender back into the GR2 format so that they can be used in Civ 5.


    Background

    These methods have great potential for Civ 5 unit graphics and potentially other graphics such as leaders, buildings and improvements. The vast majority of community made unit art for Civ 4 was done by rigging custom meshes to vanilla animations. Until recently this has been a really difficult pattern to follow in Civ 5 because it hasn't been possible to extract the skeletons and the meshes of vanilla units. Some people such as danrell have had some success copying skeletons by hand but this is very slow and un-fun work. Now it is simple and quick to create precise Blender templates direct from vanilla GR2 files via the NB2 format, or the more recent CN6 format for improved accuracy.


    Required Software Setup

    For these processes you will need to unpack the Civ 5 art resources (I recommend using Dragon Unpacker) and have the following software installed:

    Nexus Buddy 2
    Blender 2.49b OR Blender 2.79
    Deliverator Blender Scripts

    (Optional) Civ 5 SDK Old Version with working Nexus 3D Viewer *

    * This folder can be unzipped anywhere and sid meier's civilization v sdk/Nexus/x86/Nexus.exe will still work.

    These processes have been tested on Windows 7, 8 and 10.
    __________________________________________

    For my example case I am going to export and re-import archer.gr2, one of the Archer models. My complete working directory at the end of the process is uploaded here so you can play along at home.

    Extracting Civ 5 GR2 into Blender

    A1. Copy all files relating to the unit to a working folder. The simplest way is to do a file search within your unpaked resource directory (in my case C:\Program Files (x86)\Steam\steamapps\common\sid meier's civilization v\resource) for the name of the unit and just copy everything. In my case I searched resource for "archer" and copied everything to a folder called UnitWork_archer - it picks up Horse Archer and Camel Archer files too but that's not important. It helps to have everything in one place before you start. For the next step to work properly your working directory should include no spaces in the path.

    A2. Nexus Buddy 2: Open the .gr2 file, switch to the Advanced Tab and click Export to NB2. If you have a file that contains multiple Models such as a Wonder graphic you can switch which one you want to export or just export all.

    Note: You may have to load the relevant string database (a .gsd file) first using Load String Database under Advanced Actions before loading .gr2 file.

    A3.
    Blender 2.49b: Import the NB2 file using my the import script: Scripts -> Import -> Nexus Buddy 2 (.nb2).
    OR
    Blender 2.7+: Import the NB2 file using my the import addon script: Import -> Nexus Buddy 2 (.nb2).

    Alternatively, use the .cn6 format using the "Export to CN6" Button and the relevant import script. The .cn6 format preserves the vertex normals, binormals and tangents used for shading unlike the .nb2 format.


    From Blender to Civ 5 GR2

    For getting graphics from Blender back into Civ 5 gr2 there are two formats available, BR2 and FBX.

    The BR2 is reliable for mesh data and supports multi mesh, multi material models so is the preferred option.

    The FBX import is reliable for skeleton and animation data but often corrupts mesh data particularly for models with multiple meshes involved.

    So the best techniques for getting graphics from Blender in .gr2 format are:


    If you are rigging custom or converted meshes to a Civ 5 skeleton and animations use the following process:

    B1. Export the unit to BR2 format.

    B2. Make a copy of the original .gr2 file you are using the skeleton from - giving it a new name e.g. warrior_target.gr2.

    B3. Open the copied file in Nexus Buddy 2 and select Overwrite Meshes from BR2 in the Advanced tab. Select the .br2 files you exported from Blender. This will keep the skeleton in the .gr2 completely unchanged but will completely replace the meshes with the data from the .br2 file. It doesn't matter if the .br2 file has more meshes than the original .gr2 had - it will still add them into the file. Following this, you will have a .gr2 with the original units skeleton but with your meshes from Blender.

    B4. Use Nexus Buddy 2 to fix up your materials and textures - you'll probably need to remove the existing materials first before creating new ones.

    This process allows custom multi-mesh units to made in a reliable way with none of corruption issues of using FBX. Exactly the same method also works for 3D Leaders. For Wonders, Improvements, etc you can select the model whose meshes you want to overwrite - more info here.

    Alternatively to steps B2-4, now that the BR2 import process has improved in reliability you can just open the BR2 (using the Open BR2 button) to directly convert to GR2 and not use FBX at all.


    If you have custom or converted skeleton and animations then do the following:

    C1. Export from Blender to both FBX and BR2 using the scripts in Deliverator Blender Scripts.

    C2. Open FBX in Nexus Buddy 2.

    C3. Use Overwrite Meshes from BR2 to make sure your meshes are uncorrupted.

    C4. Use Nexus Buddy 2 to fix up your materials and textures.

    Alternatively to steps C1-4, now that the BR2 import process has improved in reliability you can just export to BR2, open the BR2 to directly convert to GR2 and not use FBX at all.
     
    Last edited: Nov 6, 2019
  2. Deliverator

    Deliverator Graphical Hackificator

    Joined:
    Feb 12, 2008
    Messages:
    4,720
    Location:
    London, UK
    Reserved for more tutorial.
     
  3. Deliverator

    Deliverator Graphical Hackificator

    Joined:
    Feb 12, 2008
    Messages:
    4,720
    Location:
    London, UK
    Old Tutorial

    Spoiler :
    This is a tutorial on how to:
    A) Extract 3D models, including their meshes AND skeletons, from the GR2 format used in Civ 5 into Blender.
    B) Export 3D models from Blender back into the GR2 format so that they can be used in Civ 5.


    Background

    These methods have great potential for Civ 5 unit graphics and potentially other graphics such as buildings and improvements. The vast majority of community made unit art for Civ 4 was done by rigging custom meshes to vanilla animations. Until now this has been a really difficult pattern to follow in Civ 5 because it hasn't been possible to extract the skeletons and the meshes of vanilla units. Some people such as danrell have had some success copying skeletons by hand but this is very slow and un-fun work. Now hopefully we can start building a library of .blend files for each unit so that anyone who can rig meshes to skeletons in Blender can make new unit graphics for Civ 5. Woop!

    Here is a collaborative Google spreadsheet for tracking which units have been extracted to .blend files. If you do a conversion then please upload your zipped .blend to the CFC database here and update the spreadsheet.

    Required Software Setup

    For these processes you will need to unpack the Civ 5 art resources and have the following software installed:

    Civ 5 SDK Old Version with working Nexus 3D Viewer (a) (b)
    Nexus Buddy 0.1a (a) (c)
    grnreader98 1.4.0.3 (a)
    Deliverator versions of Milkshape 3D ASCII Import and FBX Export Blender scripts (a)(d)
    Blender 2.49a
    Milkshape 3D 1.8.4
    GrannyViewer (optional) (a)(e)

    (a) Included for convenience in the Civ 5 3D Tools download
    (b) If not downloading Civ 5 3D Tools then download and merge part1 and part2 to make one sid meier's civilization v sdk folder. This folder can be anywhere and sid meier's civilization v sdk/Nexus/x86/Nexus.exe will still work.
    (c) This program should be at .../sid meier's civilization v sdk/Nexus/x86/IndieStoneNexusBuddy_01a.exe - it is useful to make shortcut to this.
    (d) The Deliverator version scripts should overwrite the scripts of the same names in the Blender scripts directory - by default this will under ../Users/<username>/AppData/Roaming/Blender Foundation/Blender/.blender/scripts. You can backup/copy the existing files if you want.
    (e) Tip - make this the default program for opening GR2 files for quick checking

    These processes have only been tested on Windows 7.

    __________________________________________

    For my example case I am going to export and re-import archer.gr2, one of the Archer models. My complete working directory at the end of the process is uploaded here so you can play along at home.

    Extracting Civ 5 GR2 into Blender

    A1. Copy all files relating to the unit to a working folder. The simplest way is to do a file search within your unpaked resource directory (in my case C:\Program Files (x86)\Steam\steamapps\common\sid meier's civilization v\resource) for the name of the unit and just copy everything. In my case I searched resource for "archer" and copied everything to a folder called UnitWork_archer - it picks up Horse Archer and Camel Archer files too but that's not important. It helps to have everything in one place before you start.
    Spoiler :

    A2. Nexus Buddy 0.1a: Open the .gr2 file and immediately save with a new name. In my case, I opened archer.gr2 and saved it as archer1.gr2.
    Spoiler :

    A3. grnreader98: Using the command "grnreader98.exe <GR2FileNameWithPath> -a -t" convert the .gr2 from step 2 into an .smd file. You should see something like this:



    You could set up a .bat file to execute this to avoid having to use the command prompt.

    A4. Milkshape 3D 1.8.4: Select Import-> Half-Life SMD and select the .smd file you just generated. Uncheck "Rename Bones" in the dialogue - leave Triangles and Skeleton checked. Now select Export -> Milkshape 3D ASCII and enter a filename to export to.
    Spoiler :

    A5. Blender 2.49a: Import the Milkshape3D ASCII .txt file using my customized script: Scripts -> Import -> Milkshape 3D ASCII (.txt). Save your .blend file. Pat yourself on the back for creating a Blender unit template!

    (You may need to make some manual tweaks - see Blender Tweaks below.)

    Exporting from Blender to Civ 5 GR2

    B1. Blender 2.49a: Export to FBX using my customized script: Scripts -> Export -> Autodesk FBX (.fbx). Leave all settings as default.

    B2. Nexus Buddy 0.1a: Open the FBX file. Add a Material using UnitShader_Skinned. Select the Material. Set the BaseTextureMap to be your _diff.dds and the SREF map to be your _sref.dds, in my case archer_diff.dds and archer_sref.dds. Select the mesh (which will be called Object unless you've renamed it) and assign the Material to it. Save the file to GR2 format.
    Spoiler :

    B3. You can now quickly check that the GR2 looks OK in GrannyViewer. The Nexus Buddy occassionally fails or does something strange for no reason so you may have to try step 2 a couple of times to get a working GR2.

    B4. Update the Mesh tag in the .fxsxml file in your working directory. In my case <Mesh file="archer2.gr2"/>.

    B5. Run the old version of Nexus. Select AssetViewer and open the .fxsxml file you just edited. Click the green-cube-with-red-arrow icon to open the 3D Viewer and double click the fx_triggers_<unitname>.ftsxml file to open the Animation Triggers window. With the 3D Viewer and Animation Triggers side-by-side you can click on the different actions to confirm that your model is rigged correctly.
    Spoiler :

    Blender Tweaks

    In this section I'm going to log some of the tweaks I have had to make at the Blender stage to get an accurate template.

    It may be that after step 5 in the extract process, importing to Blender, that there are some issues with the model that require some adjustments to be made. I would advise always checking the export with animation poses in Nexus 3D Viewer - things can look fine in GrannyViewer but be a real mess when looked at in 3D Viewer.

    For example, when I first followed the full extract and re-import process the first time around this was the result in Nexus 3D Viewer.
    Spoiler :

    Dude, what happened to your arm? The beauty of this process is I can go back to my blender file and take a close look at what the issue could be. I didn't have to look to hard to see that the skeleton rotation was off and not lined up with the mesh.
    Spoiler :

    So I rotated it to where it looked like it should go. Opening Nexus 3D Viewer/GrannyViewer with the original GR2 and displaying bone connections and axes can help you line up the skeleton to the mesh - where should the bone go through the hand? How far should the arm bone be from the back of the mesh? etc

    My second attempt was much better, but still not quite lined up, the bow wasn't quite sitting in the hand right. Following another 3D Viewer comparison I was able to see that the skeleton just needed to be move forward a fraction relative to the mesh - a quick fix in Blender and I have a perfect unit template!
    Spoiler :


    Old Import Tutorial:
    Spoiler :
    B1. Blender 2.49b: Export to FBX using my customized script: Scripts -> Export -> Autodesk FBX (.fbx). Leave all settings as default.

    B2. Nexus Buddy 2: Open the FBX file. Add a Material using UnitShader_Skinned. Select the Material. Set the BaseTextureMap to be your _diff.dds and the SREF map to be your _sref.dds, in my case archer_diff.dds and archer_sref.dds. Select the mesh (which will be called Object unless you've renamed it) and assign the Material to it. Save the file to GR2 format.

    B3. You can now quickly check that the GR2 looks OK in GrannyViewer. The FBX into GR2 occassionally fails or corrupts the mesh for no reason so you may have to try step 2 a couple of times to get a working GR2.

    B4. Update the Mesh tag in the .fxsxml file in your working directory. In my case <Mesh file="archer2.gr2"/>.

    B5. Run the old SDK version of Nexus. Select AssetViewer and open the .fxsxml file you just edited. Click the green-cube-with-red-arrow icon to open the 3D Viewer and double click the fx_triggers_<unitname>.ftsxml file to open the Animation Triggers window. With the 3D Viewer and Animation Triggers side-by-side you can click on the different actions to confirm that your model is rigged correctly.
    Spoiler :
     
  4. Deliverator

    Deliverator Graphical Hackificator

    Joined:
    Feb 12, 2008
    Messages:
    4,720
    Location:
    London, UK
    Blah Blah Blah Blah Blah Blah...
     
  5. Pazyryk

    Pazyryk Deity

    Joined:
    Jun 13, 2008
    Messages:
    3,584
    It's great to see you working on this again Deliverator! This gives me hope that we'll have more than 3 fantasy units someday. Are you planning to update your Civ4 -> 5 unit conversion thread at some point? (It may be best to start a new thread for that, since installation process is so arcane now...)
     
  6. Deliverator

    Deliverator Graphical Hackificator

    Joined:
    Feb 12, 2008
    Messages:
    4,720
    Location:
    London, UK
    It's good to be back!

    What do mean by the installation process being arcane?

    If you mean the way that ArtDefines work now it took a while to figure that out today - and I only managed it by looikng at Ekmek's recent B52 unit plus trial and error.

    Otherwise I've updated the download links in the Civ 4 -> Civ 5 tutorial. AFAIK the information there is still relevant and works. Let me know what you think needs updating.

    Overall, rigging meshes to Civ 5 skeletons and animations is going to give better quality results seeing as they are actions in Civ 5 that don't have equivalents in Civ 4. Also, converting a Civ 4 unit and animations is more laborious than using Civ 5 anims (which is why few have happened) so it is probably only going to be worth considering for units that have no equivalent animation in Civ 5 such as dragons and other exotics.

    I am surprised that there haven't been more fantasy conversions though given the huge popularity of FFH2 - I didn't even know there were 3!
     
  7. Pazyryk

    Pazyryk Deity

    Joined:
    Jun 13, 2008
    Messages:
    3,584
    I'll try again. I made an effort on this about a year ago (or maybe longer). What I remember is going through multiple threads trying to find the exact outdated versions of various software. It's a steep hill to climb if you don't have previous experience with 3D art.

    We have exactly: 1 dragon, 1 wolf and 1 assassin. It's a vicious circle: There are no fantasy units because there are no good fantasy mods. But there are no good fantasy mods because there are no fantasy units. :sad:
     
  8. Deliverator

    Deliverator Graphical Hackificator

    Joined:
    Feb 12, 2008
    Messages:
    4,720
    Location:
    London, UK
    If you can get Blender 2.49b installed and running then that would be a great start. If you can manage to rig a Civ 4 fantasy mesh to Civ 5 template that would teach you a lot of blender skills. Probably easier to do that than do a full Civ 4 -> Civ 5 conversion.

    Yeah, I understand that vicious circle thing. I'm also not sure if we can re-texture the terrain in mods - that would do a lot to help create a more fantasy vibe. What do think the top wow-factor fantasy units would be to get the ball rolling?
     
  9. Deliverator

    Deliverator Graphical Hackificator

    Joined:
    Feb 12, 2008
    Messages:
    4,720
    Location:
    London, UK
    Here is a collaborative Google spreadsheet for tracking which units have been extracted to .blend files. If you do a successful conversion then please upload your zipped .blend to the CFC database and update the spreadsheet.

    A bit of an experiment in cloud-based collaboration... ;)
     
  10. Gedemon

    Gedemon Modder Super Moderator

    Joined:
    Oct 4, 2004
    Messages:
    10,016
    Location:
    France
    there are Danrell's templates, re-uploaded by Nutty here.

    I can add them if you want but the spreadsheet seems to be read only ? (or it's me having a problem :think: )

    I think we need (well I need) a small step by small step instruction for dummies, especially for the blender part. I've tried most of my weekend (and failed miserably) to convert one of Snafusmith's jetfighter from civ4, and I have the feeling that I won't find any easier conversion :blush:
     
  11. Deliverator

    Deliverator Graphical Hackificator

    Joined:
    Feb 12, 2008
    Messages:
    4,720
    Location:
    London, UK
    I've made the speadsheet editable by everyone now. Yes, add danrell's unit templates. The process in this tutorial will create more accurate results, but it is still impressive the number of skeletons danrell created by hand. We can replace his handmade ones with extracted ones eventually but it's good to have them recorded for now.

    There is definitely a learning curve with Blender. As I said to Pazyryk doing a full Civ 4 to Civ 5 conversion is much harder than rigging a Civ 4 mesh to existing Civ 5 skeleton and animations. That would be the best way to convert your jetfighter anyway. I'm sure there are some decent guides here on rigging - but if there's not I'll try and do a baby-steps one.
     
  12. Deliverator

    Deliverator Graphical Hackificator

    Joined:
    Feb 12, 2008
    Messages:
    4,720
    Location:
    London, UK
    Remembered that I wrote a tutorial on making Civ 4 units use Civ 5 animations here. It may need some updating or fleshing out for 2013.

    Bernie has a more detail description of rigging a mesh to a skeleton from where he says "NOW we open BLENDER......." in this tutorial. The process for rigging a Civ 4 mesh (imported from NIF) to a Civ 5 skeleton is the same.

    I can probably create a simpler keystroke-by-keystroke, click-by-click tutorial for rigging to help the next generation of modders. ;)
     
  13. Gedemon

    Gedemon Modder Super Moderator

    Joined:
    Oct 4, 2004
    Messages:
    10,016
    Location:
    France
    Thanks, I think I've added them all.
     
  14. Wolfdog

    Wolfdog Unit Butcher

    Joined:
    Jun 29, 2012
    Messages:
    659
    Location:
    Australia
  15. Pazyryk

    Pazyryk Deity

    Joined:
    Jun 13, 2008
    Messages:
    3,584
    This is exactly the kind of understanding that I don't have. Doesn't help if I'm trying to tackle the very difficult first.

    Yes. Civ5 is just too bright and sparkly. It's more fitting for a Ponies and Unicorns mod.

    For my mod, see pg 6 and 38 of the Éa Manual here. A basic unit line-up of baddies would be a good start: orcs (warriors and some light to heavy infantry types), goblin archers (basic and elite), some mounted guys (on wolves or hogs or whathaveyou), and then some big nasty folks like ogres, hill giants, trolls, etc. After that, your standard elves and dwarves I guess (though I don't have dwarves in my mod).

    And animals and monsters of various sorts. I was thinking of implementing a dragon that had both melee attack and a ranged "bomber" attack. (This could be done by creating two dragon units and swapping them out via Lua. I already do some stuff like this in my mod...)
     
  16. Deliverator

    Deliverator Graphical Hackificator

    Joined:
    Feb 12, 2008
    Messages:
    4,720
    Location:
    London, UK
    Cool! Let's see how many we can convert before we all have to buy Milkshape licenses - I have 27 days and counting! ;)

    It seems you've found where I put mine. Utility Programs isn't exactly accurate but until we have a Blender Unit Templates section...

    Also - I've added a new section on Blender Tweaks in post 2.

    I'll probably use a good FFH2 fantasy unit for my updated rigging Civ 4 mesh to Civ 5 skeleton tutorial. Kill two birds with one stone! :)
     
  17. hangman

    hangman almost-scientist

    Joined:
    Dec 31, 2010
    Messages:
    776
    Location:
    New Yoronavirus
    Yes, please do a baby-steps tutorial for civ4 unit conversion. I always get hung up on things that tutorial writers deem obvious :)
     
  18. Novator

    Novator Doomed to finish everything from that cursed list.

    Joined:
    Jul 3, 2012
    Messages:
    267
    But I still can't open textures of LHs...
     
  19. Ekmek

    Ekmek on steam: ekmek_e

    Joined:
    Aug 7, 2002
    Messages:
    6,060
    Location:
    San Diego, California
    LH textures have a header file added to them so they can't be opened it by GIMP EDIT: Or anything else
     
  20. Novator

    Novator Doomed to finish everything from that cursed list.

    Joined:
    Jul 3, 2012
    Messages:
    267
    They can't be opened by anything, even NVDIA tools. I can see only black square instead of texture. There are also .fic files. I can't open them as textures.
     

Share This Page