...a lot of it boils down to what can be modified graphically and how easy it is.
I worked on the Civ IV Dune Wars mod off and on since 2009, and it think it's fair to say it's one of the most comprehensive total conversions of Civ IV. There is however quite a long list of things that things that we did with the Civ IV graphics engine that aren't possible in CiV, most notably:
1) Removing the oceans to create a sand sea that units could walk across.
2) Heightmap editing of terrain: Turning Hills into Sinks (valleys) and Mountains into Hills then re-adding Peaks as impassable features.
3) Editing visual effects to make Sandstorms, Spice Blows, Chemical Trooper gas attacks, Sonic Tank attacks, etc.
4) Creating any number of terrain types as required each with their own graphical style. Using this to create areas of Spice in the sand sea.
The scope to turn the CiV earth into another world is quite limited (unless you have the tools and access to things that the Beyond Earth dev team had!). Compare this with Civ IV where you had Final Frontier, FFH2, Dune Wars, Planetfall, Mars Now... someone even mod-modded FFH2 into a dungeon crawler with multiple maps. I'll be surprised if Firaxis
ever match the modability of Civ IV.
There is also a chicken-egg problem with non-real-world total conversion mods - you need quite a large amount of artwork to get the project going, but without an established project graphical artists might not be too motivated to create art for it. Also these total conversions take a lot of work. FFH2 took 3 years of solid work from quite a large team of designers, coders and artists. Dune Wars was also a team effort of 4 or 5 people at times. A lot of CiV mods seems to be the work of individuals or small teams in general meaning it is much harder to realise a grand vision.
Also EVERYTHING graphical is easier (and therefore more enjoyable and fun) in Civ 4. You can reskin units and see the changes instantly in Nifskope. With CiV you have to edit the model file to do a reskin and you can't see the results without using the pretty useless Nexus 3D Viewer, or Blender or the game itself.
I only started working on what became Nexus Buddy 2 in Spring 2013 to try and start to get to a place where reliable modification of CiV 3D assets was possible. There have been numerous breakthroughs and much is now possible from editing 3D Leaders to editing animations.
When Civ 4 modding was at its peak and CiV was first released there was a much larger number of graphic modders with Blender skills. Sadly very few of these made the switch to CiV and those that did have mostly disappeared. I do wonder whether if we'd had the tools and techniques we have now back in 2010 that might have been different.
Hopefully Firaxis will take more of an XCOM2 approach with Civ VI and build it to support rather than frustrate graphical modders and allow non-graphical modders to make graphical changes. For example why not have a simple to use 3D Leader making tool that works just like the XCOM2 character customization? A basic stock of attire from different historic eras would probably allow uses to create 90% historic leaders and modders could add additional items if they had the skills.
lanetfall mod overshadowing Civ:BE. Let's hope that this time they dont have such intentions 
