Important Note: this isn't a guide for modmodding, though modmodders may still find it worth by reading this.
We have recently completed The Merge, which completed the work of integrating C4DF, CSD, More Luxuries, and various modular elements like Corporations into the main VP folders.
Files were reorganized into correct folders and renamed to clear and precise names.
Almost all SQL and XML have been rewritten. This also dug up various long-hidden bugs and allowed them to be fixed.
Most unused game components (not in base game) have been removed.
Note that other than moving Events from (1) to (2), the (1) Community Patch folder is mostly untouched, so that can continue to act as a base for other mods.
In this post, base game always refers to Civilization V with the Brave New World expansion and all DLCs.
Assets:
All art assets are put here.
File extensions: dds, dge, fsmxml, ftsxml, fxsxml, ggxml, gr2
All of them are imported into VFS.
Since I didn't check the contents of these files, they're only vaguely sorted into folders, and there can be potentially unused assets.
Core Files:
Non-database, non-asset files.
File extensions: lua, xml
- Core Changes: Contains non-database (not starting with <GameData>) xml. Need to follow base game file names and imported into VFS to work. Currently has only farm_types.xml, which handles farm graphics.
- New Lua: Contains new functions that trigger on game events. These should be kept at a minimum. Each has a InGameUIAddin entry in <ModContent>.
- New UI: Contains new UI components that are not present in the base game. Each has a InGameUIAddin entry in <ModContent>.
- Overrides: Contains UI and event triggers that are already present in the base game. Need to follow base game file names and imported into VFS to work.
LUA:
Base game UI components. Unlike the Overrides folder in Core Files, this folder is for non-EUI only, and is not installed if the EUI option is selected in the installer.
The equivalent EUI folder is (3a) Compatibility Files/LUA.
All of them have the same names as base game files and are imported into VFS.
Mapscripts:
Contains Communitu_79a map and other base game maps that have to be edited to keep compatible with VP. All are imported into VFS and have a MapScript entry in <ModContent>.
EMP folder: identical to the Explorer's Map Pack DLC. VP unloads the DLC in order to get the difficulty blurbs to update. This folder adds it back. Never modify its contents.
Database Changes:
The main content of the Merge project. Contains ALL and ONLY files that modify the database. All of them have an UpdateDatabase entry in <ModActions>.
As many bulk inserts, updates and deletes are used, the order in <ModActions> and where you put each SQL statement matters a lot. More on that in later sections.
Main tables are database tables that have a unique "Type" column, and usually also has an "ID" column. Each row in main tables represents a game component, whether a building (Buildings), an AI military strategy (AIMilitaryStrategies), or a game option (GameOptions).
The types are commonly referenced in other main or non-main tables. If a reference is not met, i.e. the referenced type doesn't exist in the corresponding main table, at the end of mod loading, the line "Failed Validation" will show up on Database.log, which signals some errors in the SQL/XML.
Other, non-main, tables define interactions between game components. For example, the Policy_BuildingClassYieldChanges table defines which building class gets which yield type, and how much of it, if you have a certain policy.
Main tables usually have numerous columns, but most have default values of 0, false or null and don't have to be set. This makes XML more suitable for adding game components, as using mass SQL insert would have extremely long and unreadable rows filled with default values. However, it's also not ideal to insert every column of a row using XML, only the ones that are 1) text keys, or 2) not inter-related (e.g. PrereqTech, MoveRate), or 3) component-defining (e.g. IconAtlas, SpecialCargo). Any column that is easily bulk updated should be done using SQL.
Non-main tables have few columns and usually can be bulk inserted/updated/deleted via use of temp tables and where conditions, so SQL is always more suitable for changing them.
- CommunityOptions.sql: Community table values. Game options for the less-technical people who still know how to open a file. Always loads first.
- New<plural>.xml: inserts new main table entries.
- CustomModOptionChanges.sql, DefineChanges.sql, GameOptionChanges.sql, GameSpeedChanges.sql, WorldChanges.sql:
- Pre<singular>Changes.sql: bulk deletions or updates that need to be done before individual updates, e.g. categorizing resources or promotions.
- <singular>Changes.sql: individual updates of rows or columns of main tables, and insertions of non-main table entries that are related to the main table.
- <singular>Sweeps.sql: bulk deletions, insertions or updates of main table columns or non-main table entries.
- New<singular>Text.xml: inserts new text entries. Only Language_en_US table allowed. Other tables don't belong to the Text folder, which is used for localization only.
- <singular>TextChanges.sql: updates existing base game text entries, or those added in (1). Only Language_en_US table updates allowed. Move your unit text key updates elsewhere.
- Remapper.sql: the deletions of main table entries and the complete replacement of the UnitPromotions table leave many gaps in ID values, which would crash the game when loaded. This magical file re-inserts every affected main table to have their ID start at 0 without any gaps.
- Triggers.sql: VP doesn't use triggers for itself, but has some to save work for modmodders. This file always loads last and must not contain any statements other than CREATE TRIGGER.
We have recently completed The Merge, which completed the work of integrating C4DF, CSD, More Luxuries, and various modular elements like Corporations into the main VP folders.
Files were reorganized into correct folders and renamed to clear and precise names.
Almost all SQL and XML have been rewritten. This also dug up various long-hidden bugs and allowed them to be fixed.
Most unused game components (not in base game) have been removed.
Note that other than moving Events from (1) to (2), the (1) Community Patch folder is mostly untouched, so that can continue to act as a base for other mods.
In this post, base game always refers to Civilization V with the Brave New World expansion and all DLCs.
Overview
There are only 5 main folders left in (2) Vox Populi.Assets:
All art assets are put here.
File extensions: dds, dge, fsmxml, ftsxml, fxsxml, ggxml, gr2
All of them are imported into VFS.
Since I didn't check the contents of these files, they're only vaguely sorted into folders, and there can be potentially unused assets.
Core Files:
Non-database, non-asset files.
File extensions: lua, xml
- Core Changes: Contains non-database (not starting with <GameData>) xml. Need to follow base game file names and imported into VFS to work. Currently has only farm_types.xml, which handles farm graphics.
- New Lua: Contains new functions that trigger on game events. These should be kept at a minimum. Each has a InGameUIAddin entry in <ModContent>.
- New UI: Contains new UI components that are not present in the base game. Each has a InGameUIAddin entry in <ModContent>.
- Overrides: Contains UI and event triggers that are already present in the base game. Need to follow base game file names and imported into VFS to work.
LUA:
Base game UI components. Unlike the Overrides folder in Core Files, this folder is for non-EUI only, and is not installed if the EUI option is selected in the installer.
The equivalent EUI folder is (3a) Compatibility Files/LUA.
All of them have the same names as base game files and are imported into VFS.
Mapscripts:
Contains Communitu_79a map and other base game maps that have to be edited to keep compatible with VP. All are imported into VFS and have a MapScript entry in <ModContent>.
EMP folder: identical to the Explorer's Map Pack DLC. VP unloads the DLC in order to get the difficulty blurbs to update. This folder adds it back. Never modify its contents.
Database Changes:
The main content of the Merge project. Contains ALL and ONLY files that modify the database. All of them have an UpdateDatabase entry in <ModActions>.
As many bulk inserts, updates and deletes are used, the order in <ModActions> and where you put each SQL statement matters a lot. More on that in later sections.
Main Tables
Before we dig deeper, I need to explain main tables.Main tables are database tables that have a unique "Type" column, and usually also has an "ID" column. Each row in main tables represents a game component, whether a building (Buildings), an AI military strategy (AIMilitaryStrategies), or a game option (GameOptions).
The types are commonly referenced in other main or non-main tables. If a reference is not met, i.e. the referenced type doesn't exist in the corresponding main table, at the end of mod loading, the line "Failed Validation" will show up on Database.log, which signals some errors in the SQL/XML.
Other, non-main, tables define interactions between game components. For example, the Policy_BuildingClassYieldChanges table defines which building class gets which yield type, and how much of it, if you have a certain policy.
Main tables usually have numerous columns, but most have default values of 0, false or null and don't have to be set. This makes XML more suitable for adding game components, as using mass SQL insert would have extremely long and unreadable rows filled with default values. However, it's also not ideal to insert every column of a row using XML, only the ones that are 1) text keys, or 2) not inter-related (e.g. PrereqTech, MoveRate), or 3) component-defining (e.g. IconAtlas, SpecialCargo). Any column that is easily bulk updated should be done using SQL.
Non-main tables have few columns and usually can be bulk inserted/updated/deleted via use of temp tables and where conditions, so SQL is always more suitable for changing them.
File Name Conventions and Load Order
File names in the Database Changes folder mostly follow these patterns (in load order):- CommunityOptions.sql: Community table values. Game options for the less-technical people who still know how to open a file. Always loads first.
- New<plural>.xml: inserts new main table entries.
- CustomModOptionChanges.sql, DefineChanges.sql, GameOptionChanges.sql, GameSpeedChanges.sql, WorldChanges.sql:
These are foundational values for the mod, and should always go before other changes.
- Pre<singular>Changes.sql: bulk deletions or updates that need to be done before individual updates, e.g. categorizing resources or promotions.
- <singular>Changes.sql: individual updates of rows or columns of main tables, and insertions of non-main table entries that are related to the main table.
Tradition.sql etc. also belong to this category.
- <Civilization>.sql: updates Civilization UA, UU, and UB/UI. Any pseudo unique components like Ordo, Mission and West Point also belong here.Needs to be loaded after all UnitChanges and BuildingChanges files.
- BarbarianUnitChanges.sql, CityStateUnitChanges.sql, PolicyUnitChanges.sql: updates barbarian, CS and policy exclusive units. CS and policy units are also given their exclusive unit class here.Needs to be loaded after all UnitChanges files.
- <singular>Sweeps.sql: bulk deletions, insertions or updates of main table columns or non-main table entries.
As one bulk insert statement is faster than many individual update statements, non-main tables are commonly deleted (either in here or in the PreChanges file) and re-inserted. This also makes referring to base game values unnecessary for modders.
Some main table columns are highly inter-related or follow a pattern, like unit costs and building capture chance. These are suitable for bulk updates, usually done by first assigning a default value to all rows, then updating with conditions.
Both Changes.sql and Sweeps.sql can do bulk updates, and sometimes the line between them is muddied. Just bear in mind that Sweeps.sql is always loaded after ALL Changes.sql files. This is particularly important for Buildings and Units.
- New<singular>Text.xml: inserts new text entries. Only Language_en_US table allowed. Other tables don't belong to the Text folder, which is used for localization only.
XML is used instead of SQL since quotes are common and don't have to be escaped. However, there are still two SQL files (NewUnitText.sql and NewUIText.sql) inserting short text.
- <singular>TextChanges.sql: updates existing base game text entries, or those added in (1). Only Language_en_US table updates allowed. Move your unit text key updates elsewhere.
SQL is used for text changes instead of XML since the <Update> tag seems to fail to handle single quotes and still requires escaping double quotes.
- Remapper.sql: the deletions of main table entries and the complete replacement of the UnitPromotions table leave many gaps in ID values, which would crash the game when loaded. This magical file re-inserts every affected main table to have their ID start at 0 without any gaps.
- Triggers.sql: VP doesn't use triggers for itself, but has some to save work for modmodders. This file always loads last and must not contain any statements other than CREATE TRIGGER.