Guide to VP modding

[axatin]

Hi everyone,

following the announcement in this thread, I have written a first version of a guide to modding for Vox Populi. For now it only covers database modding and DLL modding, a section on lua modding isn't written yet. I hope everything's understandable, if you have any questions feel free to ask. Any comments and suggestions are also welcome.

 

[azum4roll]

I wish artwork modding could also be covered since that's probably the part that's least documented.

Another reason why we aren't using Lua for non-UI stuff:
The Lua interface is slow so it's a performance hit every time it's used. See for example how long it takes to generate a map and place starts, and how long it takes for the city view to load in later eras with the More Wonders mod.

 

[azum4roll]

Some corrections and additional info:

• The UI_bc1 folder and its contents should never be touched. For historical reasons, we're supposed to keep the folder identical to whatever original EUI version it is. Any change to it is applied by overwriting files via the (3a) mod or the VPUI.
• Additional steps that may be needed for manual installation from a specific commit (instead of a release version):
  • Compile the DLL, as included in the guide
  • Use ModBuddy to rebuild the .modinfo files (or manually edit them; the MP5 values don't matter), if there are any additional files or name changes in the respective mod
  • Check Database.log and xml.log for inevitable errors, as we're all bound to make mistakes
  • Still crash the game anyway because there's something wrong in the DLL code
• The English localization is stored in the table "Language_en_US"
  • Language_en_US is actually a view, unlike the other languages. It doesn't show up on the table list, but it shouldn't matter when you use SQL to query/update it.
• Regarding database column types:
  • SQLite itself is flexible and doesn't validate its own values. For example, you can insert text into a boolean column, and it won't complain. In fact, "boolean" is completely synonymous to "NUMERIC" in SQLite.
  • The C++ SQLite library always attempts to cast the values into the respective types when calling GetInt() or GetBool(). I'm not sure what happens if you try to GetInt() from a string value (the library source isn't available to us), but you definitely shouldn't do so.
  • The Lua GameInfo iterator treats the column types seriously. Any boolean column not typed exactly as "boolean" (case sensitive!) is treated as a number. This can be very troublesome as 0 the number is treated as true when passed as a condition!
  • Which means you should always type your boolean columns as "boolean", not "BOOLEAN" or "BOOL" or any variations. All existing columns have been corrected as of VP 5.0.
• If you add a row to the database, you must provide a value for every column that does not have a default value.
  • The default default value of SQLite is NULL. If a column is NULL and it doesn't have a default value, you don't have to specify it.
  • If you ever have to add <ColumnName>NULL</ColumnName>, it's usually wrong and you should double check it.
• I'm not sure about CvAssert doing nothing. When in debug mode, sometimes the game is crashed by an assert statement. Or maybe that's just ASSERT, which calls the base C++ assert (only in debug).
• Remember that a player object always exists for all 64 slots, to exclude the unused slots use the function 'isAlive'
  • This would also exclude players that are eliminated. To include all players that exist at the start of the game plus city states that may be created midgame, use isEverAlive() instead.
  • This also applies to CvTeam.
• "When a city changes owner, the original city object is destroyed and a new object is created."
  • It's the same for units when they're captured or upgraded; the original unit object is destroyed and a new object is created.
  • This is why we don't rely on IDs when passing around city and unit references.
  • Only some properties transfer over, and they're all explicitly specified in the code.
• "If the define is only used for game logic exclusive to VP, state that in a comment."
  • It's not used consistently. It's sometimes used just because the value is NOT used in CP, but not being gated by a MOD_BALANCE_VP check, which itself isn't consistent either.
  • We're slowly fixing these when going through the code.
• We recommend only using fixed-sized arrays for yield tables, as those are constant(ly small-)sized and there are already utility functions (SetYields() etc.) that are based on them. For other complex tables (there aren't that many), use vectors, sets, and hash_maps instead depending on use case (whichever is most memory/performance efficient).
  • Use fixed-sized vectors only if the table is expected to have a lot of entries for each primary type, and you need to iterate through them.
• "If necessary you can provide a more detailed explanation of your changes in the PR. If the lead developers request adjustments, address the issues by adding new commits to the branch and push them. Once the PR has been approved and merged into the master branch, you can delete the branch you had created for it."
  • It's highly recommended to stick to ONE commit per change, or similar changes (like fixing a bunch of text). If a lead developer requests adjustments, you can always undo the commit, make the adjustments, then recommit it. Note that your local branch is now diverged from the one on Github. You can now FORCE push your local branch to Github overwriting the previous (unadjusted) commit.
  • Warning: Only use force push when you're the only one making commits to the branch! And never to a master branch unless you know what you're doing.
  • If a branch's history is too muddied, you should consider starting over from a clean state by making a new branch, after saving your edited files elsewhere.
  • NEVER use the merge function, whether from remote to local or local to local. We're on a strict no-merge policy. Use REBASE instead.

Some amendments that will apply to VP 5.0 or later:

• The (1) structure has been overhauled like (2).
  • New tables are in separate New*Tables.xml files.
  • New table rows for main tables are in separate CoreNew*.xml files. Sometimes "Core" is omitted when there's no conflict with files of the same name in (2).
  • Insert/Delete/Update of non-main tables and Update of main tables are in Core*Changes.sql or Core*Sweeps.sql files.
  • New text are in CoreNew*Text.xml files.
  • Updates of existing text are in Core*TextChanges.sql files.
  • Triggers are in CoreTriggers.sql. I recommend not to touch these unless you know what they are for.
  • Database indices are in DatabaseIndexes.sql. These are for performance improvements for UI and some ancient parts of the DLL that don't use the database cache; don't touch these unless you know what they are for.
• It is very important to always include a null check on the pointers, as shown in the example. If a modmod deletes values from a database table, it may leads to gaps in the IDs of the table entries. Enum categories will be set for the missing IDs nonetheless, and the pointers corresponding to them will be null. Many crashes when using modmods were caused by not including null checks.
  • We're planning to add code in the DLL to automatically remove gaps from the database when the mod is loaded, hopefully included in 5.0. In that case, only a NO_* check is needed in some instances (where it's a possible value) and all null checks for *Info can be safely removed.
  • Out of index *Types values will still result in a null *Info, but we WANT those to crash (it always indicates a logical error) so we can fix them.

 

[Rick Drayson]

God bless your efforts that will likely increase the long term success of vox populi.

 

[Agiwan]

This is really big !

Thank you axatin

 

[Rick Drayson]

I took a look at the sourcecode for VP
Yep can now see why the plot unit limit no longer works lol
Biggest issue facing the idea of contributing is the visual studio requirements. Im not risking blowing up my vs2022 setup . Personally would need a virtual environment i guess. I also couldnt find vs2008

 

[azum4roll]

I don't think you need to config anything in Visual Studio itself? It should automatically build with the correct compiler version since the config is on the project, not the IDE.

 

[Rick Drayson]

VS needs the v90 platform toolset to compile it. I did try compiling with v143 that installs with vs 2022 but no Bueno . This is configured in your project config.
I could be wrong having never successfully compiled it.

 

[azum4roll]

Yes, so you need to install the old VS versions and it'll work (on any VS version).

 

[hokath]

I also couldnt find vs2008

Yes when I tried following that (old) guide, I couldn't find it either. I don't remember which part, it might even have been the microsoft site.

I don't know if it's kosher but, if we have those files already, perhaps we can host our own copy of them here?

 

[Rick Drayson]

Yes when I tried following that (old) guide, I couldn't find it either. I don't remember which part, it might even have been the microsoft site.

I don't know if it's kosher but, if we have those files already, perhaps we can host our own copy of them here?

Try

Microsoft Visual Studio 2005-2015 Professional Edition collection 8 files : Microsoft : Free Download, Borrow, and Streaming : Internet Archive

Microsoft Visual Studio 2005-2015 Professional...

archive.org

Link on right for dl options

 

[azum4roll]

I don't know if it's kosher but, if we have those files already, perhaps we can host our own copy of them here?

It's probably way too huge to host here.

 

[Rick Drayson]

I actually got it to compile .
The solution was VS 2008 express WITH SP1. Doesn't work without SP1 and you cannot install SP1 seperately. Use the link

Microsoft Visual Studio 2005-2015 Professional Edition collection 8 files : Microsoft : Free Download, Borrow, and Streaming : Internet Archive

Microsoft Visual Studio 2005-2015 Professional...

archive.org

 

[Tekamthi]

Big thank you to @axatin for this guide -- this is just what I need to start looking beyond the Lua and db understanding I currently have.

RE: a guide to Lua modding, what I'd suggest is just adding a section explaining how a Lua file interacts with the game (ie the settings required in the .modinfo to load it), how the Lua hooks and tests work, where to find the api's etc. Getting into actual Lua coding instruction is probably unnecessary beyond some simple examples, maybe. There are other resources for general Lua learning

The Lua interface is slow so it's a performance hit every time it's used

While this is true, maybe it won't always be a big deal... This project lives on nearly a decade and a half after the game's release, and I'm sure we'll still come back to it at least occasionally another decade from now. Maybe we have 10 ghz single core CPU clock speed by then, and who cares anymore about "slow" lua

Civ 5 db and Lua is very approachable for those new to modding, the dll a much bigger step; the former is still worth covering with the resource-hit warning imo

 

[Rick Drayson]

Hi everyone,

following the announcement in this thread, I have written a first version of a guide to modding for Vox Populi. For now it only covers database modding and DLL modding, a section on lua modding isn't written yet. I hope everything's understandable, if you have any questions feel free to ask. Any comments and suggestions are also welcome.

Can you add this to the repository please. Makes sense moving forward.

 

[azum4roll]

Maybe we have 10 ghz single core CPU clock speed by then, and who cares anymore about "slow" lua

Still probably isn't going to cut it if you call the DLL inside a triple-layered loop instead of caching the values you get from the DLL.

 

[Agiwan]

Hi, can we stick this thread on the main forum ? It really helps ! @Recursive

Moderator Action: Stickied. - Recursive

 

[balparmak]

Hi everyone,

following the announcement in this thread, I have written a first version of a guide to modding for Vox Populi. For now it only covers database modding and DLL modding, a section on lua modding isn't written yet. I hope everything's understandable, if you have any questions feel free to ask. Any comments and suggestions are also welcome.

Just read this, great work, thanks! I think it’d be nice to host the guide on GitHub (doesn't have to be the VP repo) so that improvements and adjustments can be implemented more easily. IIRC there are quite a few guides on Lua modding on Civfanatics, so that part could mostly be covered by linking to those and documenting the new Lua functions added by CP&VP. We’ve got some of them on the GitHub wiki, but it’s probably outdated, and editing the wiki in its current state is a chore. Maybe we should look into options for generating the wiki from markdown files.

I haven't created any original artwork, but have done a fair share of compressing, you can link this for compression standards for textures. Also, I’d recommend Git newcomers (or those with limited storage or slower internet) to use shallow clones, it's much faster and imho starting with a clean slate is always better when you're a beginner and making a lot of mistakes. I can write an alternative workflow section about that if the guide becomes editable.

@azum4roll If I'm not mistaken you've cleaned up a lot of poorly written code both from Firaxis and other modders, right? A section on best practices and pitfalls to avoid would be nice to keep up the coding quality, I learned C++ from imitating the code, which included #if defineds, the bane of a sweeper's existence, I wouldn't want to commit those errors again. Probably a section on what makes the code multiplayer compatible is also warranted.

 

[Tekamthi]

Lua functions added by CP&VP. We’ve got some of them on the GitHub wiki, but it’s probably outdated, and editing the wiki in its current state is a chore. Maybe we should look into options for generating the wiki from markdown files.

Yeah I still refer to this wiki often, and make use of much of the additions -- if there are hooks or functions not documented on the wiki, I don't know how to find/use them, and I imagine many modders are similar

A section on best practices and pitfalls to avoid would be nice to keep up the coding quality,

Second this, most of my c++ learning, if I ever get there, will be from this dll

 

[azum4roll]

I think it's better to learn to read the CvLua*.cpp files, and C++ code in general.