C15's CMM Troubleshooting Guides

[Chrisy15]

Archive of Troubleshooting articles written for the Civilization Modding Monthly since although publishing them there allows them to be presented to a wider audience, it's less useful for people searching for answers and for directing questioners. Some of these articles can , or may in some cases exclusively, be applied to Civ VI instead but for the sake of cohesion and homeyness I'll just repost them all here.

Index:
Issue 3: The Purposes of Logs
Issue 4: Locating SQL Errors
Issue 5: The Issues with Templates
Issue 6: The Role of Firetuner

Related Resources:
How To Enable Logging
SqliteSpy101

 

[Chrisy15]

From CMM Issue 3: The Purposes of Logs

Spoiler :

*This guide is designed to be applied to both V and VI.*
Picture the scene. You're happily going along, playing your game, but you suddenly find something doesn't work. What do? The instinctual reaction is to make a post on some workshop page using as few words as possible, but if you're literate enough to be reading this article then you should also be able to read the common line in a workshop description asking you to "post logs". But what are these mysterious lumber products? And how do they help Modder determine which mod is at fault? Today we will be shining a light onto the dark magics of logging, and giving you an idea of how they are crucial to debugging.

If you've ever come to me with an error I'll have no doubt pasted a link to WHoward's logging guide. This explains which settings need to be enabled to turn logging on, but goes into less detail about which logs are used for what and doesn't include some more recent advice. WH talks about three logs: the Database log, the XML log, and the Lua log. Bringing up the XML log at all feels like a false start to me, as I honestly can't remember the last time I found something useful in it; instead, the Stopwatch (V) or Modding (VI) log is a much more important third log that will be discussed here. The XML log's only role seems to be reporting missing references regarding audio defines, which is a very minor issue that won't cause a significant issue with a mod; it doesn't show errors caused by XML files, certainly, as these are all tracked by the Database log as they are found after the XML has been translated into SQL.

The Database log lists errors that occur while the database is being configured: for V this will be after clicking Next on the Mods screen; while VI's setup means that Database errors prefixed by [Configuration] will be occuring before the Main Menu, while [Gameplay] will be occuring on the black "Please Wait" screen. Database log errors will highlight things that the game is expecting to exist that don't, or syntax errors that might explain why these things are missing. If a component is missing from your game, or Yields aren't showing where they should, the Database log will be the most likely to explain the cause; exceptions will be discussed later. Database errors can be very easy to locate, but in some cases they can be more ambiguous and the Stopwatch/Modding log will be necessary.

The Stopwatch/Modding log is a partner to the Database log, as its primary purpose isn't to report errors itself but rather to explain which files are causing errors through timestamp comparison, since the Stopwatch/Modding log tracks when each file is interpreted by the game. Therefore even if the modder can't locate the specific line of an SQL error, the issue can be narrowed down significantly to a handful of files. The finer details of this debugging process are not relevant to the mod user, but will be discussed in the future for the benefit of modders who do not yet know the tricks behind locating SQL errors using these two logs.

The Lua log is distinct from the previous two logs, for it is only concerned with runtime errors. It is populated from the Dawn of Man screen onwards in V, while VI has it tracking errors throughout the main menu too. During the Dawn of Man screen Lua files are loaded, meaning the first syntax errors will be found during this period. These errors will be issues with the code itself, such as an incorrect number of ends to if or for statements. These errors will also cause the Lua file to be discarded, meaning they must be resolved for the content to be usable in-game. Errors found after this time will usually originate from within functions, and will often be caused by variables being given data that isn't valid in a given situation - indexing a table with a nil value is a common one. These errors will end their function, but won't stop the function from being called again and so can lead to the log becoming cluttered if the culprit is hit frequently. Lua errors provide the most specifics of any of the logs, and allow their origin file and line to be identified immediately. A Lua error can explain why an effect described by a UB isn't working if said effect is implemented through a Lua function placing a dummy Building in the City, or some similar scenario; the more exotic an effect the more likely it is that Lua scripts are involved, but it's always safest to just include all three logs when reporting an error.

Hopefully this explanation of the purposes of the three primary logs makes it easier to understand their importance: they provide specific details that explain exactly why issues are occuring, and without them the modder is left guessing in the dark about the cause of an issue. Not every issue will require a copy of the database itself or an account of the mod list to be resolved; but no issue will be made worse by a log being provided.

 

[Chrisy15]

From CMM Issue 4: Locating SQL Errors

Spoiler :

This guide is designed to be applicable to both Civ V and Civ VI.

All too often I hear people decry SQL as being an inferior language to code in on the basis that the errors reported in the Database log aren't descriptive enough. While I won't deny that XML errors allow for interpretation at a glance, this simply reduces the time it takes to track down the issue; and compared to SQL's numerous benefits over XML, this hinderance pales in comparison.

Last month I explained how the Database log can be used in conjunction with the Stopwatch(V)/Modding(VI) log to limit down the culprit files behind an error. The next task therefore will be to determine where in the specified files the error lies. The fundamental principle we're following here is that the game interprets the SQL in a file up until it reaches a statement containing an error; it will then undo all actions performed on the database by this statement in accordance to the ACID principles and cease interpreting the file. Therefore, in order to locate an error in an SQL file we should look for the first piece of code that doesn't run.

There are two ways to see what code has been interpreted by the game and what code hasn't. The first would be to look to see what's actually in game - the Civilopedia would be a good place to start with this approach. However, a far superior method would be to read the database itself: this would not only save waiting for the game to go through its processes but also allows you to see things that aren't as visible to the Player, such as inserts to mod support tables. Each game has a selection of databases for you to look in, depending on the error: for Civ V, text is stored in Localization-Merged.db while everything else is in Civ5DebugDatabase.db; for VI you have DebugConfiguration for errors prefixed by [Configuration], DebugGameplay for errors marked as [Gameplay], and DebugLocalization for those labeled [Localization]. For both cases these databases can be found in your Cache folder, adjacent to your Logs and MODS folders.

To read these database files a variety of programs can be used: Kael suggests using a Firefox addon, although if memory serves this plugin may have been discontinued; WHoward (and I) recommend SQLSpy (the installation of which is explained here); I also once found a Chrome extension that is presumably fully capable for these purposes, but the main message is that the tool doesn't matter so long as it lets you view the database. Once you have opened the database of interest in your viewer of choice, the task is simply to read your way through the file in question and verify that each statement has been completed; once you find the point where the statements stop being reflected in the database for no expected reason, your error has been found. The error message should then be referred to, and with our search area reduced to a small number of lines around the point where the file stops being interpreted the cause should be deducable with ease. If you're struggling to locate the error still, keep in mind that "NEAR" is a relative term; SQL interpretation ignores white space, and so NEAR will be referring to the last character or recognisable keyword in most cases even if this and the error are separated by a good number of blank lines. If your error continues to evade, your database viewer should allow you to execute SQL into the database; using this to input statement blocks individually can help you determine which ones work in isolation and which one continues to fail.

Hopefully this explanation can help inform modders on how they can debug their SQL code, and encourage XML coders to use the far more versatile option without fear of becoming lost. The beauty of code is that every syntax error can be corrected, and every logic error can be resolved; hopefully now locating these issues should be a stepping stone rather than the challenge itself.

 

[Chrisy15]

From CMM Issue 5: The Issues with Templates

Spoiler :

Title: The Issues with Templates

Introduction
Writing basic database inserts is by far the most boring part of modding. You can have the time of your life writing scripts and convoluted SQL magic, but at the end of it all you still just have to insert into the same five or so different Civilization_ tables as you do every time. Since this kind of work is basically the same across all civilization mods you'd hardly be blamed for reusing your own code as a framework for a new one; this is fine, because you know what all your code does and you know that you've inserted into all the places you need to. The main issue I have with these kinds of templates are when new modders, usually the people who just want to quickly make something with minimal effort, use them without knowing what they're doing or how to fix any issues that come up.

Your First Point!
If you download someone's template and follow their instructions for Find and Replacing in your unique keys, not only will you not know what any of this code does but you likely won't even remember what you're doing for the future. This becomes an issue when you ask people for help with things and they start throwing terms like "table", and "column", at you when all you've seen are things inside arrows. Therefore it is important that you understand what your code does, although unfortunately few explanations of this come to mind; no doubt WHoward has a post explaining it somewhere, but if I dare comment on his work I'd admit that his SQL explanations can appear intimidating when first approached. As such I've tended to use this point as an excuse to get people to read Kael's Guide, as he does explain how your XML translates to the database's columns and rows; I'll also admit that this explanation can't be brilliantly effective given that I took an awful long time to understand the process, and so perhaps a future article should cover the topic. Regardless the importance of understanding the results of your code -and my dislike of a template being a new modder's first port of call - should be clear; instead reading up on the topic, however you can, and actually understanding the code that you're writing are much more effective ways of going about learning to mod.

Your Second Point!
Besides the issues templates present for those of us who spend our time trying to help and teach, templates also have a negative impact on the mod itself. The typical template that comes to mind will present you with all the possible columns for a table with no value attributed; at best this will needlessly bloat your mod's file size, but a more significant issue would occur if these empty column definitions lead to NULL values being put in place of the column's DEFAULT value; I can't say I remember enough about XML for this to be any more than scaremongering, but I can say with confidence that there's no benefit to leaving these columns in place. The additional effect of presenting these columns is that it encourages the user to limit their design ideas to those presented, rather than entertain the ideas Lua can (often easily) facilitate. This leads back towards our previous discussion on complexity, and while it could be argued that someone who can only be bothered to use a template will have no interest in writing Lua I'd ask why they've got any interest in making a mod at all, if what they want to make amounts to little more than a reskin of an existing civilization that could be more easily accomplished as such. While the primacy of modding unfortunately lies in its visual presentation, I'll continue to encourage people to place at least equal value in the design beneath.

Conclusion
Hopefully it goes without saying that this has been a discussion regarding code templates; I'm in no place to hold any kind of opinion regarding icon templates or leaderscene references, but can't help but feel that in an ideal world where artistic skill was as empirical as programming competency the above arguments would apply. With that aside I hopefully have at least convinced some people interested in creating mods to put the required work in, rather than taking shortcuts that will simply make debugging harder and limit your scope; as ever my advice continues to be to look at existing code, for mods or for Firaxis' civs, and not just copy it but understand what it does and how it does it so that you'll remember for your next mod or at least understand better any technical advice given to you.

 

[Chrisy15]

From CMM Issue 6: The Role of Firetuner

Spoiler :

This guide is relevant to both Civilizations V and VI.

IGE is commonly regarded as a crucial part of the V modder's inventory, allowing for the creation of test scenarios without having to play and specifically shape entire games to reach sample cases. While it is of course smart to play natural test games to check for unforseen logic issues or to simply see whether the mod is fun, this kind of investment is less endurable when the game ends up being spoiled by a syntax error somewhere down the line. It is interesting to note however that IGE is not really as mandatory as it feels, for although it is much more user-friendly than Firaxis' FireTuner the latter features a perfectly suitable compliment of features to make a VI version of IGE not really necessary - once you've gone through the correct setup, of course.

The core role of Tuner is to act as the game's console: in the same way as your console commands for Skyrim or Minecraft, code can be executed in Tuner to influence the currently running game. Tuner's associated panels then act as shortcuts for performing various actions, saving the user from needing to learn method names or write functions to perform tests. Tuner can be opened through the respective SDK launchers for each game - either game in fact, as the FireTuner2.exe will simply connect to whichever game is open, although I recall the aformented panels not all loading if the wrong "version" of Tuner is opened - but to the new user's confusion won't immediately work. In Civ V Tuner must be enabled in your Config.ini file, where you enable your logs etc; VI presumably works the same way but allows you to enable the setting through the in-game Options menu, with the warning that Achievements will be disabled if the option is enabled (this seems to be wrong though since I have achievements in VI, but I won't tell if you won't). Now that Tuner is enabled, it will connect to the open game and act as a self-refreshing version of the Lua log. The input box along the bottom of the Lua Console panel will allow you to execute Lua in the Context defined in the top-left dropdown; this defaults to "Main State", which confusingly doesn't have any of Civ's global variables or methods availiable, so if you're getting nil errors make sure you've changed to literally any other Lua context.

The Lua Console will allow you to execute any Lua that you could in a mod, and so already we theoretically have all the capabilities of an IGE. This is hardly feasable for anyone who hasn't memorised the Lua API however, and it'd be time consuming and clunky for anyone who has. Therefore, both games have a set of "panels" that can be added onto Tuner to provide shorthand functionality. These are located in the games' own directories, rather than their SDKs', in folders called Debug. These folders will both contain a collection of .ltp files that Windows doesn't know what to do with; by Open With-ing these files with the respective FireTuner2 these XML-like files will be opened as interface panels upon entering a game session: they can be examined in a text editor to determine how their components are defined, if you wish to try and reuse their logic in your own Lua code, and they can even be modified in a text editor or in Tuner itself to add your own functionalities - this is not a good place to delve into this subject, however. These panels will allow you to add Buildings or Techs to Cities or Players, change accumulations of various yields, and set the game to autoplay for set numbers of turns; while limited in some ways compared to IGE, the compliment of tools is generally comparable and certainly sufficient.

I hope this month's guide sheds light onto the SDK's most useful utility, and makes your modding experience easier. However, if you're looking for something more basic/applicable to your MacOS-running machine I'd like to take the opportunity to highlight LeeS' new Civilization 6 Modding Guide, which will certainly prove to be a bible to many who are new in the field; it's long, but will no doubt be highly influential to the society that grows from it.

 

[norwalkvirus]

Where is the database in Civ6 that I can view with SQLiteSpy? It believes all the .sql files I can find in ..\Steam\steamapps\common\Sid Meier's Civilization VI\Base\Assets\ (and other areas) might be encrypted, but I can open them with Notepad++ just fine, so that doesn't grok. What am I looking for?

 

[Chrisy15]

In the Cache folder in Documents, the (relative) same place it is in WH's SQLSpy guide.

 

[LeeS]

The SQL database files that are viewable using SQLiteSpy or some other SQL database viewer program are located at:

Civilization 5:

C:\Users\YourUserName\Documents\My Games\Sid Meier's Civilization 5\cache

Filename = Civ5DebugDatabase.db

===========================================================================================

Civilization 6:

C:\Users\YourUserName\Documents\My Games\Sid Meier's Civilization VI\Cache

Filename for the "FrontEnd" Database = DebugConfiguration.sqlite
Filename for the "InGame" Database = DebugGameplay.sqlite