Fun with Imports

[LilBudyWizer]

I could have sworn I made a post yesterday with an overview of where JavaScript files for the game are located and how they are organized. It seems to have been deleted without notification. I'm not going to go into a lot of effort on this one until I see if it stays. Anyway, I extracted import statements from the vanilla source. I then played around with them and put in a few different views. The zip below has the various JSON files I created which includes:

1. jsfiles.json
2. jsfilesbyscript.json
3. duplicates.json
4. imports.json
5. noimports.json
6. importedby.json
7. noimportedby.json
8. notfound.json
9. mostimported.json

jsfiles is all the information collected in one place keyed by the full path from a Civ perspective, i.e. relative to base\modules. jsfilesbyscript is the same information keyed by the script name, i.e. no path. It turns out that's not unique. Thus there is a duplicates file listing the script name and full Civ path to the scripts by that name. imports is rather mundane. It's what files each script imports by full Civ path. That's only the files with imports and noimports are the ones with no imports. It's mundane because you could easily just look at the imports at the top of the script, but it is how I get the next part. importedby is more interesting because that's which scripts imported a given script. That you can't easily find by browsing. noimportedby is what's not imported by anything. That leads to notfound. importby and noimportedby should add up to the number of scripts but instead of 686 it added up to 700. Importing files that don't exist would cause that thus notfound. When I looked at that I found some scripts used relative rather than absolute paths. Some left out the initial / on the import. So I corrected that and notfound is empty. I then threw in mostimported listing the scripts that are imported by the most scripts to the least leaving out those not imported by anything. That reduces it to 243 files that are imported by at least one other script. Less than 100 are imported by more than 3 scripts.

This is a work in progress. I'll be adding more files but it's about the end of the road for this approach. There's just a few more views I think would provide insight. Beyond that I'll have to parse the JavaScript files. That's how I got the imports but they're much simpler to locate and parse than an entire script. One goal is to try to isolate the globals not declared in any JavaScript and thus injected by the runtime. String, Array, Map would be examples, but that's simply JavaScript. Every implementation has those. It seems easy to eliminate those, but there's others that are specific to the game such as UI and Configuration. Particularly finding what they expecting as parameters to their function from what the vanilla scripts used as parameters.

 

[LilBudyWizer]

One more iteration. This one adds:

1. byDir
2. byPath
3. indImports
4. circImports
5. indImportedBy
6. circImportedBy
7. mostIndImported

byDir is a tree view mirroring the relevant directories. It what I used to calculate statistics for the directories. It's not real handy for browsing in JSON. So I then flattened it into byPaths for easier browsing. indImports is the import tree. That made for the potential of circular imports. There was only one:

/base-standard/ui/victory-progress/model-victory-progress.js
    /base-standard/ui/victory-progress/screen-victory-achieved.js
        /base-standard/ui/victory-progress/screen-victory-progress.js
            /base-standard/ui/victory-progress/model-victory-progress.js

So that show what all a script is dependent upon. circImports shows the circular imports. Flipping those is indImportedBy and circlImportedBy. Those show what all is dependent upon a script. mostIndImported show which scripts were imported directly or indirectly by the largest number of scripts. That is by unique scripts. There's 122K lines in indImports and 26K in indImportedBy. There's only 686 JaveScript files with total of 2,166 import statements across them an 186K lines of code. There's nearly as many lines in indImports as there lines of code. Say mod wants to add items to list managed by manager. It needs to import items and manager. manager needs to know what it's managing so it imports items as well. So mod imports manager which imports items and also imports items itself. mod importing manager doesn't give it access to what manager imported. When it comes to indirect imports the same file can appear dozens of times. Each imported by a different file. Assuming there's no needless imports it shows what a script is dependent upon even if it can't reference it. Script a might have a function funca that calls funcb in b which calls funcc in c. a has to import b to call funcb but it doesn't need to import c to indirectly call funcc through funcb.

I could , perhaps, clean that up a bit. I don't plan to though. It's on to the next phase. I just pull the name of the file imported from the import statements. An import statement can be in any of the following forms:

import defaultExport from "module-name";
import * as name from "module-name";
import { export1 } from "module-name";
import { export1 as alias1 } from "module-name";
import { default as alias } from "module-name";
import { export1, export2 } from "module-name";
import { export1, export2 as alias2, /* … */ } from "module-name";
import { "string name" as alias } from "module-name";
import defaultExport, { export1, /* … */ } from "module-name";
import defaultExport, * as name from "module-name";
import "module-name";

module-name is in all of them and that's all I currently extract from it. Next is getting the rest of the information of the statements. Specifically what did it import in terms of what it's call in the imported file and what it's called in the importing file. Once I have that then I can check which names they actually used. If they used none of the names then it's a needless import. That requires extracting the exports as well. That is a much more involved process. What I have now is about the limit of what I can do with just a list of JavaScript files and which files they import.

There's some pretty big files, 126K lines. It's JSON though. If you load it into something like Notepad++ that supports JSON files then you can fold it. They are, at most, 686 root nodes, no more than the number of scripts. indImports only has 503 root nodes despite having 126k lines in it. Some are definitely not good for browsing, like byDir, but you can quickly get to specific information. This is entertainment for me, but hopefully it proves useful to someone other than just me. It is a work in progress. The ultimate goal is to programmatically gather data on in game objects, scripts, tables and such into an accessible reference, like a wiki. I'm not going to comment on 686 scripts. I'm too lazy for that and that's why I program. Hopeful I can get it in a format where other people can comment on what they know.

 

[LilBudyWizer]

Updated for V1.2.0

 

[LilBudyWizer]

Diff fromV1.1.1 to V1.2.0