[dev] Architecture Thread

[Ozymandias]

@ All! -

@WildWeazel - Exactly an example of what I meant by OO-describable "Rules." (NB: I'll be playing around with Charm for walled towns.)

 

[Ozymandias]

@WildWeazel - My condolences, as well.

Otherwise - Is it safe to assume that my OO model suggestion might prove fruitful here as well, including scripts?

 

[WildWeazel]

Coming back around to event design... I've been browsing and learning and agonizing over the different options because coming from Java I'm not used to events being a built-in language feature much less having multiple default implementations, but I've decided that EventHandler<TEventArgs> seems to make the most sense in general:

public delegate void EventHandler<TEventArgs>(object? sender, TEventArgs e);

This is the type used by almost all of the Framework's built-in events. EventArgs is a simple marker interface that can be extended to include whatever args you need to include for the type of event. The advantage is that you can encapsulate all of the args with named fields, and unless you switch subclasses there's no need to change the event signature. For example a basic UnitEventArgs might start out like this:

public class UnitEventArgs : EventArgs
{
    public Unit theUnit;
    public UnitOrder order;
}

Thus the "events" listed in post #5 could correspond to different uses of EventArgs subclasses. Not necessarily each a different class, but as needed according to common arguments. At the very least the 5 top level categories would be subclasses: GameEventArgs, CivEventArgs, etc.

This pattern is more appropriate when informing external classes that something has happened* which is the basis for the event-driven component design. There are a few other cases where other delegate types come in:
When we want to do something simpler like a property change notification, Action<T> would be better since it just delivers one or more values. These could for example drive UI updates instead of polling for data.
And when there's a need to provide code to fill in some configurable logic - one implementation that provides a result - a custom delegate can be used. Lambdas are simple inline delegates.

That's a lot of technical jargon that I probably didn't explain well, so read up on delegates and events if you're not familiar. While the syntax is pretty simple, I admit it took me quite a while to get my head around the differences and when to use each. But again, EventHandler should be the general choice for indicating that something happened.

*or WILL happen. As a side note, another pattern that can be used especially with EventArgs is pre- and post-hooks where an event is sent out before a change is made with a flag that listeners can set to cancel, and then the sender checks the flag before applying the change and sending the post event. This would let mods intercept and override events before they happen, which could be useful in advanced scripting use cases, but is a lot of additional code.

References:
Events: https://docs.microsoft.com/en-us/dotnet/csharp/programming-guide/events/
EventHandler: https://docs.microsoft.com/en-us/dotnet/api/system.eventhandler-1?view=net-6.0
Action: https://docs.microsoft.com/en-us/dotnet/api/system.action-1?view=net-6.0
Delegates: https://docs.microsoft.com/en-us/dotnet/csharp/programming-guide/delegates/

 

[Quintillus]

That looks like a pretty good option, reading the linked docs. I was experimenting with delegates on one of my branches in the past week, and it sort of seemed okay, but also seemed like it was missing something... now I realize I was only working with half of how Microsoft designed the system to be used. I'll have to think about it more when I'm better-rested, but at a high level I think it makes sense. I'm thinking maybe it will make sense to try to implement a few of those after merging in my in-progress Settler AI work... building a city seems like a good candidate for events, as does the already-existing city-produced something. Adding before/after handles also makes sense, I've used a few Java frameworks that have those.

My only real concern is whether we add too much overhead/abstraction if we add event handlers for all 200+ places we can have events. But on the flip side, having some established patterns might help avoid having a mess of different converging patterns conflicting with each other as we add more features. It's definitely worth trying it in practice and seeing how it goes, there's a lot of potential upside.

 

[WildWeazel]

The thing that made events click for me was realizing that the event keyword essentially makes a property backed by a delegate type, with exposed += and -= operators instead of the normal get and set. So a delegate is kind of like a function pointer, and an event is like a delegate property. Both can use multicast (assigning and calling multiple functions) but in practice delegates are used for plugging in some logic like with the Strategy or Command patterns, while events are for triggering some external reaction as in the Observer pattern. Which types to use, ie custom vs Action for delegates or Action vs EventHandler for events, is mostly stylistic with some tradeoff of explicit vs generic code.

200 is a lot to think about, but we only need to implement them as we come to each feature, and it costs nothing at runtime to have an event with no listeners. I think it's better to have them in place than to start coupling components more than necessary.

 

[Quintillus]

I noticed that addition-like syntax. Definitely more native support than in Java, from a language design standpoint.

Strategy and Command patterns... the #1 predictor of whether I'll get a job offer from an interview is whether they care about whether I've memorized coding patterns. Observer is one of the few I have memorized. This is probably why I'm not planning to become an architect.

But looking them up, strategy pattern = choose an algorithm at runtime. That makes a lot of sense for us, having the ability to plug in different algorithms for things like parts of the AI (or a top-level one). Command pattern... kind of reminds me of some of the functional methods in Java 7+, but I'll have to spend more time to figure out how theory translates into practice, or perhaps practice into theory.

I think delegate in this context might be like callback in Java, with the note that there can be more than one, which isn't always the case with callbacks in Java. delegate wasn't a term I'd seen used often in my programming projects prior to this project, and while I'm figuring it out, often the quickest way for me to pick something up is by analogy to something I've worked with before (hands-on practice is another effective way, and examples-with-diagrams are a third).

Edit: Actually now it occurs to me, once we've decided that this (or something else) is what we're going with and it's working, one of the long-term viability challenges will be having documentation that is easy for newcomers to pick up, when they're coming from a variety of programming backgrounds. I suspect it will work better in many cases to use concrete examples we've implemented, with descriptions of how/why they work, than to use abstract terminology that the newcomers may or may not understand. But where possible, it could still help in some cases to say, "this is the XYZ pattern", "this is similar to how SomethingOrOther works in Ruby", etc.

 

[Flintlock]

What's the reason for having separate engine and game data modules? When I saw that I assumed that the idea was to separate the data itself from the code that modifies it. I've been going along with that, for example if I need the engine to modify some game object I put that code in an extension method in the engine module instead of putting it in a regular class method in the game data module. But I've noticed that @Quintillus hasn't been doing that, and as I recall he's the one who made the separation in the first place (am I remembering that right? was this discussed somewhere?). So now I'm unsure as to what the point is. If it's okay for the game data to modify itself then the extension methods should be made into regular methods, except some of them couldn't be because the game data module doesn't have access to EngineStorage. We could give it access, but at that point the modules might as well be merged together, unless I'm missing something.

 

[Quintillus]

Without digging through old posts, most likely I started separating them out so a clear delineation of the data (including what's used in the C7 save format), versus the engine logic. That in turn was probably highly influenced by how I did it for my editor (or more precisely, how I restructured it a year or so after starting work on it) - I have a library for handling Civ3 data that anyone can use in their own Java project without having to reinvent that wheel, and a project with the editor code. If someone wanted to write CivAssist III in Java, they could use that library to handle reading and writing all the Civ3 data, without also having to drag in all the unrelated editor code.

I'm not aware of anyone having taken advantage of that possibility, but it seemed like it might be nice to have here too, since maybe someday people will want to write utilities to go with C7. Possibly including ourselves.

It might be helpful to have a link to a specific example of what you are referring to with modifying game objects, but you probably mean adding methods directly to the class that can manipulate the data? If so, there are two contributing factors. One is that I didn't know C# had extension methods when the project started; I think it was Rust where I'd used them? Maybe Kotlin. But I didn't know C# had them.

The other is that the longer I've written my editor, the more I've concluded that it can be useful to have some utility methods in the "data" project. Given how the Civ data is stored, that often means a convenience method for setting data without having to resort to bitwise modifications - for example, a method to set a river to be present on a particular side of a tile, rather than having to bitwise manipulate both that tile and its neighbor to do so. I do try to think about, "is this something another utility might use, or is it editor specific?" but have gradually moved more towards "if it's practical and will help avoid errors, it can be included" over a more pure data-only library.

It's also worth noting that Java doesn't have extension functions, so those aren't an option in the editor. But regardless of the implementation, methods such as adding a river, or (to use one in C7) the isWater() method on a tile, tend to fall into what I consider the data telling you something about itself beyond just the raw fields that define it. It's definitely an art and there are probably cases where a method is on a data class that shouldn't be, or vice versa, but that's the thinking behind why I write things that way.

 

[Ozymandias]

 

[Quintillus]

@Flintlock I have looked through some of your animations/threading/etc. PR, and I think I see what you mean about this more, with the MapUnitExtensions. To me, those do seem higher level than what I'd put in the base MapUnit class, especially the larger ones like fight() that have a lot of mechanics and integration with higher-level components (animations). It's great as an extension method being able to write, "bool unitWonCombat = unit.fight(defender);", but the logic is highly tied to C7.

 

[Flintlock]

It wouldn't be possible to move the "fight" method into the game data module since it depends on "animate", which depends on EngineStorage. That's not a problem since we wouldn't want to move it there anyway, but a couple of months ago I stumbled over an annoying case where I wanted to put a method in the data module but couldn't because of EngineStorage. I forget the details but as I recall it needed to call tileAt but couldn't b/c it didn't have access to the actual game map. It struck me as weird and awkward that game data has complete access to tile objects but not the map instance that contains them, so it can work with tiles but only if it gets a reference to them from the outside.

These issues also tie in with how we're going to allow for mods. To re-use my previous example, we can allow for limited railroad movement, among other things, by enabling modders to override some method that determines the cost of moving a unit between tiles. If that happens to be an extension method in the engine, then that may be a problem since as far as I know there's no convenient way to override extension methods on C#. Alternatively, if it's a class method, the standard approach would be to make the method virtual and then allow modders to create a subclass that overrides it. The problem with that is it's very coarse-grained, e.g. if that method is in MapUnit then the entire unit class would have to be replaced to implement that one change and that would conflict with any other mods that modify the unit methods somehow.

I don't what the best approach is. These are just things I've been thinking about for the past few months. I read what you said about separating out the game data module so utilities can work with it without pulling in the entire engine as a dependency, and that makes sense, but I'm skeptical that it's worth the trouble. Do you have some specific utilities in mind? The only one I can think of is CivAssist, but I think we should try to make our mod framework robust enough that that could be implemented as a mod instead of an external program.

 

[Ozymandias]

... the #1 predictor of whether I'll get a job offer from an interview is whether they care about whether I've memorized coding patterns.

Much as I have a "love/hate" regard for them, Google's interviews (along with demonstrable coding prowess, of course) are geared towards evaluating someone for, essentially, how far, and well, outside the proverbial box an applicant can go. Once Upon A Time, I hired many, many coders. So my advice is to apply at Google.

 

[Puppeteer]

It wouldn't be possible to move the "fight" method into the game data module since it depends on "animate",

In my personal vision of how things should be, the AI, local player, and remote players all use the same interface to make things–such as fight–happen.

For a proper game UI, then–depending on settings–the fight has to wait for the animation, but perhaps this could be accomplished by a messaging bus or callback function, the latter optionally passed to the fight call. I suppose a simple event listener could be worked in there somewhere along with a timeout so the game does something if the animation never signals it's finished.

 

[Quintillus]

Much as I have a "love/hate" regard for them, Google's interviews (along with demonstrable coding prowess, of course) are geared towards evaluating someone for, essentially, how far, and well, outside the proverbial box an applicant can go. Once Upon A Time, I hired many, many coders. So my advice is to apply at Google.

Gosh, no. I did that once, got an interview a few years ago. Worst interview I've ever had, anywhere in my career, including the one where 10 minutes into the interview I realized that what they were describing for the role was not at all what I wanted to do.

Long story short, the recruiter (who'd already made a scheduling snafu) didn't mention anything about the format being one long programming exercise, using Google Docs (a terrible tool for writing code, though good for other purposes). So when that was the interview, and I got stuck, I thought it was like most interviews where if you don't know one thing, the interview generally moves on, and rarely is not knowing one thing a dealbreaker. The interviewer must have thought I knew it was one long problem, as they just kept redirecting me back when I suggested we try something else. Poor audio quality as well, so it was also hard to understand them.

Very frustrating, eventually I said let's quit wasting our time (more diplomatically). They did ask for feedback after the process, so I gave them feedback on how awful of an experience it was (which is rare for me in an interview, most of them are kind of fun). Having since learned a bit more about how they structure their interviews it also sounds very arduous if you do make it past the first round. They seem to assume everyone wants to work there and looks that up first; for me it was a bit of an after-thought compared to the places I was actually interested in, which were local, smaller, and with a great reputation locally. As in, the Google recruiter happened to reach out to me at the same time I was applying to places I was interested in locally, so I decided to give it a shot because why not? I got an offer from all three of the local places and learned a ton from working at the one I chose, so I figure it's Google's loss.

Now there's also the fact that I've gone from being indifferent about working at Google at that time, to being more anti-Google due to their privacy policies. Which is the same general reason I'd gone from "it'd be cool to work there" around 2010, to indifferent, in the first place. So while I'll admit I probably had a "worst 1% experience" with my interview with them and a second go-round would likely go better... they're not on my radar from a product standpoint.

If I had to choose a big tech company to work for, it would probably be Microsoft, preferably in the software tools division. I'd also consider an interview with IBM. But my preference is for small companies, where I'm not just going to be a code monkey, but can have an impact on the product.

Sorry for the tangent... you wouldn't have known but "Google interview" is one of the sure-fire tangent subjects for me. I suppose I never got the the "love" part of them, as I did with a couple of the local companies I applied for at the time. The one I did choose, actually... part of the reason was because I realized why the guy was asking me the questions he was asking me, how he was trying to understand my thought process and why I was reasoning the way I was. I realized that guy really got it, and if all their interviewers were as good as he was, I'd have a fantastic bunch of colleagues. Sure enough, it's probably the smartest group of people I've ever worked with.

 

[WildWeazel]

Strategy and Command patterns... the #1 predictor of whether I'll get a job offer from an interview is whether they care about whether I've memorized coding patterns. Observer is one of the few I have memorized. This is probably why I'm not planning to become an architect.

If you're interested, this is the best reference I've seen for design patters. I'll put it in the resource list if it's not there already. https://refactoring.guru/design-patterns/catalog
It's better to understand OO principles than to memorize the names and UML diagrams of design patters, but it's useful to recognize the literal patterns that emerge. The actual capital-p Patterns are just shorthand for those that people have identified as common. In my totally arbitrary and limited experience, the best ones to know have been Strategy, Template Method, Factory Method, Observer, Adapter, Singleton, Builder, and Decorator - some of those just because they turn up a lot in APIs and syntactic sugar. Speaking of which, delegate is indeed a lot like a (multicast) callback, which can be examples of either Observer or Strategy depending how they're used.

I agree about not using extension methods and splitting modules. That's usually for extending code that you don't control or can't break. If it feels like they should be separate, maybe the class design needs to be rethought and broken down further.

 

[Ozymandias]

Yikes! Literally around the corner from where I live, in Manhattan, Google bought the entirety of one of those entire, city block long, and entire, avenue wide old NYC municipal buildings. I know a couple of high-power coders: one applied and failed the "box" test; the other did not, and is a rising, international Google star. Both (they don't know each other) described the same type of interview process ... Which (having "been there / done that" in places like Lehman Bros. & DLJ pre-2008) I understand can both turn on a dime, and differ from trading floor to investment bank ...

But I digress. The most important bit is (of course) where you wound up where you are, and how (And whatever happened to the "proper," non-seasonal, toasting emoji?? )

 

[Quintillus]

It wouldn't be possible to move the "fight" method into the game data module since it depends on "animate", which depends on EngineStorage. That's not a problem since we wouldn't want to move it there anyway, but a couple of months ago I stumbled over an annoying case where I wanted to put a method in the data module but couldn't because of EngineStorage. I forget the details but as I recall it needed to call tileAt but couldn't b/c it didn't have access to the actual game map. It struck me as weird and awkward that game data has complete access to tile objects but not the map instance that contains them, so it can work with tiles but only if it gets a reference to them from the outside.

These issues also tie in with how we're going to allow for mods. To re-use my previous example, we can allow for limited railroad movement, among other things, by enabling modders to override some method that determines the cost of moving a unit between tiles. If that happens to be an extension method in the engine, then that may be a problem since as far as I know there's no convenient way to override extension methods on C#. Alternatively, if it's a class method, the standard approach would be to make the method virtual and then allow modders to create a subclass that overrides it. The problem with that is it's very coarse-grained, e.g. if that method is in MapUnit then the entire unit class would have to be replaced to implement that one change and that would conflict with any other mods that modify the unit methods somehow.

I don't what the best approach is. These are just things I've been thinking about for the past few months. I read what you said about separating out the game data module so utilities can work with it without pulling in the entire engine as a dependency, and that makes sense, but I'm skeptical that it's worth the trouble. Do you have some specific utilities in mind? The only one I can think of is CivAssist, but I think we should try to make our mod framework robust enough that that could be implemented as a mod instead of an external program.

CivAssist, scenario editors (which could probably be done as a module), other ones that were popular over the years include a map seed generator (e.g. for Moonsinger's milkruns) and Puppeteer's Civ3 Show-and-Tell. I'm not really sure how much of MapStat and CrpSuite aren't duplicated by later programs, as I haven't used those extensively. Although it does occur to me that simply by having out data be relatively easily-consumable JSON instead of binary like Civ's, it will be way easier for whoever wants to, to vacuum it up into whatever language they prefer to work with.

We might yet have some gaps in how the data can reach other parts of the data - I don't think it should be necessary for it to ever call EngineStorage. It probably isn't tileAt as that's on GameMap.cs, which is in data... although perhaps the case you saw was where some other part of the data wanted to call it, but couldn't reach GameMap, so it went via EngineStorage? Although as I think about it more, since C7GameData is a dependency of C7Engine, it shouldn't be possible for that to happen... and indeed a search for EngineStorage turns up no results within C7GameData.

Still though, I wouldn't be surprised if we had some awkward logic that's at a higher level than feels appropriate because it goes across a couple different parts of the C7GameData. I've run into that sort of thing in my editor, too.

Thinking about the railroad example... it's fundamentally going to have to be a method that takes two tiles, as the cost depends on the improvements (and possibly base terrain, and possible technology [bridges]) of the two tiles. You're right that if it's done as a virtual method, then if two mods both try to modify it, it's a "last one wins" situation. It's probably impossible to fully avoid that type of situation, which is probably why some Factorio mods (for example) are not compatible with other Factorio mods. But it also has me thinking about what's the appropriate level of modification? My gut feeling is that it would make more sense to make the components of that modification modifiable, than to make it possible to re-write the whole thing. Let the user set how far rails should allow the unit to move (e.g. it costs 1/10th of a movement point), rather than let them re-write the whole method (which will be relatively complicated). Maybe they still can modify a whole TransportCosts class if they want to... but that would be akin to C++ DLL modding in Civ4, whereas just modifying the rail cost would be akin to the (much simpler) XML modding in Civ4.

tl;dr, I think it's wiser to make things easy to change in mods than to allow maximum power in making changes in mods. Factorio has demonstrated this to a fair degree; they've gradually made more things modifiable in response to demand for more aspects to be modifiable, rather than trying to predict too much up-front. Although this probably isn't a surprising point of view considering I'm someone who wrote an editor to make it easier to mode a game that's already probably the best in its series in terms of ease of modding.

Good to know that extension methods aren't overridable. I hadn't been aware of that. I wonder if it would make more sense to favor service classes that are overridable in some cases, for that reason? Though again, I'd favor a judicious approach based on expected demand.

I expect we'll try a few things, and some will work better than others, maybe in some cases approach C works best and in others approach D works best. I don't really know what the best approach is either.

In my personal vision of how things should be, the AI, local player, and remote players all use the same interface to make things–such as fight–happen.

For a proper game UI, then–depending on settings–the fight has to wait for the animation, but perhaps this could be accomplished by a messaging bus or callback function, the latter optionally passed to the fight call. I suppose a simple event listener could be worked in there somewhere along with a timeout so the game does something if the animation never signals it's finished.

I agree (in theory, not necessarily in practice) that all of those types of players should use the same interface. The asterisk on "in practice" since I'm not sure how feasible that will be.

I should probably read more of Flintlock's PR to more accurately say how closely it matches what you describe so far - I've only really read one side of it - but I'm torn in part because from a readability standpoint, what he has looks good now. Whether it will survive future additions intact - not just things like possible eventual remote players, but closer things like tile visibility - I don't know.