C# Code Testing

Puppeteer

Emperor
Joined
Oct 4, 2003
Messages
1,687
Location
Silverdale, WA, USA
There seem to be three popular-ish unit testing frameworks: xUnit, NUnit, and MSTest. I've only poked at xUnit and not touched the others so far. And really I was using xUnit as my own shortcut to compile and run libraries and not as proper unit tests. `dotnet test` is pretty easy to compile a DLL and run some commands with it.

One really weird thing about unit testing I just learned today is: Godot seems to compile with the .NET Core 3.1 or .NET v5 SDK and run with the Mono CLR. I have no idea if that impacts how we should run unit tests.

Subprojects using the non-Godot SDK reference in their csproj files should be unit testable if we move them off of netstandard targets (because netstandard is more of a virtual compatibility target than an actual platform). For any projects using the "Godot.NET.Sdk/3.3.0" Sdk attribute I have yet to figure out how to unit test those.

As for integration testing, I currently have no idea how to begin. All my Google results so far for Godot integration testing talk about the difference between unit and integration testing then explain unit testing in detail and seem to forget entirely about integration testing after that. This issue's comments may hold some value, but no silver bullet jumps out.

@WildWeazel , how does Unity handle integration testing? Do you just avoid testing graphical routines, or is there some sort of testing harness?

All that said, this is not a huge near-term goal. It's difficult to make unit tests when we have no design by contract and all the data structures are in rapid flux. But simply executing the load or save game code and making sure it doesn't crash might be a worthwhile unit test that won't blow up if the code changes. The SAV, BIQ, and media readers might could use some unit tests as their methods, inputs, and outputs shouldn't change much if at all other than to gain new code to test.
 
There is a potentially relevant modus operandi I learned from the book "Test-Driven Development for Embedded C", by James W. Grenning. One of the reasons embedded C has traditionally not had much automated testing (test-driven or otherwise) is that some things are hard or impossible to test through software. Does an LED turn on when the code is run? At a fundamental level, you have to look at the LED to see if it turns on or not. Same thing with whether a motor turns on. This code, not coincidentally, also tends to be the code that is most directly controlled by whichever microcontroller you are using to run the hardware.

So the technique is to minimize the amount of code at that low level, and use libraries that can be tested in an automated fashion - often on a platform other than the embedded microcontroller - for as much of your code as you can. This code will then call an interface, where the implementation can be either dummy methods (for testing), the actual microcontroller, or potentially another microcontroller in the future, if you need to add hardware support for another one. This gives dual benefits of increasing the quality of your code via testing, and making it relatively inexpensive and quick to port to another platform. The platform-specific code is low-complexity, and can be manually tested once for each platform.

There's a bit more to it than that (he wouldn't have been able to write a whole book otherwise), but that's the gist of it.

This has been part of my motivation for keeping the C7 Engine and C7 Data Structure separate. Our structure isn't the same as Grenning suggests setting up in embedded C - Godot is not, in its current version, available as a library, nor can we set it up that way. But still, by keeping as much as feasible outside of Godot, more can be tested. The sav and load code should, in theory, be unit and integration testable completely outside of Godot. Not the "New Game" button directly, but the functions that it invokes.

Now whether that's currently the case, I'm not sure. But if I had just finish reading Grenning's book, and set out to architect the project, that's how I'd set it up (as it is, it's been a couple years since I read it).

We do have a relatively high amount of front-end logic due to the way the game plays, and I'm not sure how much we can minimize that. OTOH, I suspect the front end is an area where we've put a relatively high amount of effort early on, compared to what we will have in the end. I'd also be curious to hear from WildWeazel about how this is approached in Unity. Hmm, I wonder if one of the links he has posted covers that?

-----------------

Looking through that issue's comments, some thoughts on potentially promising items:

- https://github.com/AlexDarigan/WAT <-- This one was mentioned by the creator, but the link had rotted. A few links later, I found the correct link. At a superficial level, it looks like it could be pretty promising. Seems to be actively maintained
- https://github.com/van800/SkyOfSteel/commit/6515c3fdd508083950d57300f1f6bf25e08a0025 - The SkyOfSteel example someone mentioned, using NUnit. Haven't studied all the details, but I see they exclude the tests in their main csproj
- van800 got it working within Rider, although he says the components are open source. He works for JetBrains, which probably helped in figuring out how to get it to work.
- https://github.com/fledware/GodotXUnit <- Godot XUnit. The GIF in the thread Puppeteer linked showed UI testing. Has a commit within the month, which mentions support for it in JetBrains Rider.

There's a few more after that as well. I think there's a good chance one or more of them will work for us. If I put my tech lead hat on, I'd create a spike card for exploring unit testing, and link that thread and some of the examples as good potential solution routes to explore.
 
I've added a proper unit test in my latest changes; see the test at https://github.com/C7-Game/Prototyp...6c2d59eed48283bfda4446d8859c17b3541f5a18f24ff

This is just at the engine level, not the Godot level, so it doesn't answer that question. But the engine is getting more and more code too, and the presence of any tests makes it easier to add more tests.

In this case, I used the xUnit framework, and added a parallel test project via Rider. It only let me choose .NET 5.0 or .netstandard someVersion at creation, so I had to change it to .NET 4.7.2 after creation, but that wasn't a big problem. I also added the dependency on the Engine project via the .csproj file in VSCode.

It runs just fine via the sidebar in Rider, and via "dotnet test" at the command line. Debugging it works in Rider, too.

A couple decisions it raises:

- Do we want to go with the traditional C# style "parallel testing project" paradigm, or include the tests in the corresponding projects? Personally, I believe tests should be integrated into the main project; this keeps the code close to what it's testing, which also makes it easier to visually check if there's a test for something, but admittedly I'm also accustomed to that because Java always puts the tests in the same project. Looking at StackOverflow, it looks like it's pretty easy to keep the tests in the same project in C# nowadays, just some changes to the .csproj file.
- Testing non-public methods... the age-old problem. One of the benefits of the Java style of including tests in the main project is that you can then test package-level private methods in your tests, without making them public. But a thousands ways have been attempted to solve this problem. Creating sub-classes that house your "private" methods via public interfaces, so the "main" classes don't make them public, but they are still testable. Hacking around with reflection to be able to test private methods even though they're private. 998 other methods. Personally, I just want to test helper methods without having to rig up a whole bunch of stuff to test all the things that surround the private helper methods. In a language without public/private/etc. access levels, you'd just write those tests. Any good ways around this in C#?

So far xUnit seems okay, though. And as the balance of our code has shifted somewhat away from the front end over the past month, we're no longer in a place where 80% of what we want to test is in Godot.
 
Top Bottom