Your experiences/thoughts on writing automated tests for plugins?

DoubleX

Just a nameless weakling
Veteran
Joined
Jan 2, 2014
Messages
1,748
Reaction score
911
First Language
Chinese
Primarily Uses
N/A
Lately I've been trying to write some automated tests to reduce the chance of my plugins to have nontrivial bugs unnoticed. This helped me catch quite some issues that I'd have missed for a long time otherwise, especially when developing a advanced complex battle system plugin. Of course, I still have to debug and dry-run my code to reason about my codebase, but automated tests can help me as well.

For instance, if my plugin relies on some invariants and I accidentally break some of them when adding some new features, those automated tests(which catch regressions in this case) might be able to inform me about that immediately, so I can fix the issues more effectively and efficiently.
Similarly, if my plugin has some compatibility issues with some other plugins and there are automated tests that can inform what assumptions from which plugins are violated(even though some of them should be violated in some cases), I can find the root causes more quickly.
Another useful scenario is to check inputted parameters/added notetags from plugin users to see if the contents are valid, even though it's more like assertions than automated tests. With a well-presented error log to plugin users(assuming that they do enable those tests), they might even be able to fix those invalid parameter/notetag values themselves without asking for the others for help.

Unfortunately, it seems to me that it's quite challenging to write testable high quality codes when developing plugins, especially with high test coverage(be it from unit tests, functional tests, integration tests, etc). For instance, I find it uneasy when trying to write many pure functions with well-defined inputs and outputs to increase testability while still conforming to the default RMMV codebase architecture and style, use dependency injections to facilitate swapping between real and test objects without tearing the plugin business logic apart, or writing automated tests that clearly specifies the preconditions and postconditions of functions solely for well-defined side effects without causing the tests to be tautological. Maybe that's just because I'm still inexperienced in writing testable codes in general :)

While some manual testings are inevitable(apart from the obvious parts like compatibility issues and how the plugin user would feel/think when using the plugins, which can only be tested manually), I still want to at least try to have a higher test coverage, like 80%, which is high enough for me in most cases. Maybe it's because I'm still a very bad junior programmer and don't know how to write tests, but right now the best I can get is 20%, which might be enough in some cases but still unsatisfactory for me(I've made a 10K LoC vanilla ES6 pet project via TDD and reached 80% test coverage, even though that's still nothing to brag for).

Strictly speaking, I guess I can get 80% test coverage with the heavy use of heavy mocking, like mocking a whole battler and even the whole battle scene, but I just don't feel that the return of investment of this approach's worth it in most cases. After all, setting up test fixtures and cleaning up mocks with quite some shared mutable states and implicit stateful dependencies can mean tons of tedious work that can be done all wrong way too easily(Not to mention the dedication needed to maintain such tests). Also, the time needed to run such tests might be rather long, but maybe that's just because I don't know how to mock properly yet.

Therefore, I'd like to know if anyone has any success in writing automated tests when developing plugins, and how you managed to get there(be it test after, test first, test driven development, behavior driven development, etc). Alternatively, if you managed to usually develop plugins effectively and efficiently, even including advanced complex ones like complicated and convoluted battle systems with lots of new features, with almost no nontrivial bugs, but nearly without the use of automated tests, I'd like to know how(like writing codes that are so easy, simple and small that no testings are needed? But how so for advanced complex plugins?) :D
 

LTN Games

Indie Studio
Veteran
Joined
Jun 25, 2015
Messages
701
Reaction score
625
First Language
English
Primarily Uses
RMMV
MV just isn't setup in a way the allows us to test things the right way. I have thought of digging into making tests and figuring out some good ways to go about it but it just seems like so much work for such little gain. I know Testing your code is really important and I personally have battled with it numerous times and I still don't do tests like I should be.

Anyway, back to making tests for plugins, I do believe we need a mock environment, or at least a minor tool that connects our plugin in development with a testing framework like Jest, Mocha or AVA. We can easily make our plugins full of pure functions which we can import into our tests but that is as far as we can get without somehow utilizing the MV codebase and data files and performing tests. If you ever take on that project you can guarantee I will use it religiously lol Heck, if you start the project and open a repo I'll gladly help where I can.

Now, what I currently do for testing is the daunting process of simply testing the game itself with the plugin in it. Its easy to miss a lot of bugs but it does allow for a fairly stable plugin. When a bug does occur it gets fixed quite fast with VSCode's built in debugger connected directly to MV's test play nwjs executable, which allows me to set breakpoints on my source code files(numerous files split by class and utilities, usually) and not have to dig through 1 giant plugin file.

I know that is not exactly the answer you're looking for but you can be sure I'll be keeping an eye on this thread because testing is really important and if someone has an easy solution I'll be using it ASAP.
 

DoubleX

Just a nameless weakling
Veteran
Joined
Jan 2, 2014
Messages
1,748
Reaction score
911
First Language
Chinese
Primarily Uses
N/A
MV just isn't setup in a way the allows us to test things the right way. I have thought of digging into making tests and figuring out some good ways to go about it but it just seems like so much work for such little gain. I know Testing your code is really important and I personally have battled with it numerous times and I still don't do tests like I should be.

Anyway, back to making tests for plugins, I do believe we need a mock environment, or at least a minor tool that connects our plugin in development with a testing framework like Jest, Mocha or AVA. We can easily make our plugins full of pure functions which we can import into our tests but that is as far as we can get without somehow utilizing the MV codebase and data files and performing tests. If you ever take on that project you can guarantee I will use it religiously lol Heck, if you start the project and open a repo I'll gladly help where I can.

Now, what I currently do for testing is the daunting process of simply testing the game itself with the plugin in it. Its easy to miss a lot of bugs but it does allow for a fairly stable plugin. When a bug does occur it gets fixed quite fast with VSCode's built in debugger connected directly to MV's test play nwjs executable, which allows me to set breakpoints on my source code files(numerous files split by class and utilities, usually) and not have to dig through 1 giant plugin file.

I know that is not exactly the answer you're looking for but you can be sure I'll be keeping an eye on this thread because testing is really important and if someone has an easy solution I'll be using it ASAP.
It seems to me that I've a very similar feel whenever I try to write test for my advanced complex plugin.
While those tests have helped me quite some time(there's an instance that a properly failed test have saved me probably hours of debugging), it's very painful for me to write and maintain them as it's just so tedious for me. On the other hand, even though the technical difficulties on writing the plugin itself is generally much higher, I find it much more pleasurable even when I've to think for hours sometimes.

To me, the pain point is that, it's very hard for me to write many proper automated test that tests really significant things; On the other hand, there are many significant things that I wish I'd be good enough to write proper automated tests. For instance, I'm still not good enough to automate the whole battle(especially when it's something like an ATB system), at least not within a reasonable amount of time and effort(so even if I might be able to do it I still feel that it's not worth it).

Also, I guess that I tend to agree on your take about pure functions here.
If I try to isolate as much of my code from the default RMMV codebase(something like a functional core with an imperative shell), it's likely for me to have either of the following scenarios, all of which are undesirable for me:
1. The core business logic would be scattered among many pure functions, meaning that it can be much harder to grasp the full picture and reason about the codebase as a whole effectively and efficiently(that especially applies to solving compatibility issues when there are just so many possible functions to deal with). Also, I'd have a hard time to tell what core business logic is compromised whenever a unit test fails, thus lowering the values I can extract from even otherwise proper unit tests.
2. Many of the pure functions would take tons of arguments, or some argument objects(or data transfer objects, value objects, etc) that are so large that I'd likely have to create classes for some of them, potentially leading to an excessive number of classes, or even risking to have some drawbacks from misapplying anemic domain models.
3. As writing many plugins involving mainly or even solely changing entity states and those changes themselves are usually very easy, simple and small on their own, extracting such transactions into pure functions can lead to those functions being too trivial(writing tests for them would most likely be tautological).
In short, I feel that the functional core would be too small while the imperative shell would be too large :)

As for mocking frameworks, I've only very little experience with it(I've only tried JUnit for several weeks when developing commercial Android apps then abandoned the whole thing due to not having enough time to write automated test to begin with) so I don't think I can talk much about that. So far my only take on mocking framework is that, I'll try my best to organize my codebase so that mocking frameworks can be reserved as the last resort.
In my experience, a well-designed composition root, raw user input recorder that are linked to a game state setting command recorder, and the existence of a renderer that actually draws something based on the received rendering commands can help a lot(game state changes results in new rendering commands instead of directly changing the view).

In short, I try to make the game flow this way in my vanilla ES6 pet projects:
1. Current Rendered UI, Raw User Inputs -> Raw User Input Records
2. Current Game States, Raw User Input Records -> Game State Setting Commands
3. Game State Setting Commands -> New Game States, Rendering Commands
4. Rendering Commands -> New Rendered UI
So I can:
1. Mock raw user inputs with some raw user input records as test cases(this can be automated)
2. Setups some test fixtures(by initializing the rendered ui with the game states and running some predefined game state setting commands) to have the desired current rendered ui and current game states as test cases, then restores the old rendered ui and old game states to those before applying the test fixtures if necessary(this can be automated)
3. Checks if the given current game states combined with the given raw user input records produce the expected game state setting commands(this can be automated)
4. Checks if the given game state setting commands produce the expected new game states and rendering commands(this can be automated)
5. Finally, I've to manually test if the given rendering commands produce the expected new rendered UI(I won't automate this even if possible)

This approach let me reach 80% test coverage in one of my such projects, even though 80% may still be not high enough.
Of course some relatively heavy mocking are definitely involved in those processes, but in those cases I did all those mockings by hand without much pain, as almost everything being mocked are easy, simple and small on their own, and the wirings are still clear due to the good composition root.
If all these are still not enough to run enough good integration tests, then I might really need a mocking framework :D

P.S.: I've wondered whether it'd be good for RPG Maker MZ to be written with such a game flow :cutesmile:
 
Last edited:

CG-Tespy

Villager
Member
Joined
Jan 5, 2019
Messages
25
Reaction score
11
First Language
Spanish
Primarily Uses
RMMV
It really would be ideal if MV (or MZ, when it comes out) would support unit-testing like Unity does. Not only can you make actual use of MV's API (why be forced to mock everything?), but there's proper support for debugging, and you can even see things in action.

The way I do unit-testing in MV is to set up a subfolder in my project that imports the plugin files as appropriate, then use VS Code and mocha to write and perform the tests. It's saved me a lot of trouble, sure, but not near as much as it would if MV had proper support for testing.

I tried testing through the editor and VS Code, but Yami's method just never works on my end. :/ The debugger doesn't even start.
 

DoubleX

Just a nameless weakling
Veteran
Joined
Jan 2, 2014
Messages
1,748
Reaction score
911
First Language
Chinese
Primarily Uses
N/A
It really would be ideal if MV (or MZ, when it comes out) would support unit-testing like Unity does. Not only can you make actual use of MV's API (why be forced to mock everything?), but there's proper support for debugging, and you can even see things in action.

The way I do unit-testing in MV is to set up a subfolder in my project that imports the plugin files as appropriate, then use VS Code and mocha to write and perform the tests. It's saved me a lot of trouble, sure, but not near as much as it would if MV had proper support for testing.

I tried testing through the editor and VS Code, but Yami's method just never works on my end. :/ The debugger doesn't even start.
I'd like to know which kind of tests do you write, like argument/precondition/postcondition/invariant assertions(these are most of the tests I write), pure function unit tests(almost none in my case), unit tests with mocks/test fixtures(I managed to test notetag reading that way), or even fully automated integration test(like automating a whole scene that I still don't know how to do it well)?
 

CG-Tespy

Villager
Member
Joined
Jan 5, 2019
Messages
25
Reaction score
11
First Language
Spanish
Primarily Uses
RMMV
I go with all of those except postcondition and full integration test. A heck of a lot of mine are pure function tests, since I often need to make sure the same inputs lead to the same outputs.
 

DoubleX

Just a nameless weakling
Veteran
Joined
Jan 2, 2014
Messages
1,748
Reaction score
911
First Language
Chinese
Primarily Uses
N/A
As you mentioned that lots of your tests are testing pure functions, I'd like to know what kinds of plugins you make mainly, and how you managed to have so many pure functions.
My guess is that most of them are deterministic side-effect free functions that takes a part of the current object state set as the input and return the new counterpart as the output, but I'm not sure if that's the only way to have many pure functions leading to drastic increase of testability, especially when making otherwise very stateful plugins :)
 

CG-Tespy

Villager
Member
Joined
Jan 5, 2019
Messages
25
Reaction score
11
First Language
Spanish
Primarily Uses
RMMV
You're largely on the mark there.

The MV plugins I've done unit testing for are an API holder (much like Kino's Amaryllis) and one that displays a diary on screen, two pages at a time. Each page can be set to have its own text, font size, font color, background image, etc... It's a very large plugin ^^; What I've mentioned just now described just a small chunk of what the plugin does.
 

DoubleX

Just a nameless weakling
Veteran
Joined
Jan 2, 2014
Messages
1,748
Reaction score
911
First Language
Chinese
Primarily Uses
N/A
I've just read some parts of the default MZ codebase, and it seems to me that I still don't know how to write very testable codes there(without rewriting everything from scratch of course) yet, I'd like to know if anyone has any ideas on how to do so there, as it might be just because I still suck at writing very testable codes in general :)
 

CG-Tespy

Villager
Member
Joined
Jan 5, 2019
Messages
25
Reaction score
11
First Language
Spanish
Primarily Uses
RMMV
If the devs didn't introduce tools to make that reasonably easy, then that's one big mistake on their part... It seems we're going to have to use the same methods we did for MV to do unit-testing, then :/
 

DoubleX

Just a nameless weakling
Veteran
Joined
Jan 2, 2014
Messages
1,748
Reaction score
911
First Language
Chinese
Primarily Uses
N/A
Actually, I'm already trying to rewrite the whole MZ codebase, with the restriction that my rewrite can't break any plugins not written with my rewrite in mind.
While it's originally an experiment to test whether having a ES6 codebase version's going to work, I also wanted to increase the value of my example of the experiment in some other ways, so I tried to improve the codebase qualities as well, and testability is one of them.

Unfortunately, so far I've only little success(even though it's still definitely better than none), because in general:
1. Functions supposed to return results are in the form of (Current Relevant State Set, New Input Set) -> New Output Set
2. Functions supposed to make side effects are in the form of (Current Relevant State Set, New Input Set) -> New Relevant State Set
3. Functions supposed to both return results and make side effects(this should be avoided as much as possible but MZ already has some of them) are in the form of (Current Relevant State Set, New Input Set) -> (New Relevant State Set, New Output Set)
4. Usually, the codebase's testable if all the current and new relevant state set, and the new input set are easy, simple and small enough to be mocked and/or setup with test fixtures
5. However, it seems to me that I still don't know how to break those relevant state sets in the default MZ codebase into smaller and more manageable and organized pieces, as least to the point of making automated testing worthwhile enough
While I haven't given up yet, it seems to me that I still have a very, very long way to go :)

P.S.: One of my little dreams in MZ is the following:
1. The default MZ codebase's extremely testable
2. The default MZ codebase comes with a comprehensive automated test suite clearly stating the assumptions in the codebase
3. Almost all plugins are written with extremely testable codes
4. Almost all plugins come with automated test suites clearly stating what assumptions in the codebase they rely on, what they knowingly break in their plugins, and what new assumption they've there
This way, most compatibility issues will be catched automatically, informing what plugins are involved and sometimes even with their root causes, because most compatibility issues stem from either of the following(other than conflicts on the conceptual levels):
1. Plugin A relies on some assumptions in the default MZ codebase, but plugin B has broken that knowingly
2. Plugin A has some new assumptions, but plugin B has broken that unknowingly
So when there are compatibility issues, the automated test suite of at least 1 of the plugins involved should have at least some failed tests, and figuring out the big picture behind the compatibility issues should become much easier, simpler and smaller tasks with those detailed info in mind :D
 

Users Who Are Viewing This Thread (Users: 0, Guests: 1)

Latest Threads

Latest Posts

Latest Profile Posts

Today's the day I post my demo!
I am having a great day today humans! May your day be great too! Praise the altar.
Recently discovered that an old member called "Kaus" stole some tilesets from other games, claim as yours, and some people associate his nick with mine. Besides having the same name, he was publishing tilesets too. Bad luck? Guess maybe I'll need to change my nickname... :(

My best practices for a new amazing plugin by this author https://kagedesuworkshop.blogspot.com

Forum statistics

Threads
104,192
Messages
1,004,553
Members
135,712
Latest member
Saediga
Top