RMUnit - Unit Test framework (collaborative project)

[Tsukihime]

I found a test framework that can be run in RPG Maker and decided let's just go with that.


It is basically the Test::Unit framework. Documentation can be found here


http://ruby-doc.org/stdlib-1.8.7/libdoc/test/unit/rdoc/Test/Unit.html


I downloaded the gem version


http://test-unit.rubyforge.org/


Why Testing?


The ultimate purpose of this testing framework is


-to allow scripters to write better, bug-free code


-reduce development time by saving the amount of time required to test your own scripts


-to identify script incompatibilities quickly for both scripters and script-users.


There may have been existing testing initiatives that I am unaware of, and they would be nice to know. But for the most part, testing in RM? Haven't seen much.


Setting up your project


There are two ways to get the framework.


Git repository


The project is set up on github You can get it at https://github.com/HimeWorks/RMUnit


I have structured it so that you should check it out in the System folder. It will create a folder called "RMUnit" which will contain all of the files required to run the framework out-of-the-box.


The repository contains all of the test suites as well (currently there is not much)


Manual download


You can also install the framework manually.


1. Download the framework and extract it to your project's System folder

2. Get this script and place it below Materials and above Main


Show your console when you playtest and you should see the default tests I've written executed and the results shown.

Moving forward


How do we make it better?


I would like there to be a full set of test cases for the default engine. This way, all we have to do is write our own scripts and then run the test code and see if anything crashes.


I would like to then write test cases for our own classes if we have any custom methods that might be nice to know about.


I would like sort of repository where anyone can add tests suites to, and we can place all of those test suites in a folder and have the test framework load up all of the test suites. This is pretty easy to do for the most part.


Ideally, the test suites written for this framework can be used in any framework if anyone else wants to write their own testing framework.


What you do


There are two parts to this, depending on what you prefer to do.


First, is to improve the framework.


Second, is to write test cases. Come up with various unit tests for all of the different parts of the engine. If you've ever come across some situation that you think would be nice for others to know about, submit that as a test case.

 

[Zalerinian]

I downloaded the gem version

http://test-unit.rubyforge.org/

You got a gem to work with RGSS?

 

[Tsukihime]

No, the gem version just happened to be "stand-alone" so I just copied the ruby files over and it worked lol

 

[DarknessFalls]

this seems interesting. as a software developer i write unit tests all day, albeit in php but still. interesting concept.

 

[Solistra]

Okay, I'm interested. I've been writing my own testing framework (which is much simpler, honestly -- the original Test::Unit is rather heavy), but never thought about trying to use an external one. Test::Unit was kind of abandoned after Ruby 1.8.7, though, if I recall correctly. The one I'm writing is loosely based off of how Minitest (from the 1.9 standard library) works.

 

[Tsukihime]

Actually I first tried to get MiniTest to work with RM stand-alone cause I read that it succeeded Unit::Test and was one of the recommended frameworks, but couldn't get it to work through simple copy-pasting so I went with Unit::Test.


Testing in RM does not seem very developed so at this point anything would work. The purpose of my post was to let people know that there's something that works for RM and was one possible solution to build on.


If your framework would be easier to use then that would be preferred since getting people to do testing is probably already a challenge on its own, not to mention writing good test cases.


The idea of being able to write one set of test cases and use it for any framework is something that I'd like to work towards though. I think most frameworks ask you to prefix your test method names with "test_" so that's probably already covered. Being able to contain test cases within their own suites would make distribution easier since it would just be a matter of fetching the latest updates or manually dropping it into your own test folder.

 

[Solistra]

I'm getting close to having my framework ready, but there are a few things that I'd still like to change about it. As it is, though, mine takes any class inherited from Test::Case and automatically includes it in the Test module's known cases, so that's handled by itself -- though it can also load cases from other sources as well. Mine actually doesn't use the 'test' method convention, though -- it's more expectation-based, so test cases are actually written as specifications at this point. Adding the ability to run assertion-style 'test' methods would be easy, though.

Belated edit:

What I mean by "expectation-based" test cases are best demonstrated by showing you what I mean. This is an actual case used to test the framework (written in the framework -- it's so meta):

Spoiler

class StubTest < SES::Test::Case describe 'Stub' do Object.new end it 'returns value from block' do subject.stubobject_id, 0) { 1 < 2 }.must_be_same_as true end it 'returns stubbed value when called inside of block' do subject.stubobject_id, 0) { subject.object_id }.must_be_same_as 0 end it 'returns original value when called outside of block' do subject.stubobject_id, 0) subject.object_id.cannot_equal 0 end it 'generates alias of original inside of block' do subject.stubobject_id, 0) do subject.ses_testcase_stubbed_object_id end.must_be_same_as subject.object_id end it 'removes alias outside of block' do subject.stubobject_id, 0) subject.cannot_respond_to :ses_testcase_stubbed_object_id endend

And this is what the output looks like:

Spoiler

Test Case: Stub  [ OK ] Stub returns value from block  [ OK ] Stub returns stubbed value when called inside of block  [ OK ] Stub returns original value when called outside of block  [ OK ] Stub generates alias of original inside of block  [ OK ] Stub removes alias outside of block  5 tests, 5 passed, 0 failed

 

[DarknessFalls]

to get a community to write tests, not to mention quality tests - would be like trying to get a crack addicted mother to give up her baby and be happy about it.  the community would need a unified test suite, a set of developers to "enforce" the testing of the scripters scripts before submission and then for the developers to test the submitted scripts to make sure they pass. 

I cant see this working.

 

[Tsukihime]

the community would need a unified test suite

Maybe we can use a repository or something.


I'd rather have all of the test cases for the default project in one place, and each scripter attaching test cases for their own scripts, with their scripts, though since a lot of people are using $imported it shouldn't be too big of an issue.

a set of developers to "enforce" the testing of the scripters scripts and then for the developers to test the submitted scripts to make sure they pass.

The purpose of the test suites is to make it easier to test your own scripts (cases you probably never thought of, for example) and check for compatibility, not to prevent people from submitting defective products.


Scripters aren't required to make sure that their own stuff actually works nor should anyone be pressured to test that their own submissions work.

 

[Solistra]

I apologize for being absent for a while, but I had no access to my laptop for a little over a week. As it is now, though, I have almost finished writing the testing framework that I was working on with full support for Test::Unit assertion-style tests, simple expectation-style specifications, mocks, stubs, and output capturing. Overall, the code comes to somewhere around 400 lines, and remains a perfectly usable testing framework with support for loading test files stored externally.


If anyone is still interested, I'm currently writing the tests for the framework itself to ensure it works entirely as expected, and may be releasing it sometime soon. Once that's done, I'd love to see some thorough tests written for RGSS3 itself.


I also support the idea of a repository for test suites. I use GitHub personally, and I know Tsukihime shared the original RMUnit with that, so I'd be perfectly happy to work with that site.

 

[Tsukihime]

I've never done expectation-style testing before.


It looks much different from assertion-style testing, but I am not sure why someone might use one style over another beyond personal preference.


I haven't written any test cases cause it just seems difficult where to start, plus I'm not sure how reliable the framework I provided is lol

 

[Solistra]

It really is about personal preference quite a bit, and I prefer expectation-style tests. I just find them much more readable than assertion-style, but to each their own. Both are supported in the framework that I've been working on (which is complete now, by the way).

As for where to start, I'm not really of any help there. The problem is that we weren't developing the code to begin with, and testing tends to go hand-in-hand with writing the code. Also, the RGSS3 code base is relatively large, and there are some aspects of it that can't be easily tested. Add the fact that a lot of the base scripts provided have abysmally poor design decisions, and... we're left in an awkward state to begin with to say the least.

 

[Samurai T]

If anyone is still interested, I'm currently writing the tests for the framework itself to ensure it works entirely as expected, and may be releasing it sometime soon. Once that's done, I'd love to see some thorough tests written for RGSS3 itself.

I also support the idea of a repository for test suites. I use GitHub personally, and I know Tsukihime shared the original RMUnit with that, so I'd be perfectly happy to work with that site.

I'm interested. Please let us know where we can pick it up.

 

[Solistra]

I'm interested. Please let us know where we can pick it up.

Nowhere at this point in time. I'm still working on a couple of things related to the framework itself, but I'll be sure to post information about it as soon as I make the script publicly available (along with a number of other developer-friendly scripts I've been working on).