Extending RGSS (ruby) with C

[??????]

So, thanks to the wonderfull work of Wannabe && FenixFyreX, the RPG Maker Community is  now able to benefit from loading .so files into the maker directly.

Alot (perhaps most) of you may be thinking "what the hell is a .so file?", so here is a short explanation from my understanding;

A .so file is essentially the same as a .dll file, with one major exception... Rather than being called each time its required via some API call, the .so is able to implant new methods into the ruby (RGSS) environment directly.

Consider the following...

# Ruby Codedef some_method(value) return value >= 100 ? true : falseend# End Ruby CodeThis simple method checks if the value passed as argument 1 is greater or equal to the value of 100. if yes, it returns true and false otherwise.

Now what if we wanted to write this method in C?  There are many reasons for why you may wish to do such a thing, performance being the most common, but I'm not really going into that...

Normally, we would have had to do something like this..

# Ruby Codedef some_method(value)  @method_api ||= Win32API.new('DllName','FunctionName','i','i')  return @method_api.call(value) != 0 ? true : falseend# End Ruby Code

// C Codeint FunctionName(int value){  if (value >= 100)  {    return 1;  }  return 0;}// End C Code

As you can see, this is fairly pointless. Its likely that performance would actually be lost in such a transition; however, if we where able to perform even more of the code on the C side of things (ideally all of it), it could potentially become far superior..

An Example of a much faster way to write such a function within a .so file:

// C Code#include "ruby.h" VALUE FunctionName(VALUE self,VALUE ruby_value){ return NUM2INT(ruby_value) >= 100 ? Qtrue : Qfalse;} VALUE rb_DEKModule = Qnil; void Init_Tutorial(){ rb_DEKModule = rb_define_module("DEK"); rb_define_singleton_method(rb_DEKModule, "function_name", FunctionName, 1);}// End C CodeThe end result of this code is almost identical to using the above Ruby + dll. ie - the ruby method function_name calls the C method FunctionName when called.

So, I wrote a tutorial on how to bring the ability to load a .so into your rpg maker projects. This tutorial can be found HERE.

Annd.. Here is a link to a demo containing a working Tutorial.so file.

Any questions ??

 

[Engr. Adiktuzmiko]

This is surely helpful for those who knows how to write in C. :3

 

[??????]

Indeed it is.

Damn code brackets deleted some of the text. Had to try remember what I had written

 

[cabfe]

Is the C function read on-the-fly, like Ruby (ie. interpreted) or compiled in some way?

If it's interpreted, I don't understand where you gain execution speed, but I'm not much of a programmer so I may be missing something in the process.

 

[??????]

Compiled in a .so.

 

[Zeriab]

I love you dekita <3

 

[gRaViJa]

Could this be used for leeeeeet's say, porting possibilities?

 

[??????]

@Zeriab - Thanks, but no need to love me I just documented my progress as I attempted to write my own and when I finally managed it (after days of failure) I compiled it into more of a tutorial for others to be able to use such features within their projects. - If anything, love to Fenix && Wannabe

Anyway, it should make for some interesting developments to say the least.

@gRaViJa - Uuuummmmmmm, no.

This is because the RGSS301.dll requires a windows environment. It's also likely that the msvcrt-ruby191.dll also requires windows dll's etc to function. Sorry

There probably is a few ways to do such a thing though. Unfortunately, that is beyond my current level of expertise ^_^

 

[FenixFyreX]

Actually, LoadSo is only limited to what it gets attached to. It can be compiled on any platform, but why bother? RGSS is written only for Windows.


RGSS can be ported to other platforms, but the work involved would be tremendous and honestly not worth the time; RPG Maker is more akin to hobbyists. I mean, the amount of time it took to reimplement Ruby extensions is quite high (I won't mention how much hair I've pulled out because of random errors or crashes with no explanation other than "RGSS doesn't like you").


I also support this, and I think it should be expanded on; I'll release a proper topic on LoadSo soon so that you have a proper link, I promise Dekita. Just busy.


For now though, just know that the link to the latest LoadSo dll and script is here:


LoadSo - Dropbox

 

[Tsukihime]

As you can see, this is fairly pointless. Its likely that performance would actually be lost in such a transition; however, if we where able to perform even more of the code on the C side of things (ideally all of it), it could potentially become far superior..

Can someone provide actual benchmarks showing the loss in performance due to the overhead in invoking win32API calls? It has been cited several times over the years with no numbers beyond "it makes logical sense". Or at least, I've never seen any such numbers.


This would be a great opportunity because now you can execute native code using several different approaches.


Yes, if you're performing a single computation in C where you are simply comparing two numbers, then that's a pretty silly case to use, but if you're making a single call from Ruby and doing millions of computations in your DLL, I hardly think the overhead would make the win32API call "pointless".


To claim that making win32API calls is pointless using such a trivial example is simply misleading, but that's how journalism works.


We definitely see a major difference between processing a bitmap in a DLL vs in Ruby.

 

[FenixFyreX]

Aha, Tsuki, an extension can and will grossly out-do any dll call anyday, and I mean it.I have here an example, it requires Ruby 1.9.3 (or set it up in RPG Maker, it makes little difference), the ruby extension benchmark-ips (which can be loaded into RPG Maker), and some of your time:

Anagram Solver - DLL vs EXT

On my system, the extension is rediculously faster, here are my results:

C:\***\anagram solver>ruby anagram.rbCalculating ------------------------------------- dll 8.193k i/100ms ext 77.240k i/100ms------------------------------------------------- dll 125.400k (± 2.5%) i/s - 630.861k ext 2.493M (± 1.9%) i/s - 12.513MC:\***\anagram solver>As you can see, the dll, in 5 seconds, was called ~125.4 thousand times per second, whereas the extension was ran ~2.5 million times. Simple math indicates a speed of ~20x faster.And, when we look at it, the overhead does in fact make logical sense. Whereas Win32API has to take data given to it and re-map it to it's c values via the prototypes ('l,p,i,c, etc'), an extension deals with data without all of that boilerplate code. The Ruby internals have a direct pointer to the C method associated to the ruby one being called, and so can pass the data directly, whereas ruby values have to be converted to C data and then sent through Windows' internals after LoadLibrary(), GetProcAddress(), and finally calling the function recieved, all in the meantime with a bunch of error checking.

I wouldn't say that extensions make Win32API 'pointless', but if you take the time to write out an extension, it'll prove much more beneficial than the dll approach

EDIT: Hahaha, just for laughs I made the exact same program in Ruby, and the C Extension runs ~390 times faster. (6,461 iterations per second as opposed to ~2.5 million).

 

[Tsukihime]

The numbers look appealing.


Does this also mean we can load any .so that comes with ruby?


I would assume that some time is required to perform the win api call, but afterwards it should be as fast as a C extension?


I mean, it seems silly for ruby to continue to get in the way after it's passed the data over.

 

[Matombo]

Is the C function read on-the-fly, like Ruby (ie. interpreted) or compiled in some way?

If it's interpreted, I don't understand where you gain execution speed, but I'm not much of a programmer so I may be missing something in the process.

it's compiled to a .so file i guess (short version: .so i the linux variant of a .dll) but linked and called during runtime (bacause that's what dynamic link librarys are for ^^)

 

[FenixFyreX]

@Tsuki - Not all original Ruby so extensions are supported yet. There are a few that are 90% there; sockets are one of them. However, it takes time to re-implemente a whole API of a programming language, haha.


As for your inquiry as to why Win32API is still slower after the first call; yes, the LoadLibrary() and GetProcAddress() are only called once, but when you call the instance of Win32API in Ruby, every value you send to it is assumed to be a new one, never sent before. Therefore it has to map every value passed to it every time it is called, checking the type(s) of the variable(s) passed to make sure calling the Win32API function in C land doesn't crash or cause a segfault.


Plus, it has to map the result to a Ruby variable as well, so it has to parse the return prototype and (for lack of better phrase) cast the C data as a Ruby value.


The list of original Ruby extensions working is not great, but a lot are on the brink of working. It's the small encoding functions from the original Ruby C API that I have to implemente for StringIO, Socket, iconv, objspace, openssl, pathname, psych, ripper, strscan, win32ole are all waiting on encoding methods to be implemented.


Digest, fcntl, the gem win32-api all work. And of course custom extensions.


The final note: LoadSo cannot make data completely persistent. rb_gc_mark() marks ruby objects from getting sweeped up and freed by Ruby's GC, and I have no way to reimplement it. I've been fighting to get a pointer to the rb_gc_mark() function, but no banana. Until I can get a hold of a way to keep the garbage collector from eating all of the things, we cannot save data properly like we can in the original API.


If anybody has any ideas on how to fix that, now would be the time to say anything. Anything at all. Hahaha

 

[??????]

@Fenix - those test figures are very appealing <3

What was the program doing for the test? was it just some empty methods being called or did they do anything?

@Tsuki - Yes, i suppose it is a little bit misleading, but of course, its also accurate in most cases. Kind of a generalization of ruby speed .v. c speed.

And yes, the speed gained from bitmap processing in a dll is crazy compared to running pure ruby methods to do the same. But again, I've already started transferring my bitmap method dll into a .so for that extra little boost.

And with being able to extend the RGSS with .so's is much better for writing 'hidden' methods - for whatever reason you require.

 

[FenixFyreX]

Dekita, I linked to the code I ran; it solves anagrams; the test is solving "reactive" and "creative"; it determines if the first string is an anagram of the second. The logic to solve the anagram is exactly the same in all three builds; the link is in my post above if you want to download it and give it a try.

@Dekita - Actually, so files aren't any more 'hidden' than dlls. In fact, you have more security with using a dll than you do with using a so file, and here's why:

Spoiler

The whole idea behind LoadSo is that Ruby saves pointers to it's functions in a struct type; by using Object#method and Object's memory address, you can remap (almost) the entire Ruby C API off of these two values). Check out this code:
// For sake of briefness, we assume Ruby's structs, etc are loaded in.struct METHOD { VALUE recv; VALUE rclass; ID id; // Look this up in ruby/ruby.h rb_method_entry_t me;};// If C++, ANYARGS is defined as ..., or nothing if C.typedef VALUE (*cfunc)(ANYARGS);VALUE rb_cObject, rb_cArray;VALUE (*rb_obj_method)(VALUE, VALUE);VALUE (*rb_mod_instance_method)(VALUE, VALUE);VALUE (*rb_const_get)(VALUE, VALUE);VALUE (*rb_ary_dup_)(VALUE);VALUE set_buf_string(char* ptr) { // Some code to recreate a Ruby string.}static inline cfunc get_method_with_func(cfunc func, VALUE obj, char *name) { VALUE vmethod; struct METHOD *method; vmethod = func(obj, set_buf_string(name)); method = (struct METHOD*)RTYPEDDATA_DATA(vmethod); return method->me.def->body.cfunc.func;}cfunc get_method(VALUE obj, char *name) { return get_method_with_func(rb_obj_method, obj, name);}cfunc get_global_func(char *name) { return get_method_with_func(rb_obj_method, Qnil, name);}cfunc get_instance_method(VALUE mod, char *name) { return get_method_with_func(rb_mod_instance_method, mod, name);}VALUE const_get_cstr(VALUE mod, char* name) { return rb_const_get(mod, set_buf_string(name));}void Init_vmethods(VALUE vmethod) { struct METHOD *method = (struct METHOD*)RTYPEDDATA_DATA(vmethod); // This is the magic right here. Long-winded, isn't it? rb_obj_method = method->me.def->body.cfunc.func; rb_mod_instance_method = get_method(rb_cObject, "instance_method");}void Init_LoadSo(VALUE rb_method_ptr, VALUE optr) { rb_cObject = optr; Init_vmethods(rb_method_ptr); rb_const_get = get_method(rb_cObject, "const_get"); rb_cArray = const_get_cstr(rb_cObject, "Array"); rb_ary_dup_ = get_instance_method(rb_cArray, "dup"); // Here, rb_ary_dup_ is now a function pointer to rb_ary_dup() in // Ruby's internals.}

So, in any setting, you can access your functions as long as you know what you are doing programming-wise, as long as you have the right conditions to load in the so file. With a dll, unless the person trying to use the functions inside knows the return type and argument types of the dll, then it's a lot more difficult to use.

If you can understand what is going on there, basically pointers to the Ruby C API functions are held in large tables, and when a method in Ruby is called, those tables are traversed to find the corresponding method to call in C, or a string to parse per Ruby definition. You can access these tables by passing methodmethod).object_id * 2 to the dll.

 

[??????]

ok, had to read over that a few times but i think i get it.

Seems I have to think a little more then about how to achieve a level of 'security' im happy with..

Currently fighting the idea of spreading all the code into various places. Like, having a method or two in ruby - loaded in via a patch type thing at runtime, then having some methods loaded in via a .so and others done in dll - probably have the so interact with the dll for speed on that side too.

Who knows.. But thats kinda off topic somewhat

 

[Tsukihime]

@Tsuki - Not all original Ruby so extensions are supported yet. There are a few that are 90% there; sockets are one of them. However, it takes time to re-implemente a whole API of a programming language, haha.


As for your inquiry as to why Win32API is still slower after the first call; yes, the LoadLibrary() and GetProcAddress() are only called once, but when you call the instance of Win32API in Ruby, every value you send to it is assumed to be a new one, never sent before. Therefore it has to map every value passed to it every time it is called, checking the type(s) of the variable(s) passed to make sure calling the Win32API function in C land doesn't crash or cause a segfault.


Plus, it has to map the result to a Ruby variable as well, so it has to parse the return prototype and (for lack of better phrase) cast the C data as a Ruby value.

Ok, so I can see where the overhead would be when you pass in values and then return it.


But it's still strange that the native extension is running 2.43 million iterations per second, but the DLL is running 20 times slower.


They're both compiled in C and once you've converted everything into a format the machine like, what would cause there to be such a large discrepancy?


Also, what about using DL instead of win32API?

 

[FenixFyreX]

The large discrepancy comes from not having direct values; Win32API internals have to loop through the prototype (aka, 'LLLPLL' or otherwise) and check the Ruby values' types according to those characters, and then convert the corresponding Ruby value into the C equivalent; this requires a loop, and extensive typechecking and probably a few goto jumps, which are slow. That is why Win32API and DL both would be slower than a Ruby extension, in RM or otherwise; with an extension, you are passing direct addresses to Ruby objects around, whereas with DL and Win32API, the Ruby addresses must be converted to the C values they 'contain', sent off to the actual function, then the return value has to be converted into a Ruby value.


TL;DR: Win32API / DL requires iterating through a string, converting and typechecking, whereas Ruby extensions pass addresses directly.


DL is (slightly) faster than Win32API, but not well documented, and clumsy, and still quite a bit slower than a C extension.

 

[Tsukihime]

What is your benchmark calculation based on? I am not familiar with benchmark/ips, so I don't know if it actually breaks the execution into different parts (I'm not sure how it would potentially do given that you simply provide a block and it just measures it)

Is it something like

num iterations / time elapsed"?That wouldn't be a very accurate representation of how fast the code is actually running.

If something takes 2 seconds to start up, runs a million iterations per second for 1 second, and then takes 2 seconds to return the value, that formula would say it's running 200k iterations per second, but if you have something that starts up and returns in say one millisecond and runs a million iterations per second for 1 second, now you're looking at something closer to a million iterations per second.

For functions that are inherently complex (eg: requiring millions or billions of computations within a very short amount of time), I would hate to see it run at 100k iterations per second and would definitely like it running at 2 million per second.

But those numbers still look pretty strange to me (a DLL compiled from C running 20x slower than another object also compiled from C, regardless of the overhead which can't be more than a few hundred ms)