Marshal dump integer maxed out at 127

[mogwai]

I'm playing around with a marshal dump because I'm writing an app to convert RMVXA rvdata2 to RMMV JSON. It's going swimmingly until I realized I can't read the dumped numbers over 127. They are all a different identical character in random range counts after 127 integers, and I don't know how I'm supposed to identify the number.

I know I can just use the built in VXAce Marshal module, and in fact I did to make my uncompressed rvdata2 to learn it, but I'm now fumbling to write my own Marshal parser because the guy who requested this app does not own VXAce. Long story short he bought MV instead and he wants to convert the VXA game maps that he made with the demo or the lite version. I didn't read too much into his story. I just got involved in his thread out of curiosity, and writing an app for this seems more fulfilling than working on my games right now.

I actually googled this 127 integer max and it lead me to this article about integers in computing.
https://en.wikipedia.org/wiki/9223372036854775807
I don't know what to make of it. Programming is not one my strong suits because I'm a web designer.

How does Marshal know the integer above 127 when it's an identical character to other integers in all stored data?

 

[Zeriab]

Reading the source may help: https://github.com/ruby/ruby/blob/trunk/marshal.c

Please be aware that Ruby automatically converts between Fixnum and Bignum. The number implementation is stronger than in JavaScript where you just have floating points. This is important as simply converting a Ruby integer to a JavaScript floating point can cause loss of information. I bet there must be wrappers or similar built in JavaScript for handling large integers.

*hugs*
- Zeriab

 

[Trihan]

http://www.2ality.com/2012/07/large-integers.html this may also help, it explains how to work with large ints in Javascript by coding them into strings.

 

[mogwai]

Your posts have helped me more than you know.

It lead me on a google journey from ord to bin2hex to hexdec and this made me take the php route. I abandoned writing my own parser in php or javascript; I got so far, but since I'm using php now to make this thing I can simply exec on an rb script on my server, so long story short I'm now using ruby again.

Ruby does the fixnum and bignum. Ruby has Marshal. I'm going to make this app in ruby.

Only now I have another problem decoding these maps. It seems like everything _loads except for the Tileset data.

The scripts rvdata2 is simply compressed with Zlib. I'm already done with my scripts.rvdata2 parser, because VXA rb scripts will be worthless in MV. I basically only wrote that part as an unpacker. It's just that I thought I could inflate or zlib_decode the tileset too, but it doesn't work.

This is what it looks like.

Spoiler: tilesets before base64 decode

This is what it looks like after base64 decode. It didn't return null so I'm assuming part of the compression is base64.

Spoiler: tilesets after base64 decode

�aGqtqB08�=(���xO,��A�<ϩt�RJ���4`C�8�?��F��0xAG���v�m�C�0

Do you recognize this encoding? I tried the 2 automatic MAX_WBITS but it was a no go.

I don't want to start two threads for the same ruby project.

 

[Trihan]

I don't think I'm going to be much help with that I'm afraid.

 

[Zeriab]

Your issues are probably related to the custom Table class.

The multidimensional array class. Each element is an integer of 2 signed bytes ranging from -32,768 to 32,767.
Ruby's Array class does not run efficiently when handling large amounts of data, hence the inclusion of this class.

You can check a table with this little snippet:

Try creating your own Table class. Then try your hand at implementing custom marshalling: https://ruby-doc.org/core-1.9.3/Marshal.html
I would first implement dummy methods just to see whether it works or not.

I do not know how tables are serialized.

*hugs*
- Zeriab

 

[Trihan]

That's actually a good point Zeriab, I hadn't considered Table being the issue. I don't know how they're serialised either.

 

[mogwai]

You're correct. I was just about to report that I found that class too. My problem was I simply ran my test.rb over and over while making dummy classes every time that the Marshal failed to find a class to propagate, error console.

At first I thought the class table was suppose to do a second Marshal load or a Zlib like the scripts rvdata. I've been fumbling with this all night. I had just discovered this one https://github.com/Solistra/rvpacker/blob/master/lib/RGSS.rb but I still need to thank you preemptively for yours because I will probably need to study both sources if this doesn't match the MV numbers in my json. Thanks!

Before table was returning nil and my output looked like this.

Spoiler

#<RPG::Map:0x007f809c8bf118
 @autoplay_bgm=true,
 @autoplay_bgs=false,
 @battleback1_name="",
 @battleback2_name="",
 @bgm=#<RPG::BGM:0x007f809c8dbd90 @name="Ship", @pitch=150, @volume=100>,
 @bgs=#<RPG::BGS:0x007f809c8dbbd8 @name="", @pitch=100, @volume=80>,
 @data=nil,
 @disable_dashing=false,
 @display_name="Terrible Name Land",
 @encounter_list=[],
 @encounter_step=30,
 @events=
  {1=>
   #<RPG::Event:0x007f809c8befb0
    @id=1,
    @name="EV001",
    @pages=
     [#<RPG::Event::Page:0x007f809c8bee20
       @condition=
        #<RPG::Event::Page::Condition:0x007f809c8bed58
         @actor_id=1,
         @actor_valid=false,
         @item_id=1,

Now my output looks like this.

Spoiler

#<RPG::Map:0x007f9ff68ef128
 @autoplay_bgm=true,
 @autoplay_bgs=false,
 @battleback1_name="",
 @battleback2_name="",
 @bgm=#<RPG::BGM:0x007f9ff68fbd60 @name="Ship", @pitch=150, @volume=100>,
 @bgs=#<RPG::BGS:0x007f9ff68fbb08 @name="", @pitch=100, @volume=80>,
 @data=
  #<Table:0x007f9ff68fbc20
   @data=
   [2336,
    2336,
    2336,
    2336,
    2336,
    2336,
    2336,

....

I guess I don't know which RGSS version this one is meant for because it just says RGSS. All that really matters is if I can make a JSON out of it and have it work in MV in the end.

This is a big time for me. I have never run Ruby outside of RPG Maker before and this is the first time I used a script other than php on my server.

 

[Trihan]

Awesome! Sounds like things are coming along nicely.

 

[mogwai]

My third question is how do I prevent someone from uploading malicious Marshalled Ruby to my server? I really should have considered this before I got excited.

All this table trouble has made me discover that Marhsal is an eval, and it just now hit me.

(this seems like it could be another thread)

 

[mogwai]

^ Ignore this post. I decided to protect my Wordpress and make this an offline app.

I'm just going to post again in this thread because my posts are piling up in this forum. I suppose this is related to the table taken out of the rvdata too anyway.

All I can observe about the data array in the Map000 JSON is that it looks like it can be sliced into 4 arrays. Big numbers are at the top like 7-8000 numbers and then it noticeably shifts down like a layer with a lot of negative numbers and zeros at the bottom.

My hick understand is that they are something like tile ids, but I know that's probably not it from looking at them.

The rvdata table created a JSON identical to MV rules all except for the map shadows.

Spoiler: screenshot - shadows have become impassible pub signs

I wish it was something simple like a tile id, because then I could notice a pattern from my remedial tinkering and fix this.

 

[Trihan]

It's the IDs of the tile patterns. The weirdness of the numbers is because the array stores the specific IDs of autotile patterns as well so it knows which graphic to display.

 

[mogwai]

This is a snippet I'm looking at from Shaz's Ace to Mv Converter. This thing has been around and I just now discovered it. I mean it was literally right in my DLC folder or whatever the whole time...

Spoiler: snippet

    def self.get_map_data(data)
      res = []
      for z in 0 ... data.zsize - 1 # 3 map layers
        for y in 0 ... data.ysize
          for x in 0 ... data.xsize
            res.push(data[x,y,z])
          end
        end
      end
      for y in 0 ... data.ysize # new 4th map layer?
        for x in 0 ... data.xsize
          res.push(0)
        end
      end
      z = data.zsize - 1 # original 4th layer (regions + ???)
      for y in 0 ... data.ysize
        for x in 0 ... data.xsize
          res.push(data[x,y,z])
        end
      end
      for y in 0 ... data.ysize # regions
        for x in 0 ... data.xsize
          res.push(data[x,y,3] >> 8)
        end
      end
      res
    end

If zsize, ysize, and xsize is what I think it is and not some special Ruby thing, I can probably steal her technique in Javascript to get rid of my solid Pub sign shadows problem.

I've been writing on my thing for a week and now I feel like it's all for nothing, because the thing already exists and probably works better.

I also spent a week making a character made of generator parts from meta notes, but then I realized there was better and popular one already made by SumRndmDde.

I need to l2google

 

[Trihan]

Take a lesson I learned for myself not too long ago: just because someone else made a thing doesn't mean you can't make your own take on it. A lot of the plugins I'm planning to release soon have already been done in some way or other by other plugin/script developers, but there's nothing wrong with a bit of healthy competition.

 

[mogwai]

No, her thing is better. I just tried it. The maps are perfect with her thing. Now I just got to get mine to match.

Heh. I thought those for z in 0 ... data.zsize were like javascript for loops of the map data and a res.push(data[x],data[y],data[z]);

Only when trying my 9x for loop of the array of 10000 size and I crashed Firefox did I realize that is probably not it all... Urp...

So I used her thing to see how my generated JSON measures up on http://jsondiff.com/ and I compared each value with script to be sure, and my map data is identical except her thing has exactly 5000 trailing zeros in the end of the array.

I don't suppose there is a way to math this 5000 zeros out? I only tried one map. I'm not certain it will always be 5000.

I think somewhere in those zeros is the shadow data, or just empty space that bumps the shadows into their place...

 

[Trihan]

There's a padZero function built-in to Javascript that you can call on numbers.

 

[mogwai]

Thanks for helping, but that would make my numbers into a string.

I need to figure out why a good map JSON has like

[8532,8532,503,503,0,0,0,0,0,0,0,0,0,0,0]

and my map only has like

[8532,8532,503,503]

Does MV have like 4 layers where Ace only had 3? I bet this has something to do with extra tilesets or extra layers. I'm getting close.

 

[Another Fen]

I still haven't worked much with the MV, so I can't say anything for sure here, but the 4th mapping layer was advertised as one of MV's features (it is used in the editor to stack a second layer of objects from tileset B-E).

VXAce's tilemap actually has a z-size of 4. The first 3 levels are holding tile data, the 4th is holding a combination of the Region-ID and Shadow Pen (Bits 0-3 indicate the shadow corners, Bits 8-13 hold the Region-ID)

MV's tilemap has a z-size of 6, with 4 layers holding tile data, the 5th layer holding shadow data and the 6th holding the region ID.

Hope that helps^^

 

[mogwai]

Thank you sir. Everyone posting in this thread is so helpful.

I actually just went ahead and tried the ellipsis operator and the array with three indexes(keys?) like in Shaz's script. Now I know what that does.

I realize how I'm supposed to do it now. I've got to do it slightly differently because her script is making a string that can be JSON parsed and my script making a javascript object that can be JSON stringified to be written.

But I see how it works now. It helps that her script gives me a json result to shoot for. I've been using http://jsondiff.com/ all day and have been trying to get my json identical to hers.

___________________
EDIT:

If I can steer the conversation back onto the original topic about integer representation in Ruby, I've learned that any negative numbers in the Animations.rvdata2 are simply 6,5536 greater than their absolute value. I know that the animations don't go up that high in width, so I'm simply doing,

if(integer > 60000)
   integer = integer - 65536

I know the animations aren't that big, so my thing is an easy dirty trick. It's an odd coincidence that this works, but what would happen if I needed an integer that high?

 

[Another Fen]

RGSS's Tables can only hold 16-bit signed integers, meaning that you can only store values between -32768 and 32767 anyway (for bigger numbers you have to rely on Arrays or similar objects that can hold any data type).

So, as you already noticed, values above 32767 should actually be smaller by 65536.