rvpacker : Tool for unpacking and repacking rvdata2 as text for collaboration & version control

[akesterson]

Hi all. I was googling around for ways to better support version control & collaboration of RPG Maker VX Ace projects (see http://forums.rpgmakerweb.com/index.php?/topic/2166-version-control-rpg-maker-boom/ for an example of why this is hard). For those of you with experience managing teams on software projects, you will know that RPG Maker's "binary blob" dataformat makes collaboration a real pain in the neck.

I then stumbled onto SiCrane's utility on the GameDev forums (http://www.gamedev.net/topic/646333-rpg-maker-vx-ace-data-conversion-utility/), which does exactly that. I hadn't seen anything like this packaged up nicely for RPG Maker yet (I was thrilled to find it at all - I was about to write my own). So I gave the tool's CLI a facelift, and put it up on github.

https://github.com/akesterson/rvpacker

From the readme:

--------

rvpacker

A tool to unpack & pack rvdata2 files into text so they can be version controlled & collaborated on

Credit to SiCrane

These are copied/lifted/modified from SiCrane's original YAML importer/exporter on the gamedev forums. I initially just put them in github so I wouldn't lose them, and added the rvpacker script frontend.

http://www.gamedev.net/topic/646333-rpg-maker-vx-ace-data-conversion-utility/

Usage

This is a command line utility written in ruby; it should run anywhere with Ruby 1.9 or higher with psych 2.0.0 and trollop gems.

$ ./rvpacker.rb --helpOptions:        --action, -a <s>:   Action to perform on project (unpack|pack)       --project, -d <s>:   RPG Maker Project directory             --force, -f:   Update target even when source is older than target  --project-type, -t <s>:   Project type (vx|ace|xp)              --help, -h:   Show this message

For example, to unpack a RPG Maker VX Ace project in ~/Documents/RPGVXAce/Project1:

rvpacker.rb --action unpack --project ~/Documents/RPGVXAce/Project1 --project-type ace... This will expand all Data/*rvdata2 files into (PROJECT)/YAML/ as YAML files (YAML is used because the object serialization data is retained, which ruby's YAML parser is very good at - otherwise I would have changed it to JSON). The Scripts will be unpacked as individual .rb files into (PROJECT)/Scripts/.

To take a previously unpacked project, and pack it back up:

rvpacker.rb --action pack --project ~/Documents/RPGVXAce/Project1 --project-type ace... This will take all of the yaml files in (PROJECT)/YAML and all the scripts in (PROJECT)/Scripts, and repack all of your (PROJECT)/*rvdata2 files. You can trust this to completely reassemble your Data/ directory, so long as the Scripts/ and YAML/ directories remain intact.

Workflow

This is great for teams that are collaborating on an RPG Maker project. Just add a few steps to your existing workflow:

• Checkout the project from version control
• Run 'rvpacker.rb --action pack' on the project to repack it for the RPG Maker tool
• Load up RPG Maker and do whatever you're going to do; save the project
• Run 'rvpacker.rb --action unpack' on the project
• Commit everything to version control (ignore the Data directory since you don't need it anymore; use .gitignore or .hgignore or whatever)

... Now your project can be forked/merged in a much more safe/sane way, and you don't have to have someone bottlenecking the entire process.

 

[akesterson]

Also, if there is anything you would like to see added/changed, please feel free to head over to Github and file a bug (like this one):

https://github.com/akesterson/rvpacker/issues/1

... and I'll try to get on it as soon as I can.

 

[Tsukihime]

A good extension would be some way to automate that workflow transparently, so users can simply pull/push or update/commit without having to think about it.


They might need to use a custom client or something for specific VCS, preferably a stand-alone client that comes with the required gems/libraries.

 

[akesterson]

A good extension would be some way to automate that workflow transparently, so users can simply pull/push or update/commit without having to think about it.

They might need to use a custom client or something for specific VCS, preferably a stand-alone client that comes with the required gems/libraries.

Yeah, I'll be bundling it up as a rubygem at some point here shortly to make installation easier. I suppose a standard MSI installer isn't out of the question either.

As far as automating the workflow, that's the tricky part, and it might be up to the individual team. My first idea was to make a git pre-commit hook that modified the staged index to capture changes to the file relationships (regenerate out of date Scripts/ YAML/, or repack Data/, etc); but that's problematic for several reasons at a version-control level. Other teams might decide that it's better to have Maker users (who primarily use the Maker interface) commit the Data/ directory to one branch, and the programmers/etc who primarily work in YAML to commit to another, and have the CI system merge their branches into a common mainline from which everyone can pull changes. It's hard to say what everyone is going to want & what will work with various version control systems. (update: good suggestion, I opened an improvement for it: https://github.com/akesterson/rvpacker/issues/2 )

Teamwork is hard

 

[Tsukihime]

Given that it's a CLI tool, different teams might just end up writing batch files for pushing or pulling and then instructing members to run them.


That may be easier and more flexible as an automation option.

 

[akesterson]

rvpacker has been refactored into separate parts (a RPG library, RGSS library, and rvpacker script), and version 1.0.0 published as a rubygem. Installation (including all dependencies) is now as simple as:

 

[]$ gem install rvpackerEnjoy.

 

[akesterson]

Version 1.1.0 of the rubygem  is available (and on github). This version adds the "-D / --database" flag to tell it that you only want to pack or unpack one specific database (weapons, saves, scripts, etc). This makes it more useful for automated build scripts to use, turning it into a scalpel as opposed to a broadsword.

I've also put up an RPG Maker VX Ace skeleton project, which shows how you can use this for collaboration across your group. 

Enjoy

 

[akesterson]

Hi all. I was alerted this morning to a bug in rvpacker 1.1.0 (github bug report - thanks Nate) that made it appear to be horribly broken. In this bug, if you don't specify --database, rvpacker will only operate on the scripts database - this is as opposed to 1.0.0, which would pack/unpack every database file all the time. 

I have yanked version 1.1.0 from rubygems, and version 1.2.0 has been released to fix this confusing and unpleasant behavior. If you are on 1.1.0, I recommend you upgrade to 1.2.0. If you are on 1.0.0 because 1.1.0 was broken for you, please feel free to upgrade now.

Remember, if you find bugs in rvpacker, you can PM them to me here, or file a bug report on github. Thanks!

 

[akesterson]

So as any of you using rvpacker with more than one map designer have probably figured out, it's quite possible for you to encounter a map naming collision that rvpacker cannot help you with. I have been looking at this for several days now, and I have ultimately come to the conclusion that this isn't something that can be elegantly solved in software by rvpacker. Ultimately, it's made too hard because of the way RPG Maker handles the map data.

The good news is that there is a way to use rvpacker to enable a workflow that can avoid map collisions with only a little bit of work. For details, read the "Workflow : Avoiding Map Collisions" section of the rvpacker README. For more details on the reason why rvpacker can't solve this completely on its own, see the "Why rvpacker can't help with map collisions" section of the rvpacker README.

Thanks for understanding, maybe a future version of RPG Maker will be more team-friendly and there can be a better solution to this.

 

[LueLusten]

I know this is a old post and sorry, I just want to know can anyone complie this to be portable or is it possible to run this from one of RPG games since it already have 1.8 + in it? 


I want to be able to make updates to my system via my launcher then build the file so the game will work with the updates, this means I can control all game scripts via a online DB, like a online editor and then the launcher will pull the changed build the update and save it.

 

[akesterson]

I know this is a old post and sorry, I just want to know can anyone complie this to be portable or is it possible to run this from one of RPG games since it already have 1.8 + in it? 

I want to be able to make updates to my system via my launcher then build the file so the game will work with the updates, this means I can control all game scripts via a online DB, like a online editor and then the launcher will pull the changed build the update and save it.

No, you cannot do what rvpacker does from inside of a running instance of RPGMaker. Monkeypatching the game classes after the game has started would most likely lead it to crash, since we don't have a way to tell the engine to pause while we patch.

As for writing a launcher, you don't need to integrate with rvpacker to write your launcher. Your launcher could simply download the updated Marshall files (which were previously built by the development group, whether using RVPacker or the RPGMaker tool), drop them in place, and then start the game. Rvpacker would be completely unnecessary on the client side to accomplish this.

 

[akesterson]

Hello there everyone. I wanted to post an important update in regards to rvpacker and my involvement in the project. This same notice is now in the description of the project on github.

At the request of my employer, Nintendo of America, I have discontinued all work on this project. They have not told me to take it down, but I am no longer allowed to continue working on it.

At this time, this product is now unsupported. Please fork it if you want development to continue. Thank you, and I'm sorry.

 

[Zeriab]

Let me first congratulate you akesterson on your job. Congrats

Also, thank you for allowing us to continue using your projects and allowing us to make derivatives. You are awesome <3

*hugs*

 

[akesterson]

Hi everyone.

I just wanted to let you know that ownership & maintenance of rvpacker has transferred to Solistra, and they have plans to continue and improve the tool. You can now find it on their GitHub page (and my old one will redirect there). They have also assumed ownership & maintenance of the rubygem of rvpacker. If you have any questions or issues, please direct them to Solistra by opening an issue on GitHub.

Thanks everyone