Map Cache Fix

[Ashouse]

Be careful, too, because it will seem like all is well with a couple of maps, even big ones, but try it with 6 or 7 maps, and you will likely run into issues. That is why I thought it was "fixed". I was only testing on a couple maps and menu spamming, map spamming, random move were all fine, but each map still creeps up the ram "x" amount and will never drop below "x" once you have transferred there. So if you had enough maps for a whole game... You would find trouble.


Edit: just pushing on with fingers crossed is probably the best action to take. It is indeed too big of a problem to go unresolved. I stepped away from my project for a while but am thinking of just getting back to work. Custom tiles are not working out for me. The mv tile system is too much for my wee artist brain. Doesnt help that the code pages are in Japanese. Lol 


"Spam the torpedos and full speed ahead!!!" --- Ashouse

 

[Chocobo]

Be careful, too, because it will seem like all is well with a couple of maps, even big ones, but try it with 6 or 7 maps, and you will likely run into issues. That is why I thought it was "fixed". I was only testing on a couple maps and menu spamming, map spamming, random move were all fine, but each map still creeps up the ram "x" amount and will never drop below "x" once you have transferred there

As I stated in the post before my last post, this is the internal RMMV cache management which the devs did on purpose (though I don't like it the way it is).


If you really want an some kind of memory drop, you can try and use the following experimental, hastely-written plugin (combined with my other plugin FixTilingSpritesV2). It basically goes through the internal cache and kicks out every parallax when the player enters a new map. This in effect disables the internal RMMV caching for parallaxes. It only kicks out a small amount of memory, don't expect to see wonders And please keep in mind, this is just a minor amendment and I haven't tested it much. So do not be surprised if it raises hell ;P It should increase the amount of parallax maps to wander across ... but I cannot really say if this increase is substantial or just minor due to the one unfixed bug.

View attachment KickoutParallaxesFromCache.js

 

[Ashouse]

Sorry, I mainly wanted to reiterate for people like me, who get too excited too early. I will try the plugin listed and see how it does. Thanks.

 

[KaYsEr]

Yeah, waiting for the new versions would probably be the best idea, as my fix wasn't really meant as a plugin – it was rather meant to be seen as "pointing the devs into the right direction", since I found out the little fix but do not have the time or capabilities to test it on a larger user base or to dig much deeper If the new versions are incorporated and the problem is not fixed, I'll probably have another shot at it. Apart from that, for the time being it would be a good idea to take this problem to @Galv since my fix is not really that complicated and should be easy to incorporate (although I do not know much about his plugin).

Well, maybe one of the main difference is that his plugin takes images from "/img/layers" folder instead of "/img/parallax". I tried to modify that folder in your new "kickoutparallaxesfromcache" without success. (but again, thank you for doing this)


I was wondering, why not simply fix the "moving event issue" with the original plugin from this topic (or create something similar, since it was really great performance-wise), is it some kind of unfixable bug maybe?

Be careful, too, because it will seem like all is well with a couple of maps, even big ones, but try it with 6 or 7 maps, and you will likely run into issues

Sure, we would reach the limits of a rather modest device pretty soon. After more tests (on a cheap device among many) I can say that even "Kaus Ultimate Overlays" is crashing eventually if you push it (might be caused by tiled-fogs), it just takes more time since usually fogs are smaller images.


But anyway, it seems that even when I simply show a big picture with the basic "show picture" function, it never truly go away (can see with GPU-Z that the VRAM doesn't drop after a map transfer), maybe it's supposed to happen after a while, with GC. Who knows... What I can say is that even with a rather "normal use", memory consumption seems to be quite big for what it is, and it doesn't feel right.

 

[Chocobo]

I was wondering, why not simply fix the "moving event issue" with the original plugin from this topic (or create something similar, since it was really great performance-wise), is it some kind of unfixable bug maybe?

To be honest, I just came to this thread since I randomly saw that my plugin was mentioned here But I just saw that the plugin from this topic seems to borrow a few lines from my plugin (or me and the creator found exactly the same cause and fix ), so it would be a good idea to fix it.

But anyway, it seems that even when I simply show a big picture with the basic "show picture" function, it never truly go away

Yeah, my plugin only fixes parallaxes atm. I haven't dug into the picture business (yet?)


P.S.: There is a japanese plugin that tries to keep the RMMV internal cache to a set number (50 MB per default). I haven't tested it, but maybe someone wants to: http://liply.net/2015/11/14/preload/

 

[Ashouse]

I will see if i can. Was just finishing up a static monster. It looks like he could just have his robot fix the issue. Lol (look at the attached "tweet".) i need food (or so i am told) then i will toy with some plug ins.


Findings: I couldn't get the Japanese plug in to do anything. It has some .es6 files which I don't know what to do with. Perhaps someone who speaks js can have a look. I tried dumping them in with the plugin, but it didn't seem to matter. With that one, it just crashed like default. I'm guessing the .es6 files have to be in the right place(s).


There is actually improvement with View attachment 33559


I do think you are on to something here. I'm going to try to load some other maps and see how bad things get. Note that I have almost no events on these maps. I may have to scale down my city just to put people in it. (like the opposite of a real world problem) Will experiment more tomorrow. Must... sleep...

 

[xilefian]

I am the original author of this Plugin.


There seems to be some confusion as to what this Plugin actually does. @brandos came to me after noticing a large amount of operating system memory being paged about when switching maps and opening/closing the menu whilst having parallaxes in use - a typical sign of a memory leak. There also seemed to be high CPU usage during these events.


I alleviated the leak by caching the most recently visited map spritesets, this is the "Cache Size" parameter. It is default to 5 as that was how many maps brandos estimated the player would switch between for their particular project.


So each time you close the menu or revisit a map the spriteset, along with the parallax map, will be loaded from this new cache system. This helps with load times quite a bit, at the expense of increased memory consumption - however it alleviates the tilingTexture memory leak occurring for the cached maps.


It took further investigation to see what was happening with tilingTexture. A bug in MV causes the key to the texture in PIXI.js' BaseTextureCache to be lost as each time a parallax map is loaded the key used to retrieve the tilingTexture from the cache is lost and replaced with a new key, causing tilingTextures to build up in the cache unused. Due to the size of tilingTextures with parallaxes in memory this builds up drastically very quickly and the client loses performance rapidly (especially going in and out of the menu).


I wrote a quick fix with Spriteset_Map.prototype.cleanupLeaks that simply cleans up the cached texture and deletes the entry from the BaseTextureCache (this is better than nulling the entry as the space in the cache itself is made invalid with the delete operator). This is called when the cache fills up and a new map is loaded.


This plugin does not attempt to fix any problems with Node.js memory leaks. Just keeps the cache in check and prevents the constant reloading of map data and parallaxes between Scene transitions by storing the most recent spritesetMap in memory.


There is one thing not stored; event sprites seem to get disconnected from their events if you load them from cache. This can be fixed properly, but for the current version this Plugin will remove and recreate event sprites.


The Cache Size parameter should be used to balance memory consumption against CPU usage. If you want less memory consumed, set it to a low value. If you want less clean-up to occur (changing maps frequently) then set it to a higher value.


If you believe you have infinite memory then you can set the Cache Size to zero and nothing will be freed (memory will forever build up until the cache is flushed). This may be preferable for games with only a handful of maps.


The SpritesetMapCache flush command will clear the cache, which may be CPU intensive. The best time to do this would be after a number of large parallaxes have been loaded across different maps but the player is going to spend a lengthy amount of time on a less memory intense map (perhaps visiting a tiled world map after going through a town made of several parallax maps?).


Even after a tilingTexture fix is released, this Plugin should still prevent intense CPU usage and large deallocations/allocations occurring when opening/closing the menu and switching maps.

 

[KaYsEr]

I couldn't get the Japanese plug in to do anything. It has some .es6 files which I don't know what to do with.

Same for me. I already tried it previously.


But the good news is that the original plugin from this topic is updated!
http://felixjones.co.uk/RMMV/Plugins/SpritesetMapCache.js


At the time of OP it was version 1.2, but now it's 1.4 and since 1.3 the "moving events bug" was already fixed.


The RMMV staff should include this fix (or something similar) in an update (even a temporary thing) while we wait for the next PIXI or NODE.



This is working well, I did crazy stuff on a weak tablet-PC (2go RAM) for an hour and the game is still running fine. I probably called the menu and transfered between big parallax-maps more than 100 times now.

 

[Chocobo]

This is working well, I did crazy stuff on a weak tablet-PC (2go RAM) for an hour and the game is still running fine. I probably called the menu and transfered between big parallax-maps more than 100 times now.

Haha, this problem really is a strange one, isn't it? I just set up a new project with no plugins but the one from this topic, http://felixjones.co.uk/RMMV/Plugins/SpritesetMapCache.js V 1.4, and 7 maps with big parallaxes. Map 1 has a transfer event to map 2, map 2 to map 3, ... and map 7 back again to map 1. When walking across them, my computer almost crashed. In comparison, my FixTilingSpritesV2 combined with my experimental plugin does not crash my PC. Here is a comparison of the RAM usage given by the task manager while testplaying:



Are you talking about an empty, new RMMV 1.1.0 project or are you referring to a project with other plugins such as @Galv 's?


P.S.:


For the liply plugin, in the liply_preload.zip, there is a .js file which needs to be copied to your game's plugins folder (the src.zip folder just contains the sources for the .js file which is a compressed version so that it is a single file and takes less space - the ending .es6 just denotes ... let's say the used javascript version).

 

[xilefian]

http://felixjones.co.uk/RMMV/Plugins/SpritesetMapCache.js V 1.4, and 7 maps with big parallaxes. Map 1 has a transfer event to map 2, map 2 to map 3, ... and map 7 back again to map 1. When walking across them, my computer almost crashed.

Could you please upload a minimal project that recreates the crash?

 

[Chocobo]

Could you please upload a minimal project that recreates the crash?

Sure, I will PM it to you once it's ready (this will take some time as my upload speed is very sluggish here and the project has a few big parallaxes).

 

[xilefian]

Sure, I will PM it to you once it's ready (this will take some time as my upload speed is very sluggish here and the project has a few big parallaxes).

I made a demo project (MV 1.1) that loads 7 4000x4000 parallax maps for the player to walk between.


felixjones.co.uk/RMMV/Spriteset Map Cache Test.zip (9.8M


I am unable to recreate the large memory allocations you have with this Plugin enabled. Disabling the Plugin shows large allocations being retained, as expected.


I've fully profiled the behaviour in the Node.js debugger and I've watch my operating system's memory watcher, everything looks like it's behaving as expected. My development machine isn't exactly a fair machine to test on, so I ran this on my laptops and deployed to varying browsers with no ill effects occurring.


I've updated the original Plugin with a fixed copy-paste error that causes a null member dereference crash when attempting to flush the cache manually with Cache Size 0 - an unrelated problem.

 

[Chocobo]

My bad, I just noticed I got something mixed up and the parallaxes are not ca. 4000x4000, but 7800x4800 (I edited it in the picture, but I was too late, I guess ).The demo project is uploading now. I tested this demo project on another computer and the memory problems can be reproduced.


PS: I changed from 4000x4000 to more since I noticed irregular behavior pertaining to the class TilingSprite (or better: PIXI.TilingSprite) when it comes to images bigger than 4096x4096 on my machine (see the discussion with Ashouse on my profile for more info).


EDIT: I just tried out your demo project and it does not crash for me - however, I'm pretty sure it does not crash since my RAM is big enough to store every parallax in it. As you can see in the attached image, the parallaxes are never freed on my machine after changing the map talking to the right person. When talking to the left person and then to the right person on each map, the curve still has the same development, but the memory consumption is somewhat less.



EDIT2: I seem to have overlooked the post explaining the real intention of your plugin (I think I got it right at the start, but some post confused me ). As your plugin does not try to alleviate the RMMV internal caching (in ImageManager), you may neglect this post.

 

[xilefian]

EDIT2: I seem to have overlooked the post explaining the real intention of your plugin (I think I got it right at the start, but some post confused me ). As your plugin does not try to alleviate the RMMV internal caching (in ImageManager), you may neglect this post.

I chose to not touch the ImageManager in belief that loading from it is the preferred choice for developers. If people feel the memory consumption of the ImageManager cache is too much then I can easily sort that out, it's only 3 or 4 lines to locate and remove an image from that cache. It should probably be handled by another Plugin that aims to make resource usage more efficient.


This Plugin does what it says on the tin; SpritesetMapCache. It attempts to avoid the nasty amount of loading and memory allocation (and a leak in tilingTexture) that occurs when switching away from Scene_Map with a cache. It makes moving between previously-visited maps and opening the menu faster in terms of load-times and more efficient in terms of memory allocations and CPU usage (no insane processing required).

 

[Chocobo]

Yeah, I feel so, too, although I do not really like the developer's idea of caching every parallax or big image per se without any restriction. The experimental plugin I posted a few posts ago is a quick first attempt to the heavy ImageManager image caching (when it comes to parallaxes). But because of the nasty memory issues (with Pixi or NodeJS), I decided to rather postpone writing a real plugin for it. But of course it would be great if you'd want to do it  


P.S.: If you see it fit for the scope of your plugin, you could integrate a portion of my plugin (http://forums.rpgmakerweb.com/index.php?/topic/57059-memory-leak-when-changing-parallaxes-changing-a-tilingsprites-bitmap/) into your plugin to fix the on-map change of parallaxes (just create an event with an endlessly running parallel process that changes the parallax to "a", waits 5 frames and changes it to "b" and you should see your memory going through the roof after half a minute or so). So people using your plugin would have this issue sorted out and wouldn't need your and my plugin (which are incompatible atm).

 

[Ashouse]

@xilefian Ooh, an update to test! I like where this topic is going. If you do make an optimization plug in as well, please do allow me to test it. I feel like I have a "lower middle road" machine which could represent a large portion of potential players.


You say you delete and restore random move events in the new version, do you know if this has ill effects on conditional branches?

 

[xilefian]

P.S.: If you see it fit for the scope of your plugin, you could integrate a portion of my plugin (http://forums.rpgmakerweb.com/index.php?/topic/57059-memory-leak-when-changing-parallaxes-changing-a-tilingsprites-bitmap/) into your plugin to fix the on-map change of parallaxes (just create an event with an endlessly running parallel process that changes the parallax to "a", waits 5 frames and changes it to "b" and you should see your memory going through the roof after half a minute or so). So people using your plugin would have this issue sorted out and wouldn't need your and my plugin (which are incompatible atm).

I forgot about the ability to set parallaxes in events. Thank you, I'll add support for this.

You say you delete and restore random move events in the new version, do you know if this has ill effects on conditional branches?

The Plugin deletes all event sprites, these are disconnected from the event logic. This is why the old version would have ghost events, as the sprite disconnected from the logic. The events themselves are not affected.

 

[KaYsEr]

PS: I changed from 4000x4000 to more since I noticed irregular behavior pertaining to the class TilingSprite (or better: PIXI.TilingSprite) when it comes to images bigger than 4096x4096 on my machine (see the discussion with Ashouse on my profile for more info).

I'm glad you said that, because I was wondering why the problem is still here when I just tried bigger maps (like around 8000), even 4800x4800 to exactly have a 100x100 map is problematic. (sometime if you are playing a music while the map loads, the sound goes crazy like your computer is asking for mercy)


The demo (with 7 maps of 4000x4000) provided by Xilefian works pretty well (even on a weak device), the only problem seems to be with the fact that you need at least another layer for a map (so you can go under trees or walls etc), so I added another 8th simple map (while being careful to stay under 4096x4096) with 2 layers, using the plugin from @Galv for the second layer (http://galvs-scripts.com/2015/10/30/mv-layer-graphics/), and I feel like some kind of "leak" is still going on if you transfer from map 1 to 8 for a long while.


It's possible that it comes from this second layer (or the fact that you ask "too much" at the same time), on another hand it seems that even using "show picture" is not super reliable, it feels like even non-tilable stuff like pictures remains in RAM for an undeterminate amount of time.

 

[Ashouse]

Okay, I just had an idea, and I hope people are still watching this thread... Hmm.. Rather @Chocobo @KaYsEr @xilefian... That should do it. Lol. I had this crazy idea, but don't know if it is possible. Is there any way to not convert the base layer as a "bitmap"? As in just tell mv to display .png. No, i dont fully understand how, exactly it all works, but we have established that a 24mb png seems to eat as much ram as a 9mb png that is the same size in x,y demensions. As in the same map, even, just different levels of compression/quality reduction. So, maybe by not using the parallax functions at all, but creating a new one. I am waiting to see what the devs are going to do, but something like this would, if it were possible, dramaticly reduce ram usage, and increase performance, especialy in projects with large layered maps (like mine). To be clear, the idea is to rewrite how mv handles background images. I see that show picture causes just as many problems on my end so maybe I am asking for something that can't be done because the language and system involved "simply doesn't do that", but I guess I figure, if a webpage can load an image without using a gig of ram, mv should be able to also? In any case, I am trying some out of box thinking here. Perhaps new pixi will address this, bit my fear is that since this part isn't considered a "bug" it may not get addressed. Ie. "Memory leak == bug, so we'll fix it. Big static images eat ram == well we never really meant for you to do that anyhow, so commision a script writer." All thoughts and enlightenment are welcome.

 

[KaYsEr]

Don't worry, what you are talking about here is a normal behavior. It was already like this in previous RPG Maker.


Once in the RAM, an image will become "RAW" to be handled, whatever how small or compressed the image was at first, this is only a good thing when you have a limited space on a disk or internet connection.


Don't bother too much about this, a pretty big map with 2 layers in parallaxes is not supposed to even reach 500 mo.


When RMMV will be updated to PIXI4 we'll have a proper GC for video memory etc, things will be alright even for rather humble devices.