Memory question concerning data/json files reading

[MushroomCake28]

So I'm in the early stage of writing the core engine of my new game. I'm planning on handling all the game's text (story, NPC, item description, etc.) from outside the editor, in order to allow easier editing and easier translation, and the ability to toggle between languages during gameplay. However, I'm concerned about memory usage, especially if I deploy on mobile.

I intend to make most of the game text into multiple json files, and reading only the json file(s) containing the necessary text on the screen. I'm not sure how javascript's memory control works though, thus this thread. So here's the default code for loading json files into data:

DataManager.loadDataFile = function(name, src) {
    var xhr = new XMLHttpRequest();
    var url = 'data/' + src;
    xhr.open('GET', url);
    xhr.overrideMimeType('application/json');
    xhr.onload = function() {
        if (xhr.status < 400) {
            window[name] = JSON.parse(xhr.responseText);
            DataManager.onLoad(window[name]);
        }
    };
    xhr.onerror = this._mapLoader || function() {
        DataManager._errorUrl = DataManager._errorUrl || url;
    };
    window[name] = null;
    xhr.send();
};

My question is as follow: does the window[name] = null; actually remove the data from the json file from memory? Or does it still stay in the memory?

 

[caethyril]

Here, window[name] = null is used to prepare the property for input. The "load file" request is not sent until the xhr.send method is invoked.

JavaScript is a high-level language, which means low-level stuff like memory management is generally automatic. In JS, unused memory is cleared by the garbage collector (GC): it checks if there are any possible references remaining to the current memory block, and if not, the memory is cleared. null is the "deliberate" empty value, i.e. it will prevent GC of the associated memory.

To remove a property from the window, use one of the following lines:

delete window[prop];
window[prop] = undefined;

You may find this article informative:

delete operator

The JavaScript delete operator removes a property from an object; if no more references to the same property are held, it is eventually released automatically.

developer.mozilla.org

 

[MushroomCake28]

And if setting an assigned variable to null, will it clear that variable from memory until it gets assigned another value?

And alternatively, any way to control manually memory in JS like in other languages?

 

[caethyril]

Edit @MushroomCake28: just realised I skipped over your question, sorry! Setting a variable's value to something else, e.g. myVar = null, will replace the original memory reference; the original reference is removed and the associated memory might become eligible for garbage collection.​

Code such as delete window[prop] will remove the memory reference. The memory will be released next garbage collection if the collector judges that there are no further references left to that memory. Unfortunately there is no simpler answer; there might be complications (e.g. additional or circular references) that prevent the memory from being released.

In case you're interested, I found a more in-depth article on JS memory management:

Memory Management

Low-level languages like C, have manual memory management primitives such as malloc() and free(). In contrast, JavaScript automatically allocates memory when objects are created and frees it when they are not used anymore (garbage collection). This automaticity is a potential source of...

developer.mozilla.org

I think you can force a garbage collection process as follows~

• Ensure you have the following line somewhere in your project's package.json file:
  

  "js-flags": "--expose-gc",

• Run this script call during play:
  

  window.gc();

Source:

Request Garbage Collection for Node.js Processes with N|Solid

Learn to how to easily trigger garbage collection for Node.js processes, running on N|Solid, with the N|Solid garbage collection custom command.

nodesource.com

 

[MushroomCake28]

Thanks a lot! It would seem that manually managing memory is not possible in javascript, and I'll have to trust the algorithm to handle it.

The system I would implement would resemble the $dataMap one, where only a portion of the data is loaded from a json file. I would assume the garbage collection will collect the previous instance (and in the case of maps, the data of the previous map). But only way to know is to test it I suppose.

 

[Jonforum]

this should give you more tips.

Memory Management

Low-level languages like C, have manual memory management primitives such as malloc() and free(). In contrast, JavaScript automatically allocates memory when objects are created and frees it when they are not used anymore (garbage collection). This automaticity is a potential source of...

developer.mozilla.org

You will see what to do and what NOT TO DO.
In your case don't worry your fine with your code.

And for `window.gc();`
It only for force the collector pass , instead randomly.
For strings data is not very useful, but it can be useful if you have very very big thing released in GC you want a a release inside strategic frame.
Example a big video, or 2000 sprites, it can make you a big spike lag when release in random frame in your game, so it preferable to force release inside transition if you can ...

Optimize and control garbage collector [gc]

This is just a little tricky tutorial to unlock the control of gc and optimise performance to avoid gc release at random position in your game. this is a typical example from what you don't want, when people play your game. During the game, the GC release a majors memory and make a big spike...

forums.rpgmakerweb.com

but you shouldn't even worry about String from memory !
unless you intend to include 10 million lines of text!
You will get that some MB, you can do a bench memory test with the dev tool of the sdk nwjs provided in rmmv.
Try stock ~1 00 0000 big string sentence data, and you will see, memory is very cheap for string.

 

[Jonforum]

On my side i use excel

I export in .CSV
And for import and parse i use

Papa Parse - Powerful CSV Parser for JavaScript

I have very big dataBase, but string take nothing to memory.
Excel is very userfull for manager ID and also multi localisation (inclu for auto-translate by microsoft)

 

[MushroomCake28]

@Jonforum Wow, exactly what I was looking for. You're actually doing exactly what I want to do, meaning having an excel sheet with all the dialogues so it's easier to translate the game in multiple languages.

So you suggest not worrying about memory altogether in this case? I've already had a little system built to "manually" assign and "remove" stuff from memory (basically setting stuff to null hoping for the garbage collector to pick them up). I've only tested for equip description and it seems to be working. Tell me if it should work with other stuff: (I separated the equipment descriptions into different json files named following the format dt[num]_equip[equip type].json)

DataManager.loadDataFile = function(name, src, path, combine) {
    //----------------------------------------
    // * Added 2 parameters
    //   - path: if the file is in a different location than data/
    //   - combine: if true, combine the object with an object with pre-existing name
    //----------------------------------------
    var xhr = new XMLHttpRequest();
    if (path) {
        var url = path + src;
    } else {
        var url = 'data/' + src;
    }
    xhr.open('GET', url);
    xhr.overrideMimeType('application/json');
    xhr.onload = function() {
        if (xhr.status < 400) {
            if (combine) {
                window[name] = Object.assign(window[name], JSON.parse(xhr.responseText));
            } else {
                window[name] = JSON.parse(xhr.responseText);
            }
            DataManager.onLoad(window[name]);
        }
    };
    xhr.onerror = this._mapLoader || function() {
        DataManager._errorUrl = DataManager._errorUrl || url;
    };
    window[name] = null;
    xhr.send();
};

//----------------------------------------
// * Loading and erasing Data (specific)
//----------------------------------------

DataManager.getAllWeaponTypes = function() {
    return ['KnightSword', 'HeavySword', 'Bow', 'Dagger', 'Spear', 'Katana', 'Rod', 'Staff', 'Grimoire', 'Shield'];
};

DataManager.getAllArmorTypes = function() {
    return ['Helmet', 'Headband', 'Hat', 'LightArmor', 'HeavyArmor', 'Tunic', 'Cape', 'Cloak', 'LightBoots', 'HeavyBoots', 'Ring', 'Pendant', 'Bracelet'];
};

DataManager.loadEquipDescriptionData = function() {
    var symb = '$dataEquipDesc';
    var path = 'js/database/';
    var wtypes = this.getAllWeaponTypes();
    var atypes = this.getAllArmorTypes();
    window[symb] = [];
    for (var i = 0; i < wtypes.length; i++) {
        var combine = true;
        if (i == 0) combine = false;
        var file = 'dt0' + i + '_equip' + wtypes[i] + '.json';
        this.loadDataFile(symb, file, path, combine);
    }
    for (var i = 0; i < atypes.length; i++) {
        var num = wtypes.length + i;
        var file = 'dt' + num + '_equip' + atypes[i] + '.json';
        this.loadDataFile(symb, file, path, true);
    }
};

DataManager.removeEquipDescriptionData = function() {
    var symb = '$dataEquipDesc';
    window[symb] = null;
};

PS: Basically I'm calling DataManager.loadEquipDescriptionData() when initializing a menu scene or shop scene where equipment descriptions are used, and I'm calling DataManager.removeEquipDescriptionData() when existing the scene.

 

[Jonforum]

Sorry for late answer, well most user have enough memory for basic game, and String data doesn't cost enough to worried about it.
Example on my side, i load all my Excel sheets Strings and permanently store those data inside a global Class.
But for

"not worrying about memory altogether in this case "

yes for Strings.

You can also take a look on memory to see how much your string take in memory. Those values are in Bytes.

This can also help you to check if you have memory Leak with compare snapShot.

If you're interested in this kind of stuff, you have really cool article here where they also talk about string and object size in js.
https://www.mattzeunert.com/2017/03/29/v8-object-size.html
It very good to help understand basic DevTools heap inspector and understand all those weird thing.
It why I decided not to worry about the String Data in my games. They are therefore permanently stored without differential loading.


But yes in your case

window[symb] = [...stuff];
window[symb] = null;

Should release all stuff Objects in this array to GC, if they don't have any other reference somewhere.