Best practice when Loading data from json

[Lihinel]

I created a json file to store Unit data for one of Scenes, right now it is stored in data/warfare/ folder under the name Units.json.
I added the following code to my Scene, and it loads the data as it should:

DataManager._databaseFiles.push({ name: '$lihinelWarfareUnitBaseData', src: 'warfare/Units.json' })

However, it doesn't conform with the information hiding scheme of putting everything into my "Lihinel" object that hides classes and scenes from the outside. I did use '$lihinelWarfareUnitBaseData' instead of just '$warfareUnitBaseData' to reach some middleground of preventing conflicts with other scripts.

What I tried first was:

DataManager._databaseFiles.push({ name: 'Lihinel.warfareUnitBaseData', src: 'Units.json' })

but that just leads to Lihinel.warfareUnitBaseData being undefined, I guess its because it treats the entire thing as a string and tries to add that as a variable to window, but I am not entirely sure.

So my question is:
Is it fine as it is, can I fix the Lihinel.Warfare... part, or do I have to write my own methods for loading data if I want to put it into the "Lihinel" object?

 

[Engr. Adiktuzmiko]

IMHO, I'd go with using your own method for loading it instead of pushing it into the functions of DataManager so that it stays usable even if DataManager changes (though I doubt such changes). It will also keep the ._databaseFiles property as original as possible which can help avoid problems down the line.

But this is purely my opinion, there's no scientific basis or whatsoever with this.

 

[Lihinel]

I see, is there a working plugin or tutorial on loading json data?
(not reading/writing json files, but extracting the contents and storing them ingame)
I was a bit lost which methods of DataManager to copy/use as reference, here is what I came up with and what seems to work, as it adds the object "warfareUnits" to "Lihinel".


I edited/copied/used as reference:

loadDataFile
onLoad
extractMetadata

But I had pretty much no idea what to edit in "loadDataFile" so I left it as is, same with "extractMetadata".
I did reduce "onLoad" because I am not dealing with $dataMap or $dataSystem.
So my final working code is:

var Lihinel = Lihinel || {};

  (function() {

 Lihinel.DataManager = function() {
    throw new Error('This is a static class');
}

Lihinel.DataManager.loadDataFile = function(name, src) {
    var xhr = new XMLHttpRequest();
    var url = 'data/' + src;
    xhr.open('GET', url);
    xhr.overrideMimeType('application/json');
    xhr.onload = function() {
        if (xhr.status < 400) {
            Lihinel[name] = JSON.parse(xhr.responseText);
            Lihinel.DataManager.onLoad(Lihinel[name]);
        }
    };
    xhr.onerror = this._mapLoader || function() {
        Lihinel.DataManager._errorUrl = Lihinel.DataManager._errorUrl || url;
    };
    window[name] = null;
    xhr.send();
};

// only reads list, undo comment to read object data and alsways make a special exception
Lihinel.DataManager.onLoad = function(object) {
    var array;
   /* if (object === NAME_OF_THE_VARIABLE_THAT_IS_READ_IN_AS_OBJECT) {
        this.extractMetadata(object);
        array = object.events;
    } else {
        array = object;
    } */
    array = object;

    if (Array.isArray(array)) {
        for (var i = 0; i < array.length; i++) {
            var data = array[i];
            if (data && data.note !== undefined) {
                this.extractMetadata(data);
            }
        }
    }
 
};

Lihinel.DataManager.extractMetadata = function(data) {
    var re = /<([^<>:]+)(:?)([^>]*)>/g;
    data.meta = {};
    for (;;) {
        var match = re.exec(data.note);
        if (match) {
            if (match[2] === ':') {
                data.meta[match[1]] = match[3];
            } else {
                data.meta[match[1]] = true;
            }
        } else {
            break;
        }
    }
};


 })();

and I load/add/create the object with:

LihinelDataManager.loadDataFile("warfareUnits",'warfare/Units.json');

Pretty sure this only reads out Lists from a json file, so if I want to read object I have to change ".onLoad" by uncommenting the part with NAME_OF_THE_VARIABLE_THAT_IS_READ_IN_AS_OBJECT being Lihinel[customDataMapOrSomethingLikeIt]

 

[Aloe Guvner]

You're definitely on the right track and your reasoning is sound. I agree with Engr. Adiktuzmiko that writing your own functions is the right way if you want to load this data into your own namespace instead of a global variable.

There are two ways to do this, and it depends on where you game will be deployed.

1. Node File System module (only works for desktop deployment, NW.js)
   • MV v1.6.2 uses Node v9, find out by typing `process.versions` into the console in test play
2. Asynchronous HTTP request (works for any type of deployment)

If you study how loading save files work, you will see that MV uses #1 for desktop deployments by checking if the game is on NW.js, otherwise it uses #2.

Writing files is different, as writing can only be done by #1. Browsers cannot write files to your computer, but servers can, so that's the difference (NW.js is based on Node, which is a server runtime for Javascript). But going back to reading files ~

For #2, there are 2 ways to make asynchronous HTTP requests in native Javascript:

1. XmlHttpRequest, added somewhere around 2006
2. fetch, added in 2015

In my opinion, `fetch` is both simpler and more powerful than `XmlHttpRequest`, though both can be used to achieve the same goal. The caveat is that Internet Explorer does not support `fetch`, so if that's a requirement you'd have to use a polyfill.

Using async syntax, the function to load data becomes pretty simple, though the syntax might be confusing at first.

Lihinel.loadDataFile = async (fileName, prop) => {
    const response = await fetch(`data/${fileName}.json`);
    const data = await response.json();
    Lihinel[prop] = data;
    console.log(Lihinel[prop]);
    // You can extract the metadata on Lihinel[prop] here also
}
Lihinel.loadDataFile('warfare/Units', 'units');

Not bad, only a few lines of code to read the data to your object.

Try this just from your console first (press F8 or F12 during playtest to open it) and make sure you got it working before adding it to a plugin. Remember that the data loads asynchronously. It's like throwing a ball for your dog, you're saying "go fetch that ball, I will await your return". MV uses a isDatabaseLoaded function to make sure all the asynchronous requests (Actors.json, Classes,json, etc.) are finished before it proceeds with the game, if you are loading a lot of files you may want to write your own similar function.

But see if you can get the data loaded from a single file first using this method and feel free to come back with any questions about the other stuff.

 

[Lihinel]

It works fine in the console, and if it is useable in all deployment cases, it should do the job.
(Btw did my approach above also use Version 2, because of the line "var xhr = new XMLHttpRequest();"? Doesn't seem asynchronous though.)

On that note, how do I deal with it being asynchronous?
The site you linked to only has broken links (404) when it comes to how things are defined (I used it before to look up methods, same there, all "Defined" links broken (404)).

To use this approach, I need a working function similar to isDatabaseLoaded, right? Because otherwise the game might try to run with nonloaded/undefined objects, correct?

 Lihinel.DataManager = function() {
    throw new Error('This is a static class');
}

Lihinel.DataManager._databaseFiles = [
    { name: 'units',       src:'warfare/Units'      }
];

Lihinel.DataManager.loadDatabase = function() {
    for (var i = 0; i < this._databaseFiles.length; i++) {
        var name = this._databaseFiles[i].name;
        var src = this._databaseFiles[i].src;
        this.loadDataFile(src, name);
    }
};

Lihinel.DataManager.loadDataFile = async (fileName, prop) => {
    const response = await fetch(`data/${fileName}.json`);
    const data = await response.json();
    Lihinel[prop] = data;
}

Lihinel.DataManager.isDatabaseLoaded = function() {
    for (var i = 0; i < this._databaseFiles.length; i++) {
        if (!Lihinel[this._databaseFiles[i].name]) {
            return false;
        }
    }
    return true;
};

var _alias_lihinel_scene_boot_create = Scene_Boot.prototype.create;
Scene_Boot.prototype.create = function() {
   Lihinel.DataManager.loadDatabase();
   _alias_lihinel_scene_boot_create.call(this)
 

};

var _alias_lihinel_scene_boot_is_ready = Scene_Boot.prototype.isReady;
Scene_Boot.prototype.isReady = function() {
   if (Scene_Base.prototype.isReady.call(this)) {
       if (!Lihinel.DataManager.isDatabaseLoaded()){return false};
   }
   return _alias_lihinel_scene_boot_is_ready.call(this);
};

Guess it's because I am overlooking something crucial about async.
I checked, the data is already

EDIT2:

var _alias_lihinel_scene_boot_is_ready = Scene_Boot.prototype.isReady;
Scene_Boot.prototype.isReady = function() {
    if (Scene_Base.prototype.isReady.call(this)) {
        if (!Lihinel.DataManager.isDatabaseLoaded()){return false};
    }
    return _alias_lihinel_scene_boot_is_ready.call(this);

works, can someone explain why? does the return somehow mess up the alias? Is it because its not a real alias, but the function is executed and only the return value is stored in _alias_lihinel_scene_boot_is_ready?

Spoiler: solved

I tried the following, but it leads to an infinite "Now Loading..."
EDIT:
I narrowed the error down to

var _alias_lihinel_scene_boot_is_ready = Scene_Boot.prototype.isReady;
Scene_Boot.prototype.isReady = function() {
   if (!Lihinel.DataManager.isDatabaseLoaded){return false};
   _alias_lihinel_scene_boot_is_ready.call(this)
};

if I directly overwrite the method like this, it works:

Scene_Boot.prototype.isReady = function() {
    if (Scene_Base.prototype.isReady.call(this)) {
        return DataManager.isDatabaseLoaded() && this.isGameFontLoaded() && Lihinel.DataManager.isDatabaseLoaded();
    } else {
        return false;
    }
};

But I'd rather alias it, so whats wrong?

 

[Aloe Guvner]

(Btw did my approach above also use Version 2, because of the line "var xhr = new XMLHttpRequest();"? Doesn't seem asynchronous though.

XmlHttpRequest is indeed an asynchronous HTTP request, what you're actually doing is defining an "onload" function (definition, not execution). This is an example of a callback function, which is a common pattern in asynchronous programming. It's like saying to your colleague, "hey go get that data, and then call me back when you've got it".

Using xhr.send() sends the request, and when the request has returned, the "onload" callback is executed. You can see the order with some console.log

That's another reason why I prefer using fetch with async/await, the code easier to read without having to jump your eyes back and forth. XmlHttpRequest is also an unfortunate name, since a lot of web communication is in formats other than XML. Still, there's nothing wrong with using it, just a matter of preference.

The site you linked to only has broken links (404) when it comes to how things are defined (I used it before to look up methods, same there, all "Defined" links broken (404))

Yeah, all those "defined" links are broken. I use it mostly to see the relationships between classes and what functions are available, but I use a text editor to see the code itself.

On that note, how do I deal with it being asynchronous?

This is the magic question and one of the things to adjust to when coming to JS from other languages like python or ruby.

I can think of probably 3 ways to handle it for your case

1. Lazy, but probably it would work
   • Do you need the data on the title screen? If not, you could send the request in Scene_Boot and just not care when it finishes
   • Probably it'll be done before the user clicks New Game or Continue
   • This is definitely not the "right" way of doing things though
2. Loop through an array to check if all the data exists when checking if the database is loaded
   • This looks like your current approach
3. Use Promise.all to manage the loading of multiple files
   • This is the most tricky to get working, so I would give you some code later to get started (on mobile now)
   • It's maybe the "right way" to do it in modern JS. But the "right way" isn't always the best way so if another way works that's simpler, there's nothing wrong with that

As a side note, you have an infinite loop right now because you're checking whether the "name" property exists or not, but it definitely exists because you defined it. Rather, you would want to check if the data itself got loaded.

 

[Lihinel]

Thanks a lot.
For some reason I have an affinity for infinite loops.
(Though I don't consider them infinite, for the same reason I am confident to say, that I have solved the halting problem:
"All programs terminate after the heat death of the universe."
Still waiting for my Turing Award though...)

I assume you mean:

Lihinel.DataManager.isDatabaseLoaded = function() {
    for (var i = 0; i < this._databaseFiles.length; i++) {
        if (!Lihinel[this._databaseFiles[i].name]) {
            return false;
        }
    }
    return true;
};

So the object Lihinel[this._databaseFiles.name] exists, even when the data itself is not yet stored in it?
How do I check against that?
Is there a special keyword?

 

[Aloe Guvner]

So the object Lihinel[this._databaseFiles.name] exists, even when the data itself is not yet stored in it?
How do I check against that?
Is there a special keyword?

I misread your code, you are correct. (I missed it was using the name like 'units' which was testing for the existence of Lihinel.units)

I think you probably figured it out then how to check if all your data is loaded in the isDatabaseLoaded function.

For the sake of curiousity, here's how to load multiple files in parallel using Javascript promises (the #3 item in my list). This will load files in parallel and set a boolean flag only when all loads are finished successfully. That boolean flag can be checked to see if the loading is done or not.

// Define here all the files to load
Lihinel.DataManager._databaseFiles = [
   { name: 'units', src: 'warfare/Units' },
   { name: 'enemies', src: 'warfare/Enemies' },
];

// Returns a promise. That promise is resolved if the file
// is loaded successfully.
// Otherwise the promise is rejected if the file does not
// load successfully
Lihinel.DataManager.loadFile = function(file) {
    return new Promise( async(resolve, reject) => {
        try {
            const response = await fetch(`data/${file.src}.json`)
            Lihinel[file.name] = await response.json()
            console.log('loaded ' + file.name)
            resolve()
        } catch (err) {
            console.error(err)
            reject()
        }
    })
}

// Uses Promise.all to wait for all promises to resolve
// The filesToLoad array can be of any size
Lihinel.DataManager.loadAllFiles = async function() {
    try {
        await Promise.all(Lihinel.DataManager._databaseFiles.map(Lihinel.DataManager.loadFile))
        console.log('Loaded all files')
        Lihinel.DataManager.databaseLoaded = true
    } catch (err) {
        console.error(err)
    }
}

// The flag Lihinel.databaseLoaded = true is only set when all files are finished loading
Lihinel.DataManager.loadAllFiles()