Planning a Message System for the next Maker

[Andar]

There are two new options that need to be discussed as to how to add them to the features.

1) preprocessing texts
there are two or three things that could be improved by preprocessing the external text files before they are used in the game. Question is if the plugin should always check for preprozessing or only if launched from the editor or only on command etc.
The first part is that a number of the text codes will be recursive, like macros or referencing database entries and so on. The preprocessing would replace all those placeholders with their direct codes, making sure that the resulting text string could be directly processed without any recursivity remaining.
That will increase performance, and the macros or other shortcuts are no longer needed for code readability at that time.
The second part is data encryption - if the game is encrypted, the texts should also be encrypted as well. If for no other reason as to make cheating more difficult.
The third part to consider is improved loading performance. In the original definition the texts are split over dozens of text files for better management and readability during project development. But identifying and handling dozens of texts for loading can affect performance, and an scripted preprocessing can change that large number of directly readable files into something easier for the computer to load and handle.
EDIT: it seems as if the engine checks for changes in the folder to prevent some problems. This needs to be checked on creating preprocessed files


2) native namebox
The first real MZ trailer has shown that MZ will have a native namebox in its show text.
And some people have already suggested to use that instead of the regular text field for textcodes that format the entire box, similar to the way noteboxes are used.

While that idea needs to be discussed, I'm not sure if it is a good one.
The namebox needs to be able to be translated as well as the regular text (just think of names with titles, like "John the Blacksmith" or "King Arthur"), and a lot of the regular format codes (like font coloring) need to be used in the namebox as well.
So it might be easier to have both text fields fully processed instead of keeping some textcodes exclusive to either of the boxes.
However it would help to have some codes react differently even if they are available to both text fields - text codes for windows positioning for example, if a text field contains text codes for X/Y-coordinates it should move only that specific text window (name window or text window).
That would reduce the number of different text codes needed if it were done that way.

 

[Tsukihime]

The three features that I'm interested are

1. multiple windows
2. localization process via dynamic text

First, multiple windows. I tried it in MV (apparently 5 years ago) but I ran into some usability issues.



Specifically, window priority. By default, ALL message windows will receive key inputs. So if you press OK, all message windows will continue. This can be solved by explicitly setting which window is "active" or "inactive" which is quite simple to manage in the code, but for RM devs that are actually displaying the messages, it becomes a little unwieldy. I think I never figured out a good way to determine which window should be active or inactive, so that the order that they will be processed can be determined by the dev.

2. External database names

With the screenshot of the database provided, I think it's way too unwieldy. I'd hate to manage my project that way lol.

Actor 1 will always be Actor 1.
Class 1 will always be Class 1.

The database is already stored in memory. I don't think an extra object access or ten is going to make-or-break performance with modern devices.

For dynamically generated objects (eg: instance actors, instance items, etc), reference by ID will naturally be broken. My plugins for instance actors/items all use existing database entries as "templates" so the references will also be copied over, which means if Weapon 1 was called "Short Sword" and I created an instance of it, that instance will also be called "Short Sword" simply because it pulls its name from the template it was created from.

But instance actors/items is up to how plugin developers choose to implement it. My approach might not be optimal but it addresses certain issues related to managing dynamic content.

The second part is data encryption - if the game is encrypted, the texts should also be encrypted as well. If for no other reason as to make cheating more difficult.

Just store it with the data folder. I think it'll be encrypted like everything else. If you mean additional encryption processes, probably not necessary.

The third part to consider is improved loading performance. In the original definition the texts are split over dozens of text files for better management and readability during project development. But identifying and handling dozens of texts for loading can affect performance, and an scripted preprocessing can change that large number of directly readable files into something easier for the computer to load and handle.

Just have your "development" environment with your dozens of different files, and then a "production" version that combines it all together or something.

2) native namebox
The first real MZ trailer has shown that MZ will have a native namebox in its show text.
And some people have already suggested to use that instead of the regular text field for textcodes that format the entire box, similar to the way noteboxes are used.

While that idea needs to be discussed, I'm not sure if it is a good one.

I agree, I wouldn't use the name box that way lol. They should just add a note box for each message.

 

[Andar]

I think I never figured out a good way to determine which window should be active or inactive, so that the order that they will be processed can be determined by the dev.

In the uses I've considered so far, the second window is always subordinate to the first (help window for the entry selected on show choice or similar), so a priority wouldn't be a problem directly.

However I have to agree that your idea of using it within conversations would really rise that order-question.
A primitive solution like "just follow the order in which show text is given" would prevent a lot of nice effects.
At the moment my best guess would be to have one show text be the "primary" and have it give the commands to the secondary windows, including giving priority to them. There would have to be two different commands for this, one that returns the priority once that other message is closed and one that transfers primary and allows the other message box to close this one.
But I admit that it would need a lot more thought to see if this can work.

2. External database names

With the screenshot of the database provided, I think it's way too unwieldy. I'd hate to manage my project that way lol.

Actor 1 will always be Actor 1.
Class 1 will always be Class 1.

The problem is that the database names have to be translatable.
Class 1 would always be class 1, but it would be (for example) "Fighter" in english and "Kämpfer" in German and so on. So there has to be a redirect somewhere.

Where and how to place the redirect is the question however, that is why I raised two different ways on how to do it - both with their own advantages and disadvantages, and I'm still not sure which way would be the better one.

your later comments basically follow my current thinking, with for example a switch to tell the plugin to create or update the "production" variant from the readable files on either playtest or deployment.

The three features that I'm interested are

you only listed two features after that sentence. typo or did you forget the third?

 

[Trihan]

I'm guessing this is the plugin you messaged me about a few years ago, Andar?

 

[Andar]

I'm guessing this is the plugin you messaged me about a few years ago, Andar?

yes, I tried to get something like this done for a long time and was almost ready to place a commission for it for MV when MZ was announced - and then it made more sense to plan it directly for MZ.

 

[Tsukihime]

In the uses I've considered so far, the second window is always subordinate to the first (help window for the entry selected on show choice or similar), so a priority wouldn't be a problem directly.

For non-interactive windows like how in the item menu you have the item selection, and then the help window, that wouldn't be too hard. I think I played around with the idea of that "choice help window" at one point where basically the person asks "what would you like to choose?" and then presents some choices, and then when you scroll up and down the list, the message is automatically updated.

At the moment my best guess would be to have one show text be the "primary" and have it give the commands to the secondary windows, including giving priority to them. There would have to be two different commands for this, one that returns the priority once that other message is closed and one that transfers primary and allows the other message box to close this one.
But I admit that it would need a lot more thought to see if this can work.

The approach I had was like

plugin command: activate_window_1
Show Text "message in window 1"
plugin command: activate_window_2
Show Text "message in window 2"

So basically the active window will receive all commands, and the inactive one will simply wait.
When I tested it by making my own events, it was just way too annoying though. Just didn't feel natural at all.

Maybe a message code to "activate" a window might be better.

The problem is that the database names have to be translatable.
Class 1 would always be class 1, but it would be (for example) "Fighter" in english and "Kämpfer" in German and so on. So there has to be a redirect somewhere.

Where and how to place the redirect is the question however, that is why I raised two different ways on how to do it - both with their own advantages and disadvantages, and I'm still not sure which way would be the better one.

Hmm, let me play around with something.

you only listed two features after that sentence. typo or did you forget the third?

I had separated dynamic text and localization but combined them lol

 

[estriole]

The last point is the required database redirects. By using the number feature above to hold the ID of an entry, this can be very easily automated:

View attachment 146947

as can be seen there would be a specific file for the database entries, with the group being used for the database tab and the name of the field to be replaced. And then the number as the ID. That would add a lot of possibilities especially with the added codes for handling the numbers directly, randomly or indirectly.

There is one specific question to be discussed however: should these codes be used indirectly or hardcoded in? Both versions have advantages and disadvantages.
1) indirect coding
The plugin will look at the database, read the keystring entered there and then look for its replacement in the external text files.
The advantage is that everything works as usual. The disadvantage is that the additional step of getting the keystring from the database will make everything slower (whether detectable slower is not yet known)
2) hardcoded
The plugin will completely ignore the database and use the known keystring directly.
The advantage is that this will be faster to do. But it also means that there is no direct correlation to the editor, and that can make working there more difficult if the developer breaks discipline - he needs to enter at least the names into the database for the editor functions to be readable (like which actor to add - difficult with twenty empty lines that have no name). And if a mistake results in the names of the database being different than the names assigned in the external file - that will be a hell for bughunting.

That is one of the decisions where I would like opinions

i prefer the hardcoded one for the speed...
how about make a method to parse all the json files in data folder... find all the text... then dump them to txt files with LABEL where it come from.

for example if we see the Actors.json. it's an array of hashes as actor entry...
in that hash we can see the "name" which is the actor name...
store that with the LABEL in the text dump...

it could be complicated with events... for example in map001.json.
it's a hash... inside it there's "events" key which have an array of hash as value... and the structure is quite complicated... we also need to understand the "code" for the show text(401)... name box(101)... etc... etc...

the format of the text dump could be as JSON format... since we can edit json files with editor without breaking the LABEL... i found one online json editor example.

JSON Editor Online - view, edit and format JSON online

JSON Editor Online is a web-based tool to view, edit, format, transform, and diff JSON documents.

jsoneditoronline.org

but having offline program might be better...

then step two is making the IMPORT method to parse the text files and return it to the original location in json files...

so what the user should do is
1) DUMP the text of their project...
2) backup the text files first (as the first language text files)(optional... could backup the project itself)
3) edit the txt files without deleting the LABEL (using json editor)
4) then run the IMPORT method to replace all the text based on the dump text files.
5) user then could distribute the data folder with different language to the player to overwrite.

but in my opinion this require separate plugins outside the message system... seeing the difficulty of this is quite high... i also think localization plugins should NOT be a part of Message System. (personal opinion though)... and doing above method also will make it compatible with all message system on theory... for example custom escape code from another plugin could also used with this type of method...

edit: all text replacement will always need word wrapping plugins btw since the length of the text is different after replacing it.

 

[Andar]

then step two is making the IMPORT method to parse the text files and return it to the original location in json files...

I have two problems with that.
1) if I understand that correctly, the player would have to run the importer whenever he/she wants to change the language. That could be handled either by including the importer in the engine or by linking to the importer program from the game, but it is an additional step. Of course it can be handled that way - but your method would still result on several different single-language games instead of one multi-language game.
2) This will not allow the dynamic change of texts, especially not where map events can get random output instead of fixed show texts. How would you make it that a villager can have different greetings or rumors if talked to multiple times without using a conditional branch series with different show texts?

i also think localization plugins should NOT be a part of Message System. (personal opinion though)

The main reason behind this structure is to incorporate two different redirects: The localisation redirect and the dynamic text redirect. I think we both agree that having two text-redirects in two different plugins will only cause one hell of a mess.

And if you come from the other side - that is if you already have to enable a redirect on placeholder text codes in all show text and show choice and similar commands, then adding a redirect for different languages is only a simple change in a string-operation for file- or foldernames.

The critical difference of course is the database. The database entries usually don't require dynamic redirects, although there are a few fields like history/description of items, actors and so on where a change can be used to reflect story advancement.
I just think that it will be better to adapt the localisation of the database to the dynamic text localisation required for a lot of the events, than to use a different localisation method for the database.

 

[estriole]

I have two problems with that.
1) if I understand that correctly, the player would have to run the importer whenever he/she wants to change the language. That could be handled either by including the importer in the engine or by linking to the importer program from the game, but it is an additional step. Of course it can be handled that way - but your method would still result on several different single-language games instead of one multi-language game.

1. we can put the json in different data folder for example
data_german
data_italy
data_indonesian
etc
then made a simple reroute to loading data method to the 'correct' folder instead of 'original' data folder when choosing the language... i think we can do that in DataManager...

2) This will not allow the dynamic change of texts, especially not where map events can get random output instead of fixed show texts. How would you make it that a villager can have different greetings or rumors if talked to multiple times without using a conditional branch series with different show texts?

2. actually the dynamic text plugin can still be used combined with the localization method above... because this only change data folder... so EXTERNAL dynamic text plugin could still OVERWRITE after that process...
maybe example is like this... let's say we're using some imaginary dynamic text plugin which use escape code with keys...
ex: show text command
\dtextrandom[entry_001,entry_002,entry_003]
which using imagination let's say it will show random text either entry 001, 002, or 003 in certain text files... based on the language...

in data map json... it will have values:

{"code":401,"indent":0,"parameters":["\\dtextrandom[entry_001,entry_002,entry_003]"]}

just don't change that value in EVERY language...
so the dynamic random plugin can still works fine without problem...

The main reason behind this structure is to incorporate two different redirects: The localisation redirect and the dynamic text redirect. I think we both agree that having two text-redirects in two different plugins will only cause one hell of a mess.

And if you come from the other side - that is if you already have to enable a redirect on placeholder text codes in all show text and show choice and similar commands, then adding a redirect for different languages is only a simple change in a string-operation for file- or foldernames.

The critical difference of course is the database. The database entries usually don't require dynamic redirects, although there are a few fields like history/description of items, actors and so on where a change can be used to reflect story advancement.
I just think that it will be better to adapt the localisation of the database to the dynamic text localisation required for a lot of the events, than to use a different localisation method for the database.

the main problem using the redirect technique is the difficulty for the 'beginner' maker to use the redirect method. imagine seeing your event page in editor.
show text
\dtext[001]
show choice
choice 1 \dtext[003]
choice 2 \dtext[005]

and so on... it will be confusing when the maker create the game... even not so beginner maker might also confused ...

but using the data json above method... the beginner maker can still create their game like normal... export the text... translate it... import it back... etc... or maybe directly place the new json files to data_languagename folder instead of data folder

 

[Andar]

the main problem using the redirect technique is the difficulty for the 'beginner' maker to use the redirect method. imagine seeing your event page in editor.
show text
\dtext[001]
show choice
choice 1 \dtext[003]
choice 2 \dtext[005]

and so on... it will be confusing when the maker create the game...

that is only when using numbered keys, and the description in the first posts explain that I want to use named keys. Additionally, because everything would be replaced there is no need for a \dtext textcode in the show text command at all.

So your example would read:
show text:
villager.greeting.r[5]
show choice
choice 1: ask.theme.1
choice 2: ask.theme.2
choice 3: ask.theme.3

where r5 would mean "randomly select a number between 1 and 5 and use that entry of villager.greeting
and the choices (without r) would use the given number directly.

And the keys to use for finding the replacement would be defined by the game developer who hopefully knows better than to use pure numbers (like you did) or neutral descriptions (like in my example above). It would be much better to name each villager and reference that name for example.

And going this way, the texts in show text would be in no language at all and still readable, with the external text files containing the language-based replacements for villager.greeting.1 and villager.greeting.2 and so on.

So your method of replacing the json data would change nothing for the show text and similar commands.
It is only the database that has a problem with that form of replacement - as I admitted in the original descriptions.

@estriole the question is: if you already have a working and readable dynamic replacement for show text, is it better to use a different method for the database if that different method is easier, or to adapt the existing method that is better for the show text to the database?

 

[estriole]

that is only when using numbered keys, and the description in the first posts explain that I want to use named keys. Additionally, because everything would be replaced there is no need for a \dtext textcode in the show text command at all.

So your example would read:
show text:
villager.greeting.r[5]
show choice
choice 1: ask.theme.1
choice 2: ask.theme.2
choice 3: ask.theme.3

where r5 would mean "randomly select a number between 1 and 5 and use that entry of villager.greeting
and the choices (without r) would use the given number directly.

And the keys to use for finding the replacement would be defined by the game developer who hopefully knows better than to use pure numbers (like you did) or neutral descriptions (like in my example above). It would be much better to name each villager and reference that name for example.

And going this way, the texts in show text would be in no language at all and still readable, with the external text files containing the language-based replacements for villager.greeting.1 and villager.greeting.2 and so on.

So your method of replacing the json data would change nothing for the show text and similar commands.
It is only the database that has a problem with that form of replacement - as I admitted in the original descriptions.

@estriole the question is: if you already have a working and readable dynamic replacement for show text, is it better to use a different method for the database if that different method is easier, or to adapt the existing method that is better for the show text to the database?

to clarify... i said the json method above can still work with the dynamic text whatever the format is... either it's pure number (it's only example btw... for the sake of example i use easy numbered entry ) or using whatever format (named keys, etc). since the text did not get replaced in ANY language then the dynamic text plugin will still work. so it won't be conflicting...

second... either using numbered format or using named keys... it will still confusing to see in editor...
especially if you made a long project... and decide to fix some event that you created 3 month ago for example... seeing that kind of event editor page might not be an easy task... you still need to open up those text files... ctrl + F... type that key... then see what the event said... imagine doing that for each keys in an event page of cutscenes...

but in the end if a maker ALREADY manage using that kind of dynamic text entry... of course there's no need to use above json JUST for database... it will be overkill... i agree on your point there... but still... there's lots more maker that still not able to use that kind of dynamic text entry...

in my opinion making dynamic text as a localization tools is like using a pocket knife to cut up a tuna... can still be done... but require lots more effort and time for the user to use... but that's personal opinion though .

 

[Andar]

there's lots more maker that still not able to use that kind of dynamic text entry

perhaps, perhaps not.
I agree that this is not something for a beginner - none of the external text scripts ever were.
But I think that the utility options of the dynamic text are good enough to get people to learn.
And I'm not requesting this plugin, I'm paying for someone to write it, so that is my idea that takes precedence.


Another idea that I had considered before and that this discussion has confirmed is that it would probably be best to include some preprocessing the data at deployment (or playtest) for two reasons:
Before deployment, the external text files need to be readable and editable for the developer. After deployment, it would be better if they are in a computer-optimized format for easier loading and handling - which will also prevent people from cheating by looking for the solution inside those text files.

Second reason is that there will be a lot of textmacros and static text replacements that are there to make writing easier for the developer, like a macro for longer format codes or to get certain info from the database into the texts and so on.
If those static replacements are already handled on preprocessing, then the remaining dynamic replacements by the engine at runtime will be much easier to handle.

 

[Solar_Flare]

How would you make it that a villager can have different greetings or rumors if talked to multiple times without using a conditional branch series with different show texts?

I don't see what the problem is with using a conditional branch series with different show texts. It does exactly the job you want and is easy to customize in the future, whereas your dynamic text concept is confusing to edit and more work to customize in the future. Instead of just adding another conditional branch, you need to find the appropriate "key" and add a new entry in an external file.

Side note: with something like my SFG_Switch plugin, it's probably even easier to just add a new entry to a series of conditions.

that is only when using numbered keys, and the description in the first posts explain that I want to use named keys.

Named keys don't solve the underlying issue with key-based localization. The issue is that when you're reading through an event, the text is not there. You have to then open up an external file to figure out what exactly is going on in that event. Sure, it's easier to get the gist of it with named keys then with numeric ones, but it's still not where you need it to be.

If localization were integrated into the editor itself, I'd be all for a named-key system. You could type in the actual text right in the editor and also assign a key to it, for example something like the Unreal Engine does for localized text. But RPG Maker is not built with localization in mind at all, and a key-based system sadly just inhibits the developer's ability to conveniently edit their game.

That's why I ultimately went with a gettext-like system when I built my own localization. It's pretty similar to what estriole has been describing, but uses the PO file format for exported text instead of JSON. There are some rough edges and possibly a few unfortunate design decisions, and there are also some big problems I haven't solved (and a key-based system would run into these problems just the same, because they stem from the root issue of RPG Maker not being built for localization at all). But it does work, and it does allow dynamically switching languages while the game is running. You can take a look for yourself if you want - I have a downloadable demo.