From a technical point of view, what you'd want to do is:
- Calculate the amount of stats you want players to gain through leveling
- Use that to determine what the appropriate Level for an area would be
- Figure out how many encounters an average player will complete during a run through a dungeon/area
- Use that, as well as the appropriate level, to decide how much EXP to give an average Troop
- Split the EXP that should be given by Troops among each enemy in those Troops
1) This is probably the most complex part. The amount of stats you want characters to gain through leveling from one area to another would generally be equal to the difference in monster stats between the two areas (by percentage, not absolute - for example, if monster stats rise from 7 to 14, you are looking to double the characters; stats as well, not raise them by +7). Then, you need to take away the amount of stats the characters will gain by improving their equipment between the two areas, and any other form of stat-raising they will do that is not Level-based. Whatever is left is how much stats the characters should gain through Leveling. Note that while it's necessary to do this from, say, Area 6 to Area 7, it can also be helpful to look at a long scale and check the difference between, say, Area 1 and Area 7.
2) So if the player needs to gain 10 ATK and 10 DEF
from Leveling over a certain period to keep up with monsters, and each Level provides 3 ATK and 3 DEF, then the party should be gaining 3 levels during this time. We keep this in mind for later.
3) Now, it's important to figure out how many encounters the player is likely to hit (and complete, rather than Escaping for 0 EXP) along the way. In my opinion as a game designer, this isn't something you should adjust for game balance; it's something you should set for player enjoyment, as you can adjust the EXP per troop in Step 4. Ideally, have someone else playtest your game and watch them (or have them record it and send it to you), and track how many encounters they complete. Short of that, you can play through yourself, use the most efficient route you know, and
double the amount you got, because players won't know the efficient route and will get at least a little lost. If you're using Visual Encounters rather than Random Encounters, then the goal to hit is probably to intentionally run into
most encounters along the efficient route (but don't double it).
NOTE: If the number of completed encounters seems to vary wildly between players, that's a convincing reason to give a lot of EXP to boss enemies, as
@Aoi Ninami suggested.
4) Time to do some more math. We know how many levels we need to gain - for example, if the "appropriate" level (as calculated by running through this process) for Area 4 was Level 15, and we need to gain 3 levels before Area 5, then Area 4's encounters, in total, should provide enough EXP to get you from Level 15 to Level 18. If Level 15 requires 900 EXP and Level 18 requires 1650 EXP, then we need to award a total of 750 EXP in Area 4 (as an average). Subtract the amount of EXP your bosses provide - if Area 4 only has one boss and it awards 250 EXP, then the encounters in the area need to award 500 EXP. Divide by the number of Encounters in Step 3. If you found players would complete an average of 10 encounters, each one should award 50 EXP.
5) Divide the 50 EXP per troop average among enemies in the troop. For example, you might have three different troops (of equal difficulty to each other) consisting of (A) 2 Harpies + 2 Slimes, (B ) 8 Slimes, and (C) 1 Harpy + 4 Wasps. If 8 Slimes should total ~50 EXP, give 6 EXP to the Slime so that Troop B awards 50. If 2 Harpies + 2 Slimes should total ~50 EXP, and 2 Slimes represent 12 EXP, give each Harpy 20 EXP so Troop A awards 50. If 1 Harpy is 20 EXP, give each Wasp 7 EXP so Troop C awards 50.
With all of that said, sometimes this kind of technical calculation is the entirely wrong approach. It works very well in a standard, progression-based RPG, especially where character stats are built according to the designer (rather than chosen by the player) and battles tend to be "stat checks" rather than allowing for tons of skill expression (a "stat check" combat system is one in which, taking several different players, their likelihood of success in a battle depends more on the stats their characters have than on the skill/strategy that the players adapt during the battles). Where players can take wildly different paths through a game, or wildly different builds can occur, or skill expression is very high, a different approach is necessary. The above rules are still decent guidelines to use if you need a baseline, but trial-and-error becomes the better approach, and you need to
give yourself the widest balance target you can. See also
this post which goes a little deeper into my game design philosophies regarding balance, and provides a few more links to useful stuff I've written on the topic.