Alright, first off, I'm quite sorry for this. If you stick around the forums long enough, you'll learn to recognize me as the resident "jerkface" of the forums. Keep that in mind and we'll get along famously.
I'd like to start you off with the reason why "level ups" exist in RPGs to begin with. They're a psychological "reward" for playing the game. They exist to reward players periodically with that special music and notification of stat increases (you'll notice in most RPGs, the game never tells you just how useful a stat gain is... What does a +2 actually mean to a player? Players don't see the formulas, so these numbers are meaningless). Okay, so why do you reward players like that? In essence, to keep them playing, is why you do it. Not only to keep them playing, but to sort of "hook them" into playing. Now that that's out of the way, let's take a look at the way games like Dungeons and Dragons approach "Level Ups".
First of all, D&D is meant to mostly be "story based". That is, a Level Up and a Stat Increase are fairly meaningless in D&D. I know, I know, you're going to argue that they're useful and important and you use them for all sorts of combat and non-combat things! Yes, you do. Of course you do. The problem is... That's not how games like D&D use levels. D&D uses levels as a way to tell the DM what sorts of challenges to put the players up against as well as what sorts of monsters and traps they can use. It's a scaling tool. Read some DM manuals for lots of tabletops and you'll quickly see the pattern there and realize that's exactly what the level systems are for. DM's scale loot, monsters, encounters, challenges, puzzles, traps, and even scenarios based upon Player Level (and Player Count). Oh sure, Players learn new skills and can sometimes allocate points on Level Up to decide what to do with their characters... But, ultimately, that's just so the Players can feel involved with their characters as well as linked to them. You see, it takes a very long time to get a Character to Level 10 in most tabletops. It's a significant time and story investment to get that far. It's also the reason the Rulebooks often contain rules for rolling up characters that are higher levels than Level 1. Because, ultimately, the levels in tabletops are used as a scaling tool. Even then, they're not always that "accurate" of a scaling tool either. It's entirely possible for a Level 5 Character to get ganked by a Level 4 monster, just because the monster is much tougher. Or, sometimes, the Character just isn't equipped to deal with certain monsters or effects. Or, there's the random chance of botching or the enemy landing a crit.
Neither system uses a "Level Up" in terms of "getting balance". You see, balance is thrown off by a lot of different things. Monster stats, Player stats, Player Equipment, Type advantages/disadvantages, Status Afflictions, Buffs, Debuffs, Party Members, Item Usage, HP Status upon entering the battle, etcetera. There is absolutely no way to make a battle system any harder or easier by trying to "balance" it. Indeed, most battle systems become harder because the only way to do so is for a dev to make a battle absolutely unbalanced and in favor of the monsters winning. The other way to attempt this is to rely on "gimmicks". The problem with "gimmicks" however, is that once a player figures out what the gimmick is, the battle becomes easier than it would have been if you hadn't included the gimmick in the first place. MMOs go with the "Raid Format" of difficulty in that to make a battle harder, it requires you to memorize everything about the enemy and then have everyone participating execute the battle flawlessly in order to prevent a wipe. This is also a form of an "Unbalanced Battle".
With all of that being said... Battles already have a purpose in most every RPG. They're there as a means to provide substance to a game between plot locations and events. Basically, they exist to break up the story, to add tension to the story, and to help you along in handing out those addictive little "Leveled Up!" notifications. Better battle systems adopt a more interesting take on combat. How's that? Well, they use standard fights in an area as a sort of "tutorial" on some element of the combat system. Boss creatures in the area are then "tests" to see how well a player learned their lessons on the combat system. Ever play Earthbound? That is exactly how that game operates and why its battle system is such a joy. The first few battles of the game basically teach the player that reading the text of combat is important in that game. It teams you up immediately with Pokey and your dog (whose name eludes me at the moment). If you read the text, you'll notice that Pokey spends a lot of time doing next to nothing in combat. He hides behind you, cowers in battle, attacks for almost zero damage... Your dog on the other hand is less useless, but he sometimes does things like Pokey does. Once you're set into the game proper (after having been taught that reading the battle feed is important... and will be for the duration of the entire game), you are put up against a lot of relatively weak enemies. This section is to teach you about the mechanics of battle. It doesn't take long for the player to outstrip the weakest enemies (we're talking Level 3 or 4... about 20 fights or so) and show off how the encounter mechanics work. If you are too strong for an enemy, they run away from you (which you learn at level 4). Touching these running away enemies from behind swirls green and sometimes instantly defeats them without going into battle (which are useful concepts for players to learn, especially for later). Meanwhile, an enemy touching them from behind gives you a red swirl and the enemies get to attack you twice in a row. The player doesn't get a "real fight" until they're somewhere close to Level 6 and they encounter the gang "The Sharks". It's here that we learn two more lessons about combat: Just because all the Sharks have the same sprite, doesn't mean they're all the same monster (Sharks come in a variety of flavors you see, and they're the first enemy the player encounters that do this. They are not the last). The other lesson we learn is that carrying health items around is quite important because healing up at a hotel is prohibitively more expensive than simply scarfing a $12 burger (or however much it costs, it's pretty cheap). The entirety of the game is like that. They teach you about new status ailments in the same way, different mechanics of battle that way (like when enemies explode... this is a harsh lesson early in the game, but it's a useful one for much later). Basically, great battle systems are about enemies teaching you how to use the battle mechanics (or about letting players figure out how to be smarter than the enemy) and then offering a Boss Battle (a test!) to make sure the player knows how these things work.
This brings me to your comments about climbing mountains and hiking trails in real life as things that give one self-gratification from "overcoming an obstacle". The problem with that Self Gratification feeling is that not everyone is wired the same way. I wouldn't feel accomplished in hiking a mountain trail in the least. In fact, I'd wonder why I wasted my time when I could've been doing something more amusing. Not everyone has that same opinion as me, though, and therein lies the problem. There are players who will play an RPG for the story (that's me). There are players who want a challenge (these are usually players who enjoy Roguelikes or Shin Megami Tensei players). There are players who want to feel powerful (these are actually the largest amount of players, interestingly enough). There are players who want all sorts of different things from the games they play. You're making the rookie mistake of thinking you need to somehow make a game that caters to everyone. You don't. Don't ever try to do that. Decide how you want your game to play and who you are designing it for. Once you know who you are catering towards, it's much easier to design the rest of the game without that whole pesky "balance" thing you're trying to do. Let me give you a good example: Call of Duty. Pretty terribly unbalanced and unfair gameplay. Quick kills, quick points, twitchy trigger finger gameplay with really high kill counts. Not once have they ever tried to change that gameplay. Why? They know who they're designing for. It isn't about balance for "fairness" in that game. It's about keeping gameplay fast paced (which is why most of the maps are so small) and kills coming constantly. Their playerbase likes the game like that. In fact, most of the people who play CoD hate games like Battlefield because "it's so slow!" and "it takes forever to kill someone!". You see what I'm getting at? Figure out what kind of players you want playing your game, then design the game around those players. That's what your job is, as a game dev.
Honestly, I'm not sure any of that really helps you out in the least... But, a lot of your notions are a bit... skewed. I thought I would address a few of them and maybe help you out. If you want to create a game with challenging fights, then don't worry about balance. If you want a game that rewards you with Self Gratification after long stints and hard gameplay, then do that and don't worry if other players may not like it.
You see, a gamer like me would never play anything like the original Ninja Gaiden. I would find it more frustrating than fun. But, there are gamers out there who love the challenge. After all, even I have beaten Zelda 2 (on the NES!), simply so that I could say I did and how difficult it was. Now, also remember that just because you've made a difficult game, doesn't mean players who like challenge will enjoy it. You have to remember that challenge also has to be fair (not balanced, but fair). Whatever you do has to be able to be completed reasonably within the game itself, even if it's hard to pull off (another piece of advice I'll give on that is to never have anything be luck based. Gamers who want challenge hate that.).
Anyway, I hope that helps. At least, if you read that far down.
