Lately I've been wondering, why do nearly all class based games use these?
Familiarity. This lineup works, both in a balancing and a playstyle sense.
But in a single player game ... I wonder, is it necessary?
If you are making the player manage multiple charavters at any given time, then yes, I would argue that it is necessary. If you want a game that uses RPG progression but only manage 1 dude, play Shiren the Wanderer.
But why? I mean, it stems from the old Dungeons and dragons adventuring party didn't it?
The fighter is up front, tanking some skeletons
The cleric is healing this booboos.
The wizard is being a bit of a neckbeard for most of the session because he only has one spell, and he's not wasting it on 'trash' while the ...
The rogue is ... Well, he's not really doing much of anything really, because there's no traps to disarm this encounter, and skeletons are immune to Backstabbing.
You are absolutely right that D&D of old had this problem. This is why every edition after the 1st one made steps to solve it in each new edition (though arguably a favorable solution didn't happen until 4e).
A lot of wonky game design stems from 40 years ago.
Why are we still using these mechanics combined with these ... Archetypes?
I mean, I get storywise why we get Fighters Rogues Clerics and Mages. They appeal to different fantasies.
The heroic knight, the wise yet fragile mage, the nimble rogue, compassionate cleric/white mage.
These are great story tropes and character fantasies, but ...
Why do they have to link to the mechanics in such a ... predictable way?
If I choose to be the mage as a 'character fantasy' but I want to play a tank as mechanical identity?
What If I want to be the Rogue that heals?
Why can't my fighter be the one throwing around the big nukes?
Why shouldn't my white mage be slicing and dicing sometimes, Vampire hunter style?
If this is on a tabletop, then the only thing that keeps you from doing that is your DM not allowing you to have that freedom (which probably meant that he's a super rules lawyer who doesn't let you do fun things because it "breaks the rules").
If this is on video games, you have to realize that not all games will have a high detail of customization that you seek for a character, specially if you are to manage multiple characters. Aside from such a system being impossible or "not worth it", if this sytem is made poorly, you can just build a party combination of just the same one set and then just use that to dominate everything, which is a not-fun experience.
Class balance? I mean, I got a whole lot to say about that, and I think ... Classes as a package traditionally require a whole lot of things to be 'Bundled together'.
Why is the wizard frail? Because his spells deal more damage.
Why can't the cleric attack? Because he can heal.
Why does the fighter get beter defense? Because he gets nothing else.
A few things:
* Wizards are frail because they have crappy hit dice.
* Clerics can attack! They're just not allowed to draw blood or they get traumatized or something.
* Fighters do get more sword swings as well, but that does pale in comparison to Wizards getting better spells. Linear Fighters, Quadratic Wizards and all that jazz.
I just feel like there is so much more combat design space if those things are loosened up a lot more.
...Have you checked out 4e before? I feel like much of your issues has been resolved by that edition...
Maybe the rogue uses Potions better, his fireball is a molotov coctail, when he is standing as a tank he dodges and counterattacks.
Some people call that "Alchemist". 4e has a sort-of equivalent in the Artificer. Remove the item tactics, and you can argue that a Warlord can suit your needs.
A way to mitigate incoming damage (You know, Aggro and Tank stuff)
A way to to recover HP
A way to deal consistent Single target Damage
A way to deal consistent AoE damage
A way to burst Single target Damage
A way to Burst multitarget Damage
What you're saying here is not wrong, but at the same time, a party does not need all of this to be successful. It just needs the good'ol Fighter/Mage/Cleric combo with a good side of tactics. Heck, it's because of tactics that classes have built-in weaknesses: you build your party to exploit enemy weaknesses while covering up your own, and/or bolstering your advangates while hindering your enemy's. One class should not be able to come close to achieving all of those on its own, and if one class does, something truly went wrong in the game balance phase.
TL,DR: Why are the character fantasies (Brave knight, robe dressed wizard) always linked to mechanical identities (Tank, Healer, Single target or AoE DPS)?
It works and it's easy to understand. I'm sure there are games and stories out there that have broken that norm, but a cliche is a cliche for a reason.