There are a number of factors you can consider when making a decision like this. Here are some of the things I’d try to account for:
1) How much time and effort are you willing/able to put into a system like this? And directly related to this, how many different classes are you aiming to have? How many party members?
An unrestricted class/subclass system can be lots of fun for players as they work to discover class combos and unique ability interactions, but it also introduces one of the most difficult combat balance challenges imaginable for developers. Between all the classes and members of a player’s party, there can be literally millions of setups players could theoretically go into combat with, and you need to make sure that none of the interactions between the pieces break your game too badly. For instance, you’ll probably want to check over your buff and skill interactions for combos which deal disproportionate amounts of damage to enemies relative to the other options the player has at that point in the game. It’s also generally important to check how defensive abilities stack to prevent players from setting up permanent invulnerability combos. Skills which are balanced in a vacuum can become problematic in many ways when they can be freely mixed with every other skill in the game.
Besides checking individual class pairings for broken combos, you also need to review your system for problems that may arise if the player stacks more than one of the same type of class or skill effect. For instance, if your Mage class has a Thunderstorm spell which damages and stuns all enemies for one turn, you could run into an issue where the player sets all of their characters to be Mages and permanently stun-locks every enemy in every single battle.
With that said, good initial skill design can save you a lot of headaches here. Set hard caps on the stacking limits of buffs and debuffs. Damage and healing caps are also something I’d strongly advise. If a skill has any sort of disabling or invulnerability effect, make the activation conditional to prevent it from triggering regularly (maybe the Thunderstorm skill from the previous example can only stun non-boss enemies which have a Lightning weakness, and you also design your game mechanics so that applying any stun in your game automatically gives the target immunity to all subsequent stun effects for the next 3 turns).
You can also trim down some of the theoretical class combinations you need to balance by making magical subclasses scale their damage/healing very poorly when used on a physical base class, and vice-versa. In other words, don’t give most classes usable ATK and MAG stats at the same time. There may still be some niche uses for mixed pairings based on the non-damage/healing utility of each class (e.g. maybe you pair Samurai and Priest just to have an extra character who can revive and cleanse status), but you can at least narrow the field of possibilities this way. It will lighten your balancing load and also make the number of possible class combinations less intimidating for your players.
Going back to the restricted subclass system you mentioned, this would pose a much more manageable balance challenge because the field of possibilities is much narrower. If you’re not willing to put lots of time and effort into playtesting your class balance, this system could actually allow more creativity from your players, because it reduces the chance that you’ll miss broken skill combinations that render everything else irrelevant.
2) What types of challenges do your enemies present to the player? In an unrestricted system, you need to design your encounters (especially bosses) to be doable without any particular class or combination of classes. That’s not to say that any combination of classes should be able to beat any battle in your game, however. The player is still responsible for making good decisions, but you shouldn’t funnel players toward specific classes because they’re the only ones capable of solving the challenges certain enemies present. Maybe some classes will make certain enemies easier, but it’s important to always keep options on the table. An unrestricted class/subclass system should be implemented with the intent of promoting creative problem solving from the player.
3) How much importance do you place on maintaining consistency between each character’s gameplay and story role? Is it okay if a player decides to take your grizzled, bitter war veteran and make him the party’s Dancer class? Or if they decide they want to give the shy, pious acolyte a Beserker/Necromancer class combination when she goes into battle? Is the Chosen One allowed to put down his sword sometimes and be the party’s main healer instead?
From browsing past topics on the subject, I’d say lots of people on these boards place importance on maintaining that consistency between story and gameplay, and I think there are plenty of good arguments for that. But with that said, I also don’t think it’s wrong to allow room for players to make their own interpretations of characters, either. Games are meant to be fun, and people have fun in different ways. If players have the most fun when they’re totally immersed in the story, they’re still free to line up each character’s combat and story roles in an unrestricted system. But if it makes my player happy to imagine that the shy acolyte blows off steam by going wild in battle, or that the war vet studied ballet in his youth, I don’t mind letting them. If they’re tired of playing as the knight in shining armor, I think it’s fine to let them reimagine my main character as a Priest or Thief instead. I think players will generally be flexible using their imaginations to fill in the gaps between gameplay and story (if they care about it in the first place), and that can make them feel a more personal connection to your characters.
With all that said, I think most people would advise you to go for the restricted class/subclass system, especially if you’re aiming to make a smaller project. I think this is generally good advice for most devs to follow. Personally, unrestricted class/subclass systems are one of my favorite features in RPGs, but they’re a huge undertaking to implement and balance properly, so it’s tough to recommend pursuing one unless you’re really committed to the idea.