It would be a complex system, but could be done without plugins if you are interested. You could create a system where the player "thinks" they are creating unique spells, however, you actually have every single possible spell they could make already created. What you could do is a system where none of the skills are available to the players, and whenever certain switches are turned on, or variable numbers reached, you could cause the player to "learn" one of the spells.
For example, since I don't know the exact combination system you plan to use, you could allow the player to do some sort of quiz where depending on how they answer questions, certain switches are turned on, or variable values added. Depending on the end result of the answers, the player would learn new skills. You could complete this through conditional branches that check for switches and variables. This could be something as simple as if Switch 01 is on, Actor 01 learns Skill 01 all the way to if Switch 01 is on check if Switch 02 is one, check is Variable 01 is > 10, and if all of those are true Actor 01 learns Skill 01.
You could also have a select system as simple as if the player selects that they want to combine fire and water elements or items, they get the skill steam. For example, by selecting fire the player turns on switch 01, by selecting water the player turns on switch 03, the conditional branch checks if switch 01 is on then checks if switch 03 is on, and if both are true, gives actor 01 skill 10 steam. Of course there may also exist a plugin out there that could do all of this in a pre-determined system.