Playing with Excel spreadsheets is very helpful. Come up with a "test equation" for a skill, and run the numbers for a variety of situations.
- A low level character, vs a low level enemy.
- A low level character, vs a high level enemy.
- A high level character, vs a low level enemy.
- A high level character, vs a high level enemy.
Factor in what the skill would look like if used by an OP (end game) character, or against a super powered boss. Try the skill out with mid-level enemies as well.
a.atk - b.def is pretty basic, but it can result in a lot of zeros being processed, and make it impossible to damage certain enemies if you're not careful or fight enemies over your level.
I typically determine an "skill power" for each skill, and a constant to apply to a.atk and b.def, and process the damage as such.
(a.atk * skill power) * ((constant + a.atk) / (constant + b.def))
So if a player has an Attack stat of 25, uses the skill Power Slash, which has an skill power of 8, against an enemy with 45 defense, and use a constant of 30, the damage output would be:
(25 * 8) * ((30 +25) / (30 + 45)) = ~147
If instead, the player had 45 attack, versus 25 enemy defense, the output would be...
(45 * 8) * ((30 +45) / (30 + 25)) = ~491
Why the constant of 30? It grounds the damage a bit, and keeps the variance from getting too out of whack in more extreme cases.
Without the constant in the above equations, the damage outputs would be 111 and 648, instead of 147 and 491. That may seem perfectly fine, but what happens when a low level character with 6 attack stumbles upon an enemy a few levels higher with 18 defense? Or low-mid level player with 32 attack versus a level 1 trash mob with 2 defense?
With constants:
(6 * 8) * ((30 + 6) / (30 + 18)) = 36
(32 * 8) * ((30 + 32) / (30 + 2)) = 496
Without constants:
(6 * 8) * (6 / 18) = 16
(32 * 8) * (32 / 2) = 4096
....and to illustrate the problem with this, give the enemy two more defense, which should be mostly irrelevant against 32 attack...
(32 * 8) * (32 / 4) = 2048
...take away one defense instead?
(32 * 8) * (32 / 1) = 8192
Should one defense increase/decrease the damage by 4096? Probably not.
The constant of 30 is an arbitrary number. You can change it to fit your needs. Increasing it will put less emphasis on defense and pay more attention to the raw attack power of the attack (a.atk * skill power), decreasing it will put more emphasis on the importance of defense.
(45 * 8) * ((0 +45) / (0 + 25)) = 648 damage
(45 * 8) * ((5 +45) / (5 + 25)) = 600 damage
(45 * 8) * ((50 +45) / (50 + 25)) = 456 damage
(45 * 8) * ((500 +45) / (500 + 25)) = ~374 damage
Notice the higher the constant, the closer the output gets to (45 * 8) = 360. If the enemy had higher defense than the player has attack, then the constant of zero equation would yield significantly less than "(a.atk * skill power) = damage", but increasing the constant would bring that number up.
If, for certain skills - you wanted to make the equation simply something like (a.atk * 8), then you'd effectively be giving the attack an "ignores defense" effect. Such an attack would be useful against higher defense enemies, but it wouldn't really shine against enemies you already over-power.
Editting to add:
Also, consider the extreme outliers of your attacks. Let's say the lowest defense an enemy can have is "1", and the highest attack a player can have is 100. Let's assume the strongest attack in the game has a Skill Power of 25.
Here are some possible outcomes for the strongest possible player using the strongest possible attack, against the weakest possible enemy.
(100 * 25) * ((0 + 100) / (0 + 1)) = 250,000 damage (no constant)
(100 * 25) * ((10 + 100) / (10 + 1)) = 25,000 damage
(100 * 25) * ((30 + 100) / (30 + 1)) = 10,484 damage
(100 * 25) * ((50 + 100) / (50 + 1)) = 7,353 damage
(100 * 25) * ((500 + 100) / (500 + 1)) = 2,994 damage
Same skill, same player, but an enemy with 50 defense. Still much lower than the player.
(100 * 25) * ((0 + 100) / (0 + 50)) = 5,000 damage (no constant)
(100 * 25) * ((10 + 100) / (10 + 50)) = 4,583 damage
(100 * 25) * ((30 + 100) / (30 + 50)) = 4,063 damage
(100 * 25) * ((50 + 100) / (50 + 50)) = 3,750 damage
(100 * 25) * ((500 + 100) / (500 + 50)) = 2,727 damage
Notice how the scenarios with low constants yield a massive variance in damage? Should an attack do 250,000 to an enemy with 1 defense, 125,000 to an enemy with 2 defense, and 5,000 to an enemy with 50 defense? No, that's just insanity, right?
But also beware of setting your constant too high. Look what happens if I set my constant to 500. An enemy with 1 defense takes 2,994 damage, and an enemy with 50 defense takes 2,727. There's very little difference. In this case, the all the extra defense the enemy has counts for almost nothing. I would say, in this case, a constant between 30 and 50 seems appropriate.
This way, defense matters enough to justify buffing/debuffing it as much as you can, but not so much that you can enfeeble an enemy a few times and break the game.