(b.mhp *0.10) + a.atk - b.def. This would almost guarantee at least 10% of the enemies HP has gone. However, we understand.
Let's plug some numbers in.
100 Max HP
10 Enemy Defense
1 Character Attack
(100*0.10) + 1 - 10 = 1 damage
100 Max HP
10 Enemy Defense
10 Character Attack
(100*0.1) + 10 - 10 = 10 damage
100 Max HP
50 Enemy Defense
80 Character Attack
(100*0.1) + 80 - 50 = 40 damage
2,000 Max HP
50 Enemy Defense
80 Character Attack
(2000*0.1) + 80 - 50 = 230 damage
So, what did we learn? We learned that so long as the Character Attack is higher than (or equal to) the Enemy's Defense score, you will indeed do at least 10% of damage.
What else did we learn? We learned that the Character Attack isn't all that important to dealing damage. In fact, it's nearly negligible. Without the Attack and Defense, we're doing 200 damage a hit. That means, in 10 hits, the boss is dead. The Character Attack, itself, is only contributing to killing the boss 1 turn faster (boss would be dead in 9 turns if we use the Attack and Defense).
So, to make "Attack" even worthwhile to the player, you would need to ensure that Defense values remain low and Attack Values are high.
Look at the amount of "drift" there is. Enemy HP is going to exponentially increase the amount of damage done, while Character Attack and Enemy Defense are going to contribute very little (even with a gap of 30 points, all you're getting is those extra 30 points of damage). What does the formula do with maxed out stats (I'm using RPG Maker MV for this purpose)?
999999 Max HP
999 Enemy Defense
999 Character Attack
(999999x0.1) + 999 - 999 = 99,999 damage.
What if the enemy has the minimum amount of defense?
100,997 damage (or 998 if the game rounds up).
Attack power is effectively meaningless in your game with this formula. Damage dealt is directly related to just how much HP the enemy has and not how good of a sword you have or how high of stats you've got. Even with max stats, you're contributing nothing. With Defense of the enemy at 1 and attack at 999, you're only contributing 998 extra damage. This is 0.009% of the total damage dealt. It's not even 1 tenth of 1 tenth. It is 1/100th of 1/10th of the damage dealt.
This is what I meant by using "extremes" to test your formulas. Test the absolute ceiling of stats against the absolute floor of stats.
Now, if you're looking for examples of how I'd use Max HP and such in combat...
Last Stand (player skill)
(a.mhp - a.hp) * 1.5
Yep, that's literally the only skill I have in my game that uses Maximum HP to deal damage. It's very simple as well. Maximum HP minus Current HP, multiplied by 1.5. It scales effectively over the course of the game (the more HP the player gets, the more damage can be done) and requires a risk investment of needing more missing HP to deal more damage.
I also have another skill to augment this one in which the character can remain alive at 0 HP so that you can maximize damage from this skill.
So, if my character has 100 HP and they lose it all, it's 100 * 1.5 = 150 damage.
If 150 HP and they lose it all it's 150 * 1.5 = 225 damage.
Since my game is "low stat" and getting to 150 Health will likely take a while (requiring at minimum 26 Quests completed), the damage is scaled properly to the point in the game where it would be required.
The problem you're going to run into with trying to use Max HP to deal damage is that at some point, it's going to out-strip your attack stat. With your current formula, it outstrips the attack stat very quickly, which makes your attack stat pretty much useless.
Honestly, if you just want to guarantee a boss dies in 10 hits... just use a state. That way, the player can pile their own Attacks on top of each "tick" in order to accomplish what you're doing here. It frees up your characters to deal all their most powerful skills, leaves your attack stat meaningful, and still ends the fight in 10 rounds or less.
I'd simply suggest that if you're wanting to do something with Max HP, you need to carefully consider
why you want to do something with it and what
purpose it would serve to have it in the game.
As with my skill, the purpose of it is to deal a massive damage spike to the enemy. High Risk, High Reward. It also requires my players manage the TP Resource in order to pull it off as well as Heal Up before the "I can't die" state wears off. It serves a very specific purpose and is a niche skill players can choose to maximize or ignore. At their most powerful, you're looking at needing 180 TP in order to land one of these hits. 100 TP to become immortal for 6 turns, and then another 80 TP in order to deal *3 damage to a single target (in which, you are given 6 turns to come up with this amount of TP).
To use my skill with any degree of reliability and purposefulness, the player will need to have all the right equipment, all the right party members, and the right strategy.
But, that's the purpose it's meant to fill. Adding strategy to the game. Adding a tactic.
I'm not really sure how you'd use a "deal 10% HP damage to the enemy" as anything other than perhaps a state. It simply ends the fight in 10 turns. That's what it does. Is that any different than using Poison? Is it any different than simply giving the player more attack power so that they're dealing 10% HP damage each strike?
I don't really see the point of such skills. They're either insanely overpowered or extremely underpowered. When I opted to deal with them, I went "overpowered" and spent a lot of time putting systems in place to balance that overpowered nature. My goal was to create "strategy" and "tactics".
It's also worth noting I only did it for a single skill. Because of the amount of work I had to put in just to balance that one skill (technically, it's 10 skills), I decided to never muck about with it again. It's too difficult to balance it in a "low stat" game, or even make it feel "fair". It too easily ruins boss encounters. On top of which, I already have 4 levels of Poison... one of which can kill a boss in 5 rounds.