This is a little more complicated, but what if instead of just rewarding players for a single shot of overkill damage, you applied the idea to a combo attack system? The specifics would depend on the implementation of your combo system, but the idea would be to keep an enemy immortal and stunned once its HP hits 0, and then to rack up as many consecutive hits as possible before their next turn. If the combo breaks or the enemy's turn arrives, they would be immediately KO'd and the battle would end. You could even make it so this special overkill mode could only trigger on the last enemy alive in combat, which would speed normal battles up a bit while also adding a unique layer of strategy (do you try to kill a dangerous foe ASAP, or do you leave it for the end to try to get its bonus rewards?)
This implementation would allow combat centered around combo mechanics to breathe a bit more than usual outside of boss fights, since you could have mobs which die relatively easily without losing out on the long combo chains which make the system fun. You could even add in turn manipulation mechanics like delays for enemies and speed boosts for allies to help maximize the window you have to score combo hits before the enemy dies (perhaps even letting characters "lap" enemies, though you should be careful to set limits on this). The bonus rewards you earn would also be less dependent on your characters' raw stats than on their abilities and your strategies, so there would be less backtracking to farm weak enemies and more chances to enjoy the system on the first pass through each new area.
As for weaknesses in this idea, encounter length might become an issue depending on how frequently battles occur and how long you're theoretically able to keep these chains going each time. Since this concept has the enemy stop interacting with you once the battle enters the finishing combo phase, what you're left with is basically a puzzle for the player to cap off the fight. If every normal battle ends with a finishing combo, and each combo ends up playing out over many consecutive turns while the enemy just waits, it won't be long before the player starts to lose interest in the mechanic. You could attempt to address this by adding some conditions to when this phase triggers in a fight (e.g. the phase can trigger when the last enemy hits 0 HP, but only if they were defeated by a combo attack already in progress, or when you've built up enough of a certain resource, or if you cleared all enemies within X turns, etc.). Making sure players can't chain an unreasonable number of turns together before the enemy takes their turn would also be wise, since this keeps the system focused on having players make the longest possible combos within a limited timeframe, rather than the longest possible combos period.
Another concern would be adding enough variation to the situations the player encounters to make sure they're not just spamming the same sequence of skills over and over again to get an identical, optimal combo every time. A simple resource system tied to combo hits could be one solution, or you could opt for skill cooldowns (which could prompt decisions about whether to use a skill now to deal with an immediate threat, or save it for the next turn to add it to the finishing combo). Another solution could be relying on tricky enemy design to keep the player on their toes. Lots of potential solutions exist to these problems and the various others that might come along with this system, but what works best really would depend on the context of the game.
I'll wrap up here, since it's honestly kind of tough to dive into too many specifics when I've left the core idea of "combo attacks" so vague. Hopefully the concept at least gives you something interesting to consider for your own ideas, though! Good luck!
