What makes a good battle system?
Way too complex a question to answer concisely, but at a high level, I think that good games are either
strategically (or intellectually) engaging,
viscerally engaging, or
emotionally engaging. There are lots of ways to do this; here are a few dynamics you can try to hit (there's no need to hit all of them; focus on what you can do best):
Strategically:
- Force the player to make interesting decisions during battle; make them want to do two things when they can only do one
- Make sure your enemies are varied, so that each battle requires different strategies
- Minimize the effect of RNG, or introduce RNG elements that change situations rather than determining results
- Design battles so that multiple approaches can work
- Ensure your characters' skills and abilities have interesting utility beyond damage and healing
- Make information easy to access during combat
- Find an interesting use for an additional "stat" beyond HP, MP, and TP, and work it into the core of your battle flow
Viscerally (for a turn-based battle system):
- Design quick, punchy, impactful animations for attacks and skills
- Replace numbers, text, and menus with graphics, animations and character expressions wherever possible
- Err toward fast-paced, not slow-paced, battles when determining battler stats
- Minimize the impact of healing on battles to encourage a quick, driven flow
- Minimize the number of clicks/button presses that it takes to do things
- Spend a lot of time perfecting your battle balance, to increase tension and excitement without causing frustration
Viscerally (for an action battle system):
- Make the response time between input and action as quick as possible
- Add physical "weight" to actions, e.g. knockback of an enemy at the end of a combo
- Zealously replace numbers, text and menus wherever it's feasible
- Aim for a battle flow where characters and enemies don't often hit each other in the same moment; give each moment a clear "winner"
- Use visuals to clarify the hitbox of skills and projectiles
- Make characters "feel" different in the way they move and act
Emotionally (emotional engagement is more often achieved using setting, dialogue, or plot, rather than systems, but here are a few ways you can aid the effort using your combat):
- Allow the player freedom to customize their characters' combat style and talents, making the characters their own
- Insert dialogue into combat, telling stories and showing events as the combat unfolds
- Try to show the player something about the world or the plot through enemy battler sprite design
- Have battles that don't force a Game Over for a defeat, but instead alter some part of the story depending on the combat's result
how do you make a battle system that doesn't get boring after sometime?
All of the things above that contribute to a "good" battle system also should contribute to your battle system not becoming boring. But in addition to that, I'd add that focusing on
situational strategy in combat (certain strategies will be really good in some situations, but bad in other situations where you'd need to find a different strategy) as you design your system can go a really long way here. Also, opting for Visual Encounters (enemies walking around the maps) rather than Random Encounters is a good choice to reduce the player's battle fatigue - it lets them (try to) avoid encounters when they don't feel like combat at the moment.
Make it bold and easily readable. Show character faces. Tailor its style to the theme of your game, but never at the cost of readability. Use graphics instead of numbers/words unless doing so will reduce clarity. Allow the player to access detailed minutia (such as descriptions of status effects, known enemy weaknesses, etc.) through an additional button press or menu, but only include relatively important info as part of the HUD.