Oooo, math! I love math! I spend a lot of time on damage formulas but for multiple reasons (I'm the type to put conditions and skill effects in my damage formula).
Tip 1: Don't go too high or too low
20 - 50 are good numbers to always start with. If you do stuff like 5, it get's harder to balance because a single +1 makes them +25% stronger than before. If you go too high like 1000, it makes it harder for the player to comprehend what the numbers mean, and it makes divisons/multiplications really wacky due to the nature of large numbers.
20 is good because it's low but not too low. You can add/subtract or multiply/divide with less issues (usually). 50 is good because it's divisible by 100 making it good for certain situations. An example is an ATB system. Setting AGI to 50 by default makes it easier for the player to decide whose faster and by roughly how much.
But basically, for the developer, its good to go 20 - 50 for balancing. For the player, if number comprehension is important (discussed in tip 4), the same 20 - 50 applies but is, imo, more important to stay in 20 - 50 so the player can more easily comprehend what the numbers are doing.
Tip 2: Excel is your friend
Excel is indeed your friend in this endeavor. I use it a lot ot decide average damage, HP, how much a spell should cost, etc. I used to do the "play until it feels right", but I have adopted the formulaic route.
Tip 3: Go-To Damage Formulas
One decent damage formula to start:
ATK * (ATK / (DEF + ATK))
This means the ATK is multiplied by the ratio of the attacker's ATK and defender's DEF + your ATK. The reason your attack is added to their DEF is so if their DEF is 0, the player just deals damage equal to their ATK. If their ATK is, for example, 20, it's just 20 * (20/20) which is 20. if enemy's DEF is also 20, then you'd get 20 * (20/40) which goes to 10.
Found it in a game I play, and I kind of like it.
There is the simple way
ATK - DEF with a minimum damage forced in the code
it's the lazy way of doing things, but it is something that you can do. I wouldn't do the ATK * 2 - DEF that RPG Maker does because that is legit a terrible formula imo.
My current but complicated formula for my project
ATK / ((PDEF / 100) + 1) with minimum damage of 15% of ATK.
This makes it so if PDEF is 0, it's just ATK / 1. If PDEF is 100, then its ATK / 2. Each 1 point of PDEF decreases the ATK by roughly 1%. This makes it so higher PDEF have smaller returns the higher than ATK it is but since it's division, ATK increase also means the more ATK that is being lost from the PDEF.
Tip 4: Numbers Comprehension and does your Project care about it?
Like others said, numbers are pretty arbritrary. I go the formulaic approach so that it's easier to organize and decide how much ATK a weapon should have, etc etc. Best you can really do is put things in place for numbers to be easier to manage for the future. When it comes to starting, it really amounts to how comprehendable you want the numbers to be.
For example, Mobile games (a lot of them) have incomprehendable numbers. You'll get a monster that has 3000 ATK and 2735 DEF and you get something that adds 234 DEF and 835 ATK and you deal 12,567 damage, but the player has no idea why. They just know big number leads to big number.
For some games, this is fine like Skyrim, for example. A Sword does 300 damage but you don't know the formula relationship of damage and armor rating but it doesn't matter for the game. Then there are games like Dungeons and Dragons where knowing the numbers IS the game..
).
