First of all it depends on the weapon (and often time weapons like daggers and such are treated with different calculations to reflect this), and secondly when you are talking physical contact strength does hold a very valid part in the equation no matter what 'common knowledge' claims. Yes, an expert martial artist that's a 110 pound woman can throw a 200+ pound dimwitted brute around all day using his own inertia and frick him up so badly he won't know up from down, but the instant you replace that dimwit with an equally skilled and in shape man of 200 pounds it becomes impossible for the smaller fighter to win in a fair fight, it's simple physics.
You also have to consider stamina in the equation once you consider weapons, the more you use a weapon the harder it becomes to keep using it, and physical strength holds a key role in your ability to keep fighting effectively. It doesn't matter how skilled you are if you don't have the physical acumen to continue the fight, as your muscles convulse in protest to effort that is beyond them you'll slow down, get sloppy, your attacks that were once deadly if ignored are suddenly swatted away with ease and ultimately you end up killed when your stronger opponent's weapon flies by yours because your muscles are cramping and you just can't match his speed any longer. This rule applies to anything from a 2-3 pound sword to a massive battleaxe (you'd be amazed how heavy a sword can become after you've been swinging it around for an hour or so) and determines the types of weapons one can effectively wield in the first place.
I think the disconnect is in thinking these things through, in rpg's especially so much is represented in stats and numbers that it's hard to bridge the reality that they are trying to represent. If you look at the standard health for example, it should bother you that equates to being sliced open and bled dozens of times without falling, which is even more offputting as far as realism goes. Instead I generally look at the rpg health gauge as an indication of a characters overall physical acumen as described above, seeing each chip off of the gauge not as a deep, bleeding wound but as a blitz that wears him down, barely deflected that just accumulates until at 0 health, the blow that finally gets through all of his defenses and strikes true. If you look at agility, well how do you actually become faster in the first place? A lot of cardio of course, but you also have to work your muscles and improve them in order to run faster, dodge and swing your weapon faster. Cardio is stamina, strength is speed, all is one. You can go stupid with it and do nothing but lift weights until your strength encumbers you (at which point you'll even look inhuman in your bulk) but you'll never be capable of performing the opposite and letting your muscles atrophy while training yourself to be faster and faster, because one is the path to the other and cannot be achieved except through it.