Basic ways to make this work:
Different Damage Types
You don't have to get as crazy as "this weapon does slashing, this other one does piercing!". All you need is a basis for intuitive play. Most games do this with Elemental equipment. Fire Swords affect different enemies than Ice Sword. You could even do something similar to Final Fantasy X, where some weapons have "reach" and can hit flying enemies, while others don't, and have a hard time landing against them. You do have to be careful to tutorialize the "damage types" well though. If you don't, your players will spend a lot of time guessing and playing "Guide-Dang-It!".
Different Stats
This is fairly self-explanatory, but unless your combat system is really robust, almost all your players are just going to hit "optimize" and be done.
Specialties
Certain weapons can only activate certain skills/spells. Your sword can't activate some of your spells/skills, so you have to equip the Axe instead... and if you do that, you lose access to some of your Sword skills. Likewise, you can cater equipment towards "Builds". perhaps some weapons lend themselves better to "Lightning Bruisers" or "Glass Cannons". Some weapons could accentuate archetypes very well.
Features
This sort of works with Specialties, but it can play towards "side-grades" if you do it well. Maybe one sword has critical hit chance while another has a chance to inflict poison. Likewise, you can perhaps have states linked to weapon types. Stuns, Bleeds, Armor Breaks, Slows, Confusion, etcetera. Some weapons will proc a debuff on an enemy that lowers their attack power. Other weapons proc a debuff that slows them down. Etcetera.
Rarity
Maybe some weapons are just a lot harder to find in the world. Maybe, the world is full of swords, but a Pike is difficult to find. So, you might upgrade your Sword nearly every single dungeon, but that Pike only upgrades every few dungeons. It's a massive jump in stats, but changing it out is kind of rare.
What I Did:
I did a little of everything. Skills run on specific stats, so changing out your weapons to make specific stats higher so specific skills hit harder is viable. Weapons offer different typings (Slash, Bash, Pierce) which are good against specific kinds of enemies (Slashing works great against anything not armored, Bash against heavily armored, and Piercing against Moderately armored.) and different features. Slashing weapons are almost always faster weapons and stronger. Bashing weapons are fairly slow but sometimes cause stuns. Piercing weapons come with a variety of effects and secondary elements. Some skills can only be used with specific kinds of weapons as well. If you're in the Volcano and want to use Ice Strike, then you need to equip an Axe. If you're in the Ice Temple and you want to use Piercing Flame, you need to have a lance equipped.
Now, what I did is heavily generalized and stripped down for the sake of brevity, but if you put a lot of thought into how combat should work and flow, it isn't difficult to implement all or most of those basic features. Especially to such a degree that each character literally plays differently depending on what is equipped.