There's always a misunderstanding about physical damage type and physical element, both are not same.
Physical damage type is a damage that involving the performer ATK, and receiver DEF., while physical element is absoultely something different.
physcial damage type or magical damage type is set in the skill database(not features) in the Hit Type section, while physical element is set from features. More over element can be created as much as you like.
Element is just a system in the game to determine ones affinity. Let's say I make my custom element called Blood. Blood will got nothing to do with physical attack nor magical attack. It's simply an element. If a battler affinity(elemental rate) to Blood is set to below 100%, that battler will get more higher damage from an attack that have Blood elemental attack. This will add the value of the total result damage from physical attack or magical attack.
So basically if you give an enemy physical element 50%, then if your skill is not a physical element type, EVENTHOUGH it is a physical type attack, then the damage will still normal. Hopefully I'm not wrong, everyone can correct me if I'm wrong, because until now, that is my judgement about this matter.
Now for your matter : "he still dealt physical damage"
Of course he is, because by default skill "Attack" is a physical damage type. Look at the Hit Type. But the element if you give the weapon to be Fire, it should do fire elemental finely. So what happens is your attack is do it's physical damage, the damage that calculated from attacker ATK and defender DEF paramater, but added Fire elemental. Again elemental is not something that changing your attack into a magical type(if you want it to be like this), it just a system in the game to determines a battler affinity, got nothing to do with physical or magical type attack.