Yeah, that's what I did in that fight plus phy res Magitama, but he still tore me a new one since my guy is Mag-based. Optimizing my crew for four boss fights in a row left me with very few options too, lol. Still, I beat him on the first try. I had to fight Thor twice and I tell ya, going through that whole sequence again is a pain in the ass. Though I guess they didn't want to put a save point in and have people get stuck. Brutal game one way or another.
I remember I played my Strange Journey guy like he's Phy-based and kept a power healer in the party, but I don't remember if any of his attacks actually drained his hp.