I think the backtracking really needs to be worth the effort, both story-wise and item-wise. In the classic Metroid, backtracking usually nets you essential items you need to proceed (i.e. the Varia Suit or the Ice Beam), and the game is an action adventure so each individual combat, such as it is, is very brief.
Imagine how backtracking would feel if the monsters were invisible and combat took, say, 30 seconds each time, even if Samus was ridiculously overpowered compared to the opponent. It would feel a lot more tedious than it does in Metroid, or other action adventure games. If you have the typical Random Encounters and no way to bypass them or turn them off, backtracking can be quite painful without some rapid means of travel.
It's not painful in terms of party danger but it is in terms of boredom. Also, if backtracking is over-used, with "Ooooh, look at ALL of the things you can get if you have this item!", it's very easy to lose track of "I needed to go back to X when I have item Y" I know I quickly got lost with Castlevania: Order of Ecclasia, and backtracking the 30th time through the same castle, hoping you have the item you need, is just painful.
An easy way to avoid the extra combat is to have each Troop have an inherent "level" based on how difficult the opponents are. This would be an arbitrary decision by the developer. If that level is exceeded, the Abort Battle command, with some brief text saying "The XXX ran away" at least avoids non-rewarding combat by suicidal enemies (Yay! I won a whole 3 XP! But it takes me 10,000 XP for the next level.). After all, in the wild, most creatures REALLY don't go after things which they perceive as an overwhelming threat.
Even if you randomly stumble across a bear in the wild, it's NOT going to always, automatically attack you. And, even if it did, maybe Morale is important --- a bear which is getting really hurt might turn tail and run, or it might go nuts and become even stronger with rage...
If you're talking humanoid creatures, I think the unprovoked attack even LESS likely. Would a Goblin REALLY take on a party of uber-powerful heroes? If so, why? Even the dumbest Goblin would eventually see the dead bodies of its fellow and say "Maybe I WON'T attack that party who just turned Chuck into mincemeat within 3 seconds."
It might make an interesting plot point when a Goblin DOES continue fighting...
I also like bgillisp's idea of having an Optional Boss who "controls" the creatures in the area, if such a thing makes sense. Of course, you COULD reverse this --- if you kill the Thieve's Guild Head, his replacement puts a huge bounty on your head, causing a LOT more random encounters with Thieves and their ilk. In fact, I really like that idea as a way to ADD challenge to an area, particularly if it's not too much of a challenge when the player first went through.
So, yes, you need to backtrack across the Badlands --- oh, but now you have Thieves after your head, so it's a bit more challenging than it was with just the Goblins. This keeps the challenge at least constant for a higher leveled and better equipped party.
To the point, backtracking for loot can be really good if it's a great storyline reason and item reward, but it's NOT good to backtrack just to pad the game's length --- some modern AAA games pull this stunt to extend their playtime.