I really enjoy when there are bonus exploration goodies, however, I'd be careful with them. If you plan on doing lots of these, I'd try to make sure that the "reward" is proportional to the effort required to get it. As a player, I'd be a little irritated if I wound my way down a multi-level dungeon and killed a hyper boss only to unlock a lore object saying "This one dude did this one thing once" at the end. Now, for that same dungeon, if I unlock a hidden skill or an optional character, that'd be a different story.
One other thing I've noticed is that if you like to do stuff like this, consistency is really, really important. If you establish early on in the game that clicking on a bookshelf yields no results, you can hardly expect the player to check them later in the game for goodies. This is obviously a simplistic example, but it applies for most exploration tidbits.
And as for optional dungeons, YES! More content (As long as it's not just recycled maps from a different dungeon) is always welcome in my book if it doesn't subtract from the experience. If you're worried about this sort of thing being tied in or un-engaging, try making an event trigger when you enter the optional dungeon (ie. You enter, and suddenly get trapped, having to work your way out. Or a thief steals a key item and you have to get it back, etc, etc.) This way it's fun, unique content that flows nicely into the game without actually being required for progression. Heck, if you get really frisky and you like lore, you could make switches and variables trigger in these optional dungeons that alter main story conversations later in the game, for an added goody on top of whatever they found there initially.