Looks like there have already been a great number of responses - which I didn't read all of them, so sorry if my opinion is just parroting what others have said already or if someone has brought up something similar:
In my opinion, any negative for players not being able to adjust the difficulty on the fly is pretty much outweighed by the positive of being able to.
I believe it was Mark Brown who had said: 'what does the difficulty even mean?' When starting a new game.
You can provide cute descriptions that try to inform the players of what to expect, but they truly won't know how the game actually feels until they have played it. So allowing them to change difficulty midway can help them make a more informed decision about the difficulty level they want without having to completely restart the game.
A bad example of preventing players from changing difficulty I recently experienced was from Bioshock: Infinite and it's DLC, Burial at Sea.
I had picked up the whole trilogy for dirt cheap during the winter sale and found playing through Bioshock 2 relatively easy for my skill level on normal difficulty.
So when I moved on to Bioshock: Infinite, I started the game on Hard thinking that it would provide a better challenge and therefore more fun to play. It was still relatively easy, but better than my experience with Bioshock 2.
So when I moved on to the DLC, Burial at Sea...this is where problems happened. Because my experience with the main game was it being a pretty easy even on the hardest difficulty, I set the game to hard for the DLC where now suddenly hard meant something entirely different for the DLC than the main game. The balance was completely different and felt more like a test of patience than an actual challenge. Were I able to adjust the difficulty midway, I would have lowered it, unfortunately I was stuck with either: just giving up, struggling through, or restarting from the beginning with a lower difficulty. I chose the first option: give up and quit. I never finished Burial at Sea.
So that's all to say, even within the same franchise, or even the very same game, difficulty means nothing until the player is actually in the moment experiencing it.
Some games I've seen at the start ask you to pick a difficulty, then moments after the tutorial level ask if you would like to raise or lower the difficulty, but I also still find this as a bad compromise for the problem because - depending on your game's complexity - the player likely hasn't seen all the mechanics in your game and therefore won't get a real taste of it's difficulty until, well...probably much later than just the tutorial.
I think I saw someone mention in this thread that RPG's have leveling to offset difficulty. Theoretically, if the game is too easy, just don't fight, and if the game is too hard, just grind. But this really is a fools errand.
If the game is too easy, so you just expect players not to fight, well...then you are basically telling your players "don't play the game." Regardless, exp is embedded into the core nature of most RPGs and therefore I doubt it would be possible to not level up at all - not without pulling some gymnastics to avoid it that I doubt most players would want to execute just so they can maintain a level of challenge they prefer.
Likewise, for those finding the challenge too difficult and you ask them to grind...well, basically you're asking them to stop progressing with your game's story so they can do a mindless activity. A little bit of grinding is probably not too bad, but depending on how severely crippled the player feels in comparison to your game's challenge level, the grinding could amount to a multi-hour chore. And what you have effectively done in both of these scenarios is given your player a chance to ask this question:
"Do I want to keep playing, or should I just play something else?" You have created a quit moment just like myself with Bioshock.
So, if it's not clear already: Always allow players to adjust the difficulty during game play.
On a similar note, but somewhat unrelated:
Don't be that guy who locks higher difficulties behind game completion. Every chance I can get to bring up Blue Reflection, I will, and this is one of those moments.
I love this game, but even on it's normal difficulty, the power your characters gain quickly over power any challenge, making every combat encounter mindless, and totally negating thoughtfully built equipment sets since the difficulty is so low there is no reason to waste the time and energy on carefully considering what you equip. The first game was like this, and if the second game had let me, I would have started the game on hard mode from the very start, unfortunately, in order to unlock hard mode, you have to complete this 60 hour JRPG and I couldn't be asked to replay another 60 hours just so I can finally enjoy a difficulty level that is more appropriate for my skill level. If you want to lock special difficulty modes like 1 hit KO modes like 'Dante Must Die', that's fine. But your basic Easy/Medium/Hard(maybe add a "very hard" in there too) should never be locked.