For some people this kind of thing is a labor of love, but if it's not for you, then my advice would be to keep it as small and abstract as possible.
Only include the essential buildings in your town, plus one 'centerpiece' of the town that gives it identity, and a few homes which are mostly locked (or don't even have doors) so you don't have to make the interiors.
Don't go overboard with NPCs; just make enough to make the town feel lived-in (and if you're really struggling to come up with anything that an NPC can say that's interesting, make them non-interactable, and be sure to have a visual way to designate interactable NPCs).
You can save a lot of mapping time for yourself (and walking time for the player) by Abstracting shops, inns, etc. as well - as soon as the player enters the door, present a Shop (etc.) Menu over a background, and as soon as the player leaves the menu, have them leave the shop, so that they never see an interior. For some types of games, you could even get away with abstracting the entire town! You do lose some placemaking in the process, but if that lets you focus on the things that really make your game special, it could be worth it.