Yeah, Dickens is almost infamous for his by-the-word payments -- but another important reason why many of novels of the 18th and 19th centuries are so long is because they were meant to fill time. They were read by people who didn't have radio, TV or the Internet and had no fast travel. They had a lot of spare time to fill, especially on days-long (or months-long, sometimes) journeys, and novels were meant to help bridge the gap between now and then.
This is also why many of the novels of that period are explicitly political, or filled with seemingly out-of-place essays -- novels were as much a mode of base communication as they were about story-telling. Austen and Bronte were both popular as part of early, pre-feminist women's movements; Hugo's clearly a French Republican, and many of his essays celebrate and elevate democracy and the working class. The personal experiences he litters "Les Mis" with, for example, are often history lessons in the evils of totalitarian governments (not that he had that word to describe them).
I'm also of the mind that, as the novel progressed and authors sought to outdo their predecessors, they began whittling away at bits extraneous to the core message. I think that's generally an improvement -- it's better to say something important in a few words than a dozen interesting things in a 1,000. The "first" novel, "Don Quixote," is eminently readable, but it's massive -- there simply wasn't a concept of "this is too much." It was a new art form, the long-form written story, and the value of limiting one's self wasn't yet apparent.
