There once was an master cabinetmaker, whose great works were higly sought from all across the land. One day, his apprentice noticed the cabinetmaker spending just as much time sanding the back corners of the drawer as he did the front.
"Why do you do that, sir?" the apprentice asked. "No one will see them. No one will even know."
"But I will know," the master cabinetmaker said.
The same could be said about your writing. How much are you smoothing all of the edges? Are you making the dialogue to the best of your ability? Are you bringing a richness to both the story and the characters? What are you holding back?
In my alternative military history novel (boy, I should really share the name for this instead of always referring to it in this way), I came to a chapter where I delve into the life of one of the supporting characters. As I did, there was so much I discovered that I want to enhance it as I continue to write the rest of the book.
Does this always need to be done? Do you need to share all of the worldbuilding you did for your book? Certainly not. However, take one of your favorite books and watch where the author does this, sharing a richness to their created world. I'm currently re-reading George R. R. Martin's Game of Thrones and loving where he brings this rich world to life. He does it in sublte ways, giving little hints to the past. The same could be said of Tolkien's Lord of the Rings.
Does everything need to be included in your story? Certainly not. But adding even a few details here and there will make your world more alive, and your readers will grow more and more in love with it. If, in the end, it doesn't work, you can always hit the DELETE key.