I started making my first games when I was around 12 years old, with the Warcraft 3 World Editor program. None of those games ever made it out to public release (except for one that was extremely buggy), but I garnered the experience of how I should go about designing levels, mechanics and quest-lines. So from a young age, I had a sort-of rough idea of where I wanted to future career. But as never disregard the experiences that you've gotten already; they are still incredibly valuable. Since a Video Game Designer covers almost everything about games, they're expected to have a wealth of hands-on experience with different positions in the industry. Art, Programming, Sound, Writing, Producing, Marketing, etc.
It's never too late to start building these skills. ; >