Keep telling yourself that. Early voting is present in many states, and typically, that involves voting anywhere within your county. This means that someone could vote using your info within your county during that period. If you didn't bother to vote, then catching the impostor is very unlikely. If you did vote, then catching the impostor is more likely.
At the same time, there are plenty of cases where people are able to vote more than once if they are registered in more than one state. That rarely gets caught, except when someone does something flagrant or when certain groups do investigations on the topic.