We all know displaying a Nazi swaztika is a sign of some sort of racist thought, but what about the 'Rebel flag?' Just saw a neighbor's when he invited me recently and I like the guy. If he's racist it'll be quite a shock to me. Is there then any benign non-racist motive to flying that flag?
To the mainstream, the rebel flag is nothing but a piece of cloth, not even as meaningful as a dinner napkin.
The Civil War decided that fact long ago.
Maybe it is a historically interesting item to him?
Why not ask him directly what the flag means to him?
Asking him makes a bad reason a potential sore point for both of us. If I don't ask, I simply wonder. If I ask, my concern may prove founded. Sometimes ignorance is bliss.
Bottom line he's never given me reason to suspect he's racist. If anything he's an exemplorary father and that's how I came to notice and befriend him. Of course I tend to get tunnel-vision and a bit obsessive when questions arise chewing on them until they're answered.

So what I decide tomorrow may differ from where I am today.

Today I don't care.
And I agree with your approach. If he is your neighbor, it is probably best not to sow any seeds of alienation.
I have been neighbors with some admitted racists, some for many years, who actually told me:
"You're different", at which I laughed to myself, and said "yeah, right".
At the end of the day, that neighbor, racist or not, could end up being the guy who helps you in a crisis, simply because he is nearby.
Let him fly his flag and think as he pleases. It's just a piece of cloth.