You say he should hold rallies with people who are not fans, yet there is nothing preventing you from going to his rallies. He doesn't pick his audiences, yet you won't go. Again, what are you afraid of? If you think he should hold rallies in front of non-fans and he doesn't pick his audiences, why are you not going? He could hold a rally in "your neighborhood"? First, do you have a stadium big enough? These are not the phonebooth sized "rallies" Quid Pro Joe holds, remember, these are real rallies with tens of thousands in attendance. Second, if his fans follow him around, you would be inundated with them and all five of you would simply be nullified by the enormous crowds.
Seriously, what are you afraid of?