IMHO, most of the people who vote in the primaries are the people that are more knowledgable are care the most about the future. Certainly there are whackos and crackpots voting in the primaries, but I wonder if that percentage would change much if more people voted. Would they be more knowledgable, or more easily led? Some have opined that people vote for a person in the other party that they believe their candidate can defeat in November. While that may not be statistically relevant, I am somewhat unconvinced that your solution would actually work.
I think the greater problem is too many whackos and crackpots that are on the ballot rather than the voters themselves. But I am not sure how to resolve that problem, today's political environment is so toxic that it is no wonder good people avoid politics.