The Swiss didn't get invaded because there was no strategic reason to invade them. The Nazis also didn't invade San Marino, Monoco, Leichtenstein, Sweden, Spain, Turkey. As long as these neutral countries were willing to play ball on trade, Hitler left them alone.
That's the point, Bob, it was apocryphal. The quote is wrongly attributed to Admiral Yamamoto Isoroku, the guy who planned Pearl Harbor (and had no problem with attacking America).
Of course, attacking America was never Japan's plan. Japan's goal was to conquer China. They wanted to scoop up the European colonies in SE Asia for resources the US had cut off because they imposed embargos. If the US just let Japan have her way with China, there never would have been a war. (Of course, it's a good thing we did, because Japan's plan for China was more horrific than what Hitler was doing to Europe. But since the Chinese don't run Hollywood, you never hear about it.)