In a Libertarian system, would the USA be barred from forming foreign alliances?
Or are you just saying alliances can be formed as long as Libertarian-types approve of such alliances?
Barred? Of course not. There's really nothing in the constitution that forbids the government from forming an alliance. The founders warned of such a thing, but did not place any constitutional restriction on it. It could be justified by the common defense enumerated power, I suppose.
But that doesn't mean we support it as a foreign policy.
Taking sides creates unintended consequences, like creating enemies. We defend the borders and ports of foreign countries more strictly than we defend our own.