Some did most didn’t but the point is Jews were always there and even after the persecution they faced century after century they still kept returning back to their ancestral land… Many formed communities just outside the reach of the Romans or whoever it was that decided to unjustly treat them but they never forgot their ties to the land of Israel that is why every Passover we end the Seder by proclaiming next year in Jerusalem and in our prayer books we recite if we forgot thee oh Jerusalem etc etc…
Yes, they were a tiny minority. Some converted to Christianity and Islam. Other people lived there for 2000 years.