Don't worry, Trump has vowed to undo everything that has been done to protect them. (both numbers in that meme are complete BS btw)
The thing that saved them was the Marine Mammal Protection Act of 1972 which prevented powered vessels from pursuing them for hunting purposes. They were being hunted to extinction. The weather has nothing to do with them, or their numbers. They survived the last interglacial when it was over 10 degrees warmer up there and it was truly ice free year round. Those suckers can swim a LOOOOOONG way. Probably why they are classified the same as seals and whales. Not that a little thing like a fact will change your idiotic mind.