Latent heat certainly flows from cold to hot ... what manner of stupidity says it doesn't? ... well, er, "Classic Physics" is the name we give to this kind of stupidity ... so don't feel bad, I'm just as stupid ... just a Classical kind of guy living in the Modern world ...
Well all of physics is that "manner of stupidity". It is a simplification of the universe, to reduce it to a level of human comprehension.
The whole hub-bub over "Modern Physics" is that it explains better how thermodynamics works ... and so we try to avoid using the word "heat" when we mean energy ... and we have to use the word "average" just about everyplace ...
Well it's kind of hard to avoid using the word "heat" when the subject is global "warming", and we are told that the ground is being "heated" because it is absorbing IR energy that's "re-radiated" from CO2. And if the atmospheric CO2 level goes up, the ground will get even hotter, because more IR energy will get "re-radiated" by the CO2 back towards the ground.
I agree, we are measuring averages. AGW theory concerns the macro scale, not individual molecules.
I can't glibly discard the rules of thermodynamics- the atmosphere is "on average" cooler than the ground, and the heat transfer arrow points outwards.
The source of the "heat" in the atmosphere
is the ground. The ground receives it from the sun in the form of visible light (plus a little bit from geothermal and atomic processes), and radiates it away as IR (heat energy).
Some the IR energy is absorbed by water vapor and other "greenhouse gases", and that raises the average temperature of the atmosphere and keeps us from freezing to death.
So the ground heats up during the day and it cools down at night. It's still radiating energy in the day, but it is absorbing energy at a higher rate than it's radiating it away, so it gets warmer. At night it reverses, and the ground gets cooler.
The net change is zero. All the energy absorbed in the day must radiate away at night, or we fry. Conservation of energy.