Then the whole idea of backradiation is out the window. Energy won't move spontaneously from a low temperature area to a high temperature area.
Now let's explore your theory.
Imagine a binary star system with a red star and a blue star. The blue star is much hotter than the red star.
The red star emits photons in the direction of the blue star. Under your theory, the blue star can't absorb those photons, because it's hotter. So just what happens to those photons and all the energy they carry?
You are assuming that the red star must emit photons in the direction of the blue star. There is no law that says that an object must radiate in every possible direction. The second law says It is
not possible for heat to flow from a colder body to a warmer body without any work having been done to accomplish this flow. It further states that
energy will not flow spontaneously from a low temperature object to a higher temperature object.
I don't know why this is so difficult for you to grasp. Suppose the blue star absorbed energy from the red star...that would cause the blue star to burn hotter which would mean that it would radiate more which would mean that the red star would absorb more energy and become hotter and radiate more energy to the blue star which would absorb it and become hotter and then radiate more which the red star would absorb and become even hotter and radiate more which the blue star would absorb and become hotter and radiate more which the red star would absorb....
See where this is going? If things happened as you seem to believe they do, then you would have an infinite positive feedback loop where eventually both stars would become infinitely hot. From observation, we know that this doesn't happen in bianary systems so obviously the red star isn't radiating in the specific direction from which it is receiving energy from the blue star.
Again, no law says that energy must radiate from an object in all directions. Saying that a cooler object will not radiate towards a warmer object is no different than saying that a marble won't spontaneously roll up hill, or that the atmosphere at 14 psi won't cause a punctured tire with 32 psi inside to increase pressure. The second law predicts that a marble will roll down an incline, and that air will flow out of the tire till such time as the pressure inside and outside reach equilibrium...the second law also predicts that energy won't move from the cooler star to the warmer star.