This is a clear cut question to quite a few people. And, their answer perhaps would be that you are asking a wrong question. Instead, they would suggest to ask, “Why does time not go backward?” There are quite a few physicists who believe that the arrow of time cannot be reversed; one of the contemporary ones among them would be Stephan Hawking. I think even the big dog aka Einstein believed that you cannot go back in time. According to him to travel back in time, you will have to travel faster than light and that is not possible.
But since the topic of this thread is in the form of a question (“Does Time go Backwards?”), we will make an impartial effort to explore the other possibility without being biased by the opinions of heavyweights.
Philosophers are a group of people who give the idea of traveling to past serious thoughts. There is a concept in philosophy called retrocausality which seeks to answer questions like, “Can the future affect the present?” Retrocausality takes into consideration the possibility of time travel particularly to the past. Although this is purely a thought experiment but it is not entirely based on imagination. On the contrary, it is based on some solid physics.
Two famous physicists who are strong supporters of retrocausality are John Wheeler and Richard Feynman.
Richard Feynman devised a diagram which is called Feynman Diagram to intuitively explain what is otherwise a very complex mathematical formula conceived by Paul Dirac which lead to discovery of positrons later. Feynman envisioned that positrons are nothing but electrons going back in time. This diagram became very popular among theoretical physicists who were working in the field of Quantum Field Theory.
Feynman diagram - Wikipedia the free encyclopedia
Coming back to the retrocausality, the only credible proponent of time travel to the past, is quite complex. I do not think anyone has touched it since Richard Feynman and John Wheeler. I personally think it should get a lot more attention than it is getting now not just for the sake of possibility of past time travel but to explain many strange phenomenon associated with Quantum Mechanics.