No, of course it's not the stretching or breathing or things like that that are the problem. No one has said that. Yoga is more than just physical, there's a spiritual aspect as well, where they literally chant to false gods. The very meaning of the word yoga is to unite or yoke....but to who?
Obviously people are free to do what they want, and no one is stopping you. But the bible is very clear that Christians are not supposed to mix religions or mix with spiritual practices that are rooted in false beliefs.
When I have more time I'll post some videos on this topic. There are plenty of people who actually KNOW (far more than I do) about this topic, who have been on both sides and understand that yoga is much more than just a physical thing, it's spiritual as well, but not everything "spiritual" is good. The bible says to test the spirits, and there's a reason for that. I don't know if you profess to be a Christian, but anyone who isn't Christian, I wouldn't expect them to understand.