The best explanation I have read about the incident at the Temple was someone using the floor plan of the area. It was a huge complex and in the outer region was the area were money changes occurred and people brought in animals to be sacrificed. We can imagine how busy and crowded this area was just before Passover!
There was another area closer to interior the Temple (I forget what it was called) but perhaps consider it a foyer to a holy place. The theory was, that is so often the case, the rich did not want the bother of waiting among the noisy crowd, and--for a price--these particular money changers offered the opportunity to get ahead of the game. Animals, of course, were apt to step a foot or two into the holy interior--or at the very least, the noise would disturb the quiet of a prayerful place.
So Jesus drove these "entrepreneurs" out of an area they had no right to be to begin with. The rich and powerful were angry and aggravated, the most people that day probably didn't even notice the little kerfuffle occurring in that more out-of-the-way place. They were where they were supposed to be, doing what they were supposed to be doing.
Pilgrims also had to BUY sacrifices and they couldn't pay for them with Roman coins or foreign coins.. They had to be exchanged for Shekels.