I first truly studied that account when I was also majoring in journalism and covering local politics. For me, the two "political parties" of the time leaped forward. First, there was no separation of Church and State. Second, the two factions seemed to be those who aligned themselves with the Priests, and those who aligned themselves with the King and his men.
The Priests remembered the days when the Chosen People, The People Set Apart, were attacked and killed when they were hungry, exhausted, and were looking for sustenance. They also saw that when The People Set Apart began to mix with other cultures, they readily adopted and accepted the ways of that culture (including idol worship). To keep the people pure of heart, the priests determined every last person and item of those who were not with them had to be exterminated. They determined this had to be the will of God in order to keep the people Set Apart.
Meanwhile, the King (Saul), wanted a more peaceable arrangement with the people already settled in the region. He did what he could to spare them, and this infuriated the Priests who then plotted (and succeeded) in having David replace him. The victors, of course, got to write history, and David was praised while Saul vilified as the one who disobeyed God. I never liked David. He wasn't even the one who slew Goliath, but he took (or got) credit for it.
God works with what happens, with whatever man does. While Jews to this day have a much higher opinion of Jacob and David, I have always had a soft place in my heart for Esau and Saul--one the rightful heir; the other the rightful king. I wonder what God might have worked through these two if humans who were in power here on earth had actually trusted God rather than their own egos and wishes.
Never lose sight of the fact that the Bible was written by man even though it was inspired by God. The Bible tells us more how mankind has reacted to what God started (continuing in disobedience) while God picks up the pieces and moves us on. We all know in our hearts there should have been no such rape and massacre, therefore the proper question is not, "Why did God do it?" but "Why did we do it?" Note, we always seem to reassure ourselves it is what God would want.
In fairness to Jewish belief: It is also quite true that what the Hebrews did to those people was only about a tenth of what had been done to them. They considered it a well deserved punishment that was merciful when compared to what had been done to them. In fact, look what the Nazis did to the Jews closer to our time. It would be like us, today, taking great exception had a Jew dared to slap or kick a Nazi. So I also keep in mind it may well have been for the very best for Jacob and David being brought forward as leaders. We don't know everything, indeed we know very little. Esau and Saul could have led humanity to unmitigated disaster, while we know Jacob and David (though imperfect) brought about many blessings.