Several issues Jill. I found in my research for my paper on the subject that first incumbency is an almost impossible advantage to overcome. Many incumbent Congressmen run unopposed. When they are opposed, their war chests overflow with cash and they can/do outspend opponents at impossible rates. There are typically only a few ways an incumbent can be beaten: sometimes (although clearly not always), if they have been convicted of a crime; if they've been redistricted to run against another incumbent or they are unlucky enough to have a self-funded multimillionaire run against them.
This situation is compounded by people not understanding what they are voting for. So many people, usually with a bit of smug superiority in their voices, say, "I don't vote for party, I vote for the person I think can do the best job." Which is total bullshit. Oh, they may vote that way, but the people they elect don't. I had a beautiful chart in my paper that laid out the percentage of party line voting done by t members of both parties in Congress for the last 50 years. It was greater then 85% on average for both parties. So if you think you are voting for some independent individual, you/they are just fooling themselves.
All that being said, people generally like to point out how corrupt Congress is while saying how great their Congressman is. Back to Tip O'Neil's oft quoted saw that all politics is local. If a Congressman uses his franking privilege well enough and brings those road projects home, he'll win with 60%+ of the vote year in and year out.
The star of my paper was a Congressman from Mississippi that had been elected before Pearl Harbor and was still serving in 1992.

You can't tell me he was the only guy worthy of that seat in the 50 years he held it. As I said earlier, placing term limits on congress is unconstitutional, however I do believe after some period, they should not have ballot access. If they still get elected, fine.