Because the legislation affects everyone, not just your district (that whole commerce clause thing).
At what point does a campaign contribution become a bribe: $.01.
People give money to candidates for one of two reasons: I like the way you vote on my issues, so here's some cash to keep voting that way; or 2) I don't like the way you vote on my issues, but if I give you money that may help change your mind.
I didn't miss your part about only the people in the district being able to contribute. There are a couple of problems.
1) I pointed out that Acme Corp (or Union) could offer to give $105 or more to everyone in the district that gives $100 their candidate. (sorry, I'm from Chicago, they could also have dead people contributing.)
2) What about different voter populations. For example Montana has one Rep. There are 974,000 people in Montana. But there are 13,000,000 in Illinois with 19 Congressional districts. Conceivably the people running for Rep in Montana could have up to $97,400,00 to split to spend on their campaigns, while those running in Illinois would only have $68,421,053 to divide to spend on their campaign. And you better believe that they'd be running to the nearest Federal Courthouse to get that "evened out".
Any restriction on contributions is a limit on the 1A. Get the money out of .gov and the there won't be any money in politics. You don't spend money on something that doesn't affect you. Why would Acme corp (or Union) donate to any politician when there's nothing they can do either for or against them.