In politics, any time you do something even remotely shifty, or strange, people will try to have a go at you for it. It's not even so much that anybody actually gives a crap about these antics so much as it's a sports game.
Frankly, as long the good Congressman is not insane in the medical sense, he should stay. If anybody doesn't like it, they should elect a new one.
...oh god the next election campaign in this guy's district is going to be gold.