Would the fact that you're carrying have any influence on whether you call someone on their rude behavior? Suppose the situation escalates. Personally, I'd have no problem backing off, apologizing, or whatever to avoid pushing things to a violent stage. But who knows what the other person might do.
When I took my CCW class, the instructor said that it wasn't really a CCW class... it was "wussies 101." A lot of truth in that.
DD