The problem always seems to be that we let the oppositon define the terms. Blacks don't like 'white people meet'. We allow them to define that as racist.
Any time you allow someone else to define the terms, racist, sexist, homo phobic, anti Corn Bread, or whatever, you just let them win the argument, and now you are in the position of the fellow who was asked "Do you still beat your wife". No way to wiggle out of that conundrum with any chance of success.