I don't have a problem with what he said. He basically said he would love to meet with them, but they won't meet. He addressed the question and put the questioner on the defensive instead of her trying to press him to commit to something more than that.
His answer was far far better than what past Repub Presidents would have said which would be some noncommittal nonsense or some version of "I am not racist, I have lots of black friends."