Not really, neither side wins 100% or even 50% most of the time. Sides that want it all are going to be on the losing side of history, when is the last time you seen a segregated public school? I think St. Louis area was one of the last hold outs, iirc. Or a whites only fountain?
Did the bakers custom decorate any cakes for anyone in the past?
How would you feel if you mortgage was sold to bank primarily owned by Muslims and they started adding high dollar fees to payments of non Muslim note holders, or started jacking your due date around each month so you fall behind in payments. Where does it stop when personal beliefs are allowed to dictate what services are provided to people by who they are.
I don't feel I need to explain why these are somewhat different situations (to put it mildly) than demanding that a baker decorate your cake with something offensive.
If it helps you, though, I wouldn't use force (as you are so ready to do) to make some small business do what I want them to do. If a Muslim wanted to open a bank that caters to other Muslims, and isn't so friendly to the non-Muslims - well, it's their bank. I don't know much about banking regulations, though. I suspect it would not be as simple as that. If they start taking over loans from unsuspecting infidels, yeah, that would be quite a bit different than just not wanting to serve pork sandwiches. Even then, the home-owner can get his loan refinanced, so I'm not as ready to start cracking down on the Muslims as you are.
Violating personal property rights and free association rights looked a whole lot different, when black people were having a hard time finding a decent place to eat, or a room to sleep in. Perhaps it was excusable then. Today, forcing business owners to serve anyone, in any fashion they want? No, there's no excuse for it. It can't be justified.