So their presence is not a meaningful factor in terms of this discussion.
81% of the US population lives in cities, and of the other 19%, the majority do not live in the far off boonies like the individuals you described.
A lot of those "cities" are fairly small. I don't think your robo-car is going to work very well in Mayberry, or even Billlings (pop ~100K).
Let's see, I'm going to drive to the pet store, and then to the post office, and then to Walmart, and then to the feed store, and then to Walmart, and then to the gas station ...
So if
all new cars/suvs/pickups have no steering wheels, just how do you expect that 20% to get around
Or maybe you were planning on rounding them all up .....