Y'all is only used to address more than one person unless you're just an inbred, backwoods,white trash, redneck, hick. And there are plenty of them down here. But there are also plain old Southern folk.
You come on over for dinner after church on Sunday. This addresses you only.
Y'all come on over for dinner after church on Sunday. This addresses you and your spouse or your spouse and kids.
All y'all come on over for dinner after church on Sunday. This addresses you, spouse, kids, and any inlaws, out of town guests, employees, etc., who might need to be included for whatever reason beyond who the invitation would normally include.
There are some Southern folk who might disagree with me, but they'll likely fall into that first category.