I think my S4 is about the perfect size.
I don't have any gripes about the size choices that are available. My gripe is that we have no choices in form factor whatsoever except for iPhone clones. The BlackBerry-style, clamshell, and slider formats just up and died. I would kill for a phone the size of my S4, but with a slider or clamshell keyboard. Years of engineering have reduced smartphone thickness to ridiculous levels, while removing features and not increasing battery capacity. If they would have been too fat with a physical keyboard before, they should be doable in a thickness no thicker than how most phones end up being anyway once people put them in a case that is required so you can actually grab it.
my perfect phone is the size of an S4 or maybe even iPhone 4, with SD slot, slider keyboard, USB-C/OTG, and swappable batteries with a small onboard battery so batteries can be changed without rebooting. Oh, and basical functionality like the headphone jack intact.