We would routinely drill and tap stuff in the CNC shop by hand... Just hold it, spot it with some oil, and drop the z...
Of course, that was with stuff like 1/16" drills, and you wanted to feel things...
There was one drill press over in the corner, that originally ran of a jackshaft, but I think the motor for the long belt was about 10-20 ponies... That one could swing 2" drills. You'd start with a spotter, clamp it like no tomorrow, and work up to things... That was a Cincinnati, and had to be over 100 years old.
Biggest holes I routinely drilled were a little under 1.750" - they played nice with a t-shirt patched golf ball. Did those in the big lathe, and every so often it would grunt.