Ask Archduke Ferdinand how well .32 works.
According to Wikipedia, it was a 380 (I always thought it was a 32 as well), but the real point is that he was shot in the jugular. A 22lr there would have worked as well.
I'd go up to at least the Kel-Tec P32. It's no bigger, and is almost into "real" caliber territory. Well, Europe actually sort of considered it a service caliber for a long time.
I had a P32 back in the day (long before the P3AT appeared). It was a nice little gun, accurate enough for its purpose and reliable for me. I worked up some HOT handloads using a 60gr Gold Dot and Alliant Power Pistol powder (based on data and experiments by Clark at THR). They chronographed at 1200fps IIRC and would penetrate 2.5 1gal jugs of water (the first two exploded, the third had a large entry hole) while expanding to .6" (no signs of fragmentation). I did have to put a much heavier spring in it.
After about a thousand rounds of hot European 32acp (RWS and such) and a couple hundred of my hot handloads, the frame cracked on the rail. I discovered it while detail cleaning it one day. It never malfunctioned. Presumably my hot handloads caused it, but the crack wasn't in a typical high-stress area. Kel-Tec replaced the frame and it was fine until I sold it a year or so later.
I now carry a S&W 37 38special. While it isn't as small as the P32 and only offers 5 shots, I don't have to worry about any particular load not functioning with the gun.
Chris