I've had good results with RAID Wasp spray. Stuff's a neurotoxin, they go down quick.
Alternatively, if you want something that doesn't stink and is toxic, orange oil-based sprays will also kill them, though I've never tried it on a large nest. There's a chemical in the peel (the name of which I can't recall) that basically causes their exoskeleton to lose its ability to hold in water, which results in dehydration and death fairly quickly (almost as fast as the RAID stuff). It may also interfere with their ability to breath. Not sure how well it'd work on a large nest, though.
I used to keep a bottle of this stuff in my bathroom, and it served double duty on the occasional mud dobber or paper wasp nest.
You're thinking of
D-Limonene. It's non-toxic to the point it's used as a flavoring/odorant for foods. It is a solvent in the terpine family, (Like turpentine) and as such, it'll dissolve the oils/fats/waxes that chitinous exoskeleton arthropods use to keep their "insides in and the outside out".
A D-Limonene dip or bath is a really good solution for a flea infested puppy or kitten that's too young for any of the other products out there too.
It's essentially the same solution as the brake cleaner I joked about earlier. Same kill mechanism.
