Wasp spray works on them, but there are usually too many to kill with one can. A lot of foaming sprays will plug up the hole and keep killing those than touch the surface. I would fill the hole with foam and skedaddle.
Walmart sells stuff called Camp Fogger than also kills wasps and bees and stuff. It is more of a fogging spray that can blanket an area. If you can get it in the hole, it might do more damage. A bug bomb near the hole might also do a good job.
If you are calm and don't move much, you can get upwind and spray the area where they are entering. I've done it with bees a couple times at work.
Other than that, call an exterminator.