I didn't realize at the time that it was potentially dangerous. I was 17 or something at the time.
It really depends on how fast the leak was, how big of a tank, and how well sealed the rooms were.
But yes, people have
died from this stuff - and normally scientist types who should know better, sadly enough. It's a really sneaky death. Pressures good, CO2 is being flushed, temperature's good. Air might or might not be unusually dry. Depends on what flushed out the O2.
Hypoxia - as experienced by mountain climbers and others in low pressure atmospheres are normally experiencing a raft of other issues at the same time, such as hypothermia and dehydration. Not to mention we're often talking about days of marginal pressure existence.
In normal temperatures, normal pressures and such, as oxygen levels drop you'll likely feel tired and wanting to take a nap... If the room isn't cleared or you hauled out quickly enough, it can lead to death. In extreme cases, you'll just fall unconscious in minutes, and be dead in a few more.
By the way, a nitrogen bottle for filling balloons sounds weird to me. I'm more used to helium tanks. Plain air works well enough if you don't need the balloon to float. I'd tend to think that you'd go with plain compressed air. That way all you need is an air compressor and appropriate pressure reducers so you don't insta-blow the balloons.