Educamate me - computer failures and back-up systems

[vaskidmark]

http://www2.timesdispatch.com/news/news/2010/aug/27/8/vita27-ar-475161/

Taxpayers are inconvenienced, welfare and unemployment payments likely to be (almost a guarantee at this point) delayed, operations of government unpossible, workers are not furloughed but cannot do basics of job (including playing solitaire).  "Twenty-six of more than 80 state agencies were hit by the shutdown, including the office of Gov. Bob McDonnell."  With limited exception those 26 agencies are going to be non-functional until after mid-day Monday because VITA will have to wait for state employees to come in, turn on their machines, and then test the system to see if the fixes they made over the weekend actually did anything.

If everything works, welfare and unemployment checks will not go out until next Wednesday, which means they will not hit the mailboxes until Friday.  Government checks are subject to the same hold as all other checks - funds will not available for at least 24 hours at the better banks, not till after Labor Day at some others.  Electronic transfers will not go out until Wednesday, which means they may not be available anywhere between Thursday and next Tuesday after the Labor Day holiday (freaking crappy bank rules!).  Folks whose DLs expire at the end of the month will need to stand in line to get new ones, which means business interruptions and lost wages.


And the head of the contracted computer service outfit says:

Nixon also said that the interruption was of insufficient magnitude to activate a backup system at a duplicate computer center in Russell County, in Southwest Virginia.

So - oh great computer-stuff-understanding gurus, ess-plain to me why after one full business day, or two full business days, they do not switch on the back-up system.

I mean, besides the apocryphal and almost-guaranteed "if they upgrade or fix the computers they will be down for at least a week and then not run right for the next month" that has been the norm since before I was first employed by the state back in the '70's.

stay safe.

[Stetson]

I'm not positive, but I believe all welfare 'checks' are not just direct deposited to a Chase bank card, or the like.  There might be a one or 2 day delay.

Why not turn on the backup...CYA for either failure to complete setups or regular maintenance.(in my experience)

[Tallpine]

They could just play Solitaire with those paper card thingies   

[vaskidmark]

They could just play Solitaire with those paper card thingies   

No can do!  Gambling paraphenalia is verboten in state workspace.

Posted on: August 27, 2010, 10:52:35 PMPosted by: Stetson 

I'm not positive, but I believe all welfare 'checks' are not just direct deposited to a Chase bank card, or the like.  There might be a one or 2 day delay.

Perzakly!  Some folks do not have bank accounts, and even if they did the computer system failure within Dept. of Social Services cannot pull up memory to see who gets welfare and how much.  Same problem with the Food Stamp machines, which spit info to an account for your plastic card so you won't be stigmatized by having to use those horrible paper coupons.

Why not turn on the backup...CYA for either failure to complete setups or regular maintenance.(in my experience)

Hey!  I asked first.

stay safe.

[RocketMan]

It may have just been case of, "It will take a day to switch over to the backup system.  It would also take a day to fix the problem system.  Why not just fix the problem system?"
And then it all went to hell from there.

[vaskidmark]

Today's news report http://www2.timesdispatch.com/news/news/2010/aug/30/vitagat30-ar-479496/ :

The Virginia Information Technologies Agency says all but about three agencies' systems have been restored. Work continues on the remaining systems.

My confidence level has not increased.  Could it be because VITA is reporting "all but about three"?

Just how difficult is it for the post-Sesame Street crowd to decide if it is one, or two, or three, or more than three?

At least DMV is sticking fast to the requirement to bring proof of citizenship/legal presence to renew an expired DL/ID card.

The next act will be at least as exciting - watching the blame/excuse cha-cha-cha and the determination of fines to be paid and the expenses to be charged to offset those fines.

Something tells me this must have been a legitimate crisis, as there are no major political upheavals on the program for mid-term elections in Virginia.

stay safe.

[lee n. field]

Wish there was a little more information about the exact nature of the computer problem.

[GigaBuist]

Wish there was a little more information about the exact nature of the computer problem.

My guess would be a very large SAN shared by multiple agencies went down.

Then I read the article.

Oh, look, EMC was involved!  Hang on, let me get my shocked face mask out.

[sanglant]

Today's news report http://www2.timesdispatch.com/news/news/2010/aug/30/vitagat30-ar-479496/ :
My confidence level has not increased.  Could it be because VITA is reporting "all but about three"?

Just how difficult is it for the post-Sesame Street crowd to decide if it is one, or two, or three, or more than three?

At least DMV is sticking fast to the requirement to bring proof of citizenship/legal presence to renew an expired DL/ID card.

The next act will be at least as exciting - watching the blame/excuse cha-cha-cha and the determination of fines to be paid and the expenses to be charged to offset those fines.

Something tells me this must have been a legitimate crisis, as there are no major political upheavals on the program for mid-term elections in Virginia.

stay safe.

that scary vamp dude stole all our numbers and counting, we are lucky to be able to sleep at all.

[RevDisk]

My guess would be a very large SAN shared by multiple agencies went down.

Then I read the article.

Oh, look, EMC was involved!  Hang on, let me get my shocked face mask out.

ROFL, SAN crash?   I bet one geek cred that they tried to failover to their replicated SAN, and found their replicated SAN was dead as well.   I bet another geek cred that they never tested their DR procedures.

[vaskidmark]

I bet another geek cred that they never tested their DR procedures.

Test?  Test something to see if it worls before putting it into the system?

You have got to be kidding me!  Test protocol around VITA seems to be "Let's throw this in and see what happens," followed in a much lower voice by "_{If something goes wrong we can bill hours for fixing it."}

State employees are literally afraid every time it is announced that the system is going to be upgraded or repaired, as it invariably means the system goes down for a week and then is wonky for another 2 to 4 weeks.  I tell you, it is a royal pain trying to keep your Evil-bay auctions operational when your computer network is screwed up.  And on top of that, it can be difficult to get state business done.

stay safe.