Author Topic: And We're Back! (Read 4838 times)

Ben · « **on:** November 25, 2015, 04:34:11 PM »

Hooray, we're back up!

From my understanding it was a cascade failure at Geovario that resulted in them having to backup everyone they served via tape, hence the delay after they replaced their hardware.

Fly320s · « **Reply #1 on:** November 25, 2015, 04:36:21 PM »

About time! I almost had to stop goofing off.

sanglant · « **Reply #2 on:** November 25, 2015, 04:43:43 PM »

Hardware failure, sure.

41magsnub · « **Reply #3 on:** November 25, 2015, 04:58:38 PM »

The hell do I pay you people for???

Ben · « **Reply #4 on:** November 25, 2015, 05:03:07 PM »

Quote from: 41magsnub on November 25, 2015, 04:58:38 PM

The hell do I pay you people for???

Everybody else went to the alpha site. Didn't we give you the ~~stargate address~~ URL? Oops.

Perd Hapley · « **Reply #5 on:** November 25, 2015, 05:25:45 PM »

I'm suing for loss of ~~companionship~~ rantship.

Ben · « **Reply #6 on:** November 25, 2015, 05:57:08 PM »

Looks like it's still going up and down. They did give an ETA of tomorrow morning before things were fixed, so they may be doing some testing still.

This is what they said happened:

Quote

Professional Edge / GeoVario Hosting, OFFLINE

Unfortunately as you are seeing this page, you have come during the time of the worst hardware failure in our company's history.

The website you are expecting, or the email account you were trying to reach went offline at roughly 4:45pm PST 11/23/2015; after we experienced a cascade hardware problem that affected our Storage Area Network disk arrays.

At this time, all static files, email accounts and most of our web and application servers are offline, database servers were not affected, but without application and web servers the content is not reachable - we are actively working to restore service, however between expiditing replacement parts and time to restore from backups we are expecting services to restore by roughly 6am PST, November 26th.

We are very aware this has been a huge problem for everyone, we apologize greatly and continue to work to restore service as quickly as possible.

Sincerely,
Professional Edge LLC

Quote

A little background; data is stored on mirrored disks which are then mirrored to another pair which in turn are mirrored to another pair and so on. In addition there are several backup disks which come in to play should one of the main disks fail.
And boy did it fail.

Apparently disc 2 of the primary pair failed so the server called up backup drive A and started rebuilding the mirror with drive 1 and drive A.
At the completion of the rebuild drive 1 failed. Drive A, now the primary drive of the primary pair, called up backup drive B and started rebuilding the mirror.
Midway through the second rebuild drive A failed taking out the primary mirrored pair.

Somewhere in this process one of the controller boards died as well. This and the death of the drives was likely brought on by cooling issues/failures within the data center

Some of you may be wondering why there was no alert of a predictive failure, good question.
Apparently the data center failed to do a firmware update some time back that would have sent the predictive failure alert for all three drive so there was no notice given until things started dying.

FedEx should be delivering the new hardware as I am typing this if they haven't already and it will all be installed as quickly as possible.
The real delay comes from restoring the terabytes of data from tape which takes hours and hours.

As I said, a series of unfortunate events but they are working on it as quickly as they can.

K Frame · « **Reply #7 on:** November 25, 2015, 06:10:41 PM »

You were gone?

HeroHog · « **Reply #8 on:** November 25, 2015, 07:10:50 PM »

Ah! Heat, the one factor that I didn't consider that could make RAIDED and Mirrored/redundant systems turn FUBAR.

cornerrow · « **Reply #9 on:** November 25, 2015, 07:34:44 PM »

So was it a hardware or software failure or upgrade that have cause the site to go down?

HeroHog · « **Reply #10 on:** November 25, 2015, 07:46:41 PM »

MikeB · « **Reply #11 on:** November 25, 2015, 07:57:24 PM »

Quote from: HeroHog on November 25, 2015, 07:10:50 PM

Ah! Heat, the one factor that I didn't consider that could make RAIDED and Mirrored/redundant systems turn FUBAR.

Heat will do fun things to computer hardware. It is generally much more tolerant than it used to be, but still not good. I've driven into the local facility for one of the Data Centers I manage or started remotely shutting down hardware in some others I remotely manage a few times in the middle of the night when alarms had my cell going crazy from loss of cooling.

RocketMan · « **Reply #12 on:** November 25, 2015, 08:18:23 PM »

We had a cooling loss in our data center last week. Minimum of 120Â° in the room when I first walked in. We had one server do an auto shutdown due to the heat. We managed to get things cooled down before we lost any more.
But at least it was a dry heat.

K Frame · « **Reply #13 on:** November 25, 2015, 08:46:00 PM »

Jamis spilled his Leprujuice on the server stack.

seeker_two · « **Reply #14 on:** November 25, 2015, 09:40:13 PM »

Quote from: Mike Irwin on November 25, 2015, 08:46:00 PM

Jamis spilled his Leprujuice on the server stack.

It's stuff like this that made us kick him out of Texas.....

Battle Monkey of Zardoz · « **Reply #15 on:** November 25, 2015, 10:10:34 PM »

Quote from: Mike Irwin on November 25, 2015, 08:46:00 PM

Jamis spilled his Leprujuice on the server stack.

Is that a sexual masturbation technique?

Blakenzy · « **Reply #16 on:** November 26, 2015, 04:33:19 PM »

Ah, almost had an APS withdrawal death, didn't ya?

I blame the NSA. Sorry Fistful. (I'll shift some blame your way next time)

Hawkmoon · « **Reply #17 on:** November 26, 2015, 05:23:37 PM »

Quote from: Ben on November 25, 2015, 05:03:07 PM

Everybody else went to the alpha site. Didn't we give you the ~~stargate address~~ URL? Oops.

Ain't nobody ever told me about no "alpha site."

Now I'm feelin' left out. Nobody never tells me nuffin'.

RocketMan · « **Reply #18 on:** November 26, 2015, 08:25:36 PM »

And we're back yet again. Hopefully we'll stay up for a while this time.
Anybody keeping track of Jamis and Fistful? One of them has got to be the guilty party.

K Frame · « **Reply #19 on:** November 26, 2015, 09:00:40 PM »

Quote from: Battle Monkey of Zardoz on November 25, 2015, 10:10:34 PM

Is that a sexual masturbation technique?

For him, yes.

castle key · « **Reply #20 on:** November 27, 2015, 05:50:54 AM »

The Clintons managed to keep a server system going just fine in a bathroom in New York.

Hawkmoon · « **Reply #21 on:** November 27, 2015, 06:14:10 AM »

Quote from: castle key on November 27, 2015, 05:50:54 AM

The Clintons managed to keep a server system going just fine in a bathroom in New York.

No, the Clinton server was in their basement in New York. The bathroom was, IIRC, in Colorado, at the company they eventually hired to run the thing. Then there was the place in New Jersey, and the one in Connecticut ...

It seems it takes a village to run a server.

T.O.M. · « **Reply #22 on:** November 27, 2015, 08:26:12 AM »

I'm in New York state with my liberal in-laws who have been making fun of all of the GOP candidates...

I missed you all...

230RN · « **Reply #23 on:** November 27, 2015, 01:16:43 PM »

Quote from: Chris on November 27, 2015, 08:26:12 AM

I'm in New York state with my liberal in-laws who have been making fun of all of the GOP candidates...

Frustrating, no? Keeping one's mouth shut in circumstances like that is a cross to be borne.

Angel Eyes · « **Reply #24 on:** November 28, 2015, 08:49:16 PM »

Calguns was affected as well. It's finally back up, but very slow.

http://www.calguns.net/calgunforum/showthread.php?t=1136807

They do have a backup site, which seems unusual: http://www.calguns.org/

News:

Author Topic: And We're Back! (Read 4838 times)