Junior dev manages to cripple/kill company on first day

[RevDisk]

https://np.reddit.com/r/cscareerquestions/comments/6ez8ag/accidentally_destroyed_production_database_on/
https://qz.com/999495/the-tech-world-is-rallying-around-a-young-developer-who-made-a-huge-embarrassing-mistake/

Obviously, not the kid's fault. CTO fired the kid probably to try to cover his own rear end.

Company managed to put read/write passwords in their setup guide.
They didn't bother helping a new dev set up their environment.
Scripts had no safety logic. This is akin to remove all shields or covers from any type of engine.
Backups? lolno.

I don't consider myself a real DBA, and I backup all production databases every 12 hours with Veeam to external big boxes of cheap hard drives. Two of them. Because. On each virtual server is the last two day's worth of backups. Oh, and we write weekly to tape, plus monthly tapes. I'd take more extensive procautions, but realistically anything that nukes all three backups probably means the end of the business anyways. We're talking about doing live replication across the country, mostly for performance rather than disaster management.

[Brad Johnson]

From his description it appears he did exactly as instructed. Sounds like someone else's head needs to roll.

Brad

[RevDisk]

From his description it appears he did exactly as instructed. Sounds like someone else's head needs to roll.

Brad

Sorta. He copied the values from the document rather than the credentials some other tool gave him.

[K Frame]

I can't believe the number of incredible screw ups this company allowed to happen before this poor guy hired on. He needs a lawyer, because he really should consider suing for wrongful termination.


I've seen some similar screw ups happen in the past at my company, but NEVER in a production environment.

We had one guy nuke our sandbox SharePoint site almost to glass because one of the admins gave him basically god rights and the guy had no real experience with SharePoint.

[AJ Dual]

If word leaks sideways in the org chart, about the litany and unbroken chain of process flaws and best-practices violations, the CTO is going to lose his job.

[griz]

A company that hires a new guy and gives him all the tools he needs to erase EVERYTHING on his first day with no supervision?  Sounds like they were a disaster waiting to happen and he pulled the trigger.

[AJ Dual]

Something similar happened here. Tech Services/Infrastructure/Ops manager allowed his team to have all three of the company Domain Controllers on only one of our twin EMC VNX clusters, and only one of the twin EMC SAN's that served them.

Despite some of his team warning him it wasn't a good idea, and that they should be spread out.

Apparently a particular run of HGST enterprise class drives had a problem where gas would "stick" to the platters and screw up the head's read/write ability. A few drives went down, more came online, except drives that had been idle had that problem worse.

And the HSGT supplied firmware "fix" to clean off the drives just wrecked them all en-masse. Oops.

Prod users didn't notice much, a few things ran slow as things flipped over to the mirrored disaster recovery datacenter, and along with the DC's, DNS got hosed for a bit... but IT sure knew. And we lost drives well past what the RAID could rebuild from, and with the damaged/missing virtual DC's, we lost the backup index too. So everything was backed up 10 ways come Sunday, but it was unsearchable.

Fortunately our team here is top notch despite oversights like that, and within 24 hours, 90% of the environment was restored. Within 72 hours 100% was. And that counted almost 250+ virtual servers for Stage, Test, and Dev which were set back up all manually, and only fed their data from backups.

EMC and HGST was onsite by the next day working with us, and while I'm not privy to the details, we got some even newer hardware out of whatever happened, and I don't think we paid, or paid much for it.

That guy was "soft fired", given 3 months to find something else and move on.

The team member most vocal about spreading the DC's and other critical infrastructure across both VNX's and both SAN's has his job now.

[K Frame]

A company that hires a new guy and gives him all the tools he needs to erase EVERYTHING on his first day with no supervision?  Sounds like they were a disaster waiting to happen and he pulled the trigger.

Without a doubt.

It sounds as if they're a non-ITIL, non-ISSO and/or non-CMMI shop.

That's basic data continuity/protection.

[K Frame]

"And the HSGT supplied firmware "fix" to clean off the drives just wrecked them all en-masse. Oops."

Reminds me of an incident with our local cable TV/high-speed internet provider maybe 10-12 years ago.

Cox Cable here in Northern Virginia had the high speed road runner internet service, and one evening it just up and died. Completely.

Turns out that Cox had rolled out a software upgrade that totally bricked something like 15,000 Toshiba cable modems. http://www.dslreports.com/shownews/Whoops-41697

They made everyone come in to one of their service centers to exchange modems for a newer model.

The CEO, Gary McCollum, was walking around and he and I got to chat. I had one very pointed question for him -- "Don't you guys have a test environment?"

Turns out that yeah, they have a test environment.

But what they didn't have was a test environment with full representation of all modems then in use. The modems that were killed by the upgrade weren't part of the test environment.

I was not particularly complimentary of that.

[K Frame]

Just shared this with a bunch of people on my team.

[Mannlicher]

RevDisk, not a word of that makes any sense at all.   laughing.  I guess I'm just not computer savvy.    :)

[K Frame]

RevDisk, not a word of that makes any sense at all.   laughing.  I guess I'm just not computer savvy.    :)

Let me dumb it down for you.

Company make boo boo.

CTO make boo boo.

Junior developer make boo boo that turns Company and CTO boo boos into HUGE OH MY SWEET BABY JEEBUS!

Simple, really.

[HeroHog]

Let me dumb it down for you.

Company make boo boo.

CTO make boo boo.

Junior developer make boo boo that turns Company and CTO boo boos into HUGE OH MY SWEET BABY JEEBUS!

Simple, really.

When I share this, I'm quoting this!

[RevDisk]

Mike is spot on!

Something similar happened here. Tech Services/Infrastructure/Ops manager allowed his team to have all three of the company Domain Controllers on only one of our twin EMC VNX clusters, and only one of the twin EMC SAN's that served them.

Despite some of his team warning him it wasn't a good idea, and that they should be spread out.

Apparently a particular run of HGST enterprise class drives had a problem where gas would "stick" to the platters and screw up the head's read/write ability. A few drives went down, more came online, except drives that had been idle had that problem worse.

And the HSGT supplied firmware "fix" to clean off the drives just wrecked them all en-masse. Oops.

Prod users didn't notice much, a few things ran slow as things flipped over to the mirrored disaster recovery datacenter, and along with the DC's, DNS got hosed for a bit... but IT sure knew. And we lost drives well past what the RAID could rebuild from, and with the damaged/missing virtual DC's, we lost the backup index too. So everything was backed up 10 ways come Sunday, but it was unsearchable.

Fortunately our team here is top notch despite oversights like that, and within 24 hours, 90% of the environment was restored. Within 72 hours 100% was. And that counted almost 250+ virtual servers for Stage, Test, and Dev which were set back up all manually, and only fed their data from backups.

EMC and HGST was onsite by the next day working with us, and while I'm not privy to the details, we got some even newer hardware out of whatever happened, and I don't think we paid, or paid much for it.

That guy was "soft fired", given 3 months to find something else and move on.

The team member most vocal about spreading the DC's and other critical infrastructure across both VNX's and both SAN's has his job now.

Queue whistling pic.

We had a crap physical DC. I happened to order a very nifty HP cube server with all the bells and whistles for a subsidiary that I suspected would not work out. Long with a bunch of other things I thought they would need... But if things didn't work out, would be very beneficial to the main office.

My sympathy for having VNX's. I have one at the moment, and support has been absolutely horrible. EMC is charging platinum pricing for bronze service, to be generous. I'm angling for a NimbleStorage array, but likely will end up with an HP 3Par. Let's see. So far, EMC yanked out the good SP when attempting to replace the bad SP. Lost data, recovered from tape. Another time we lost two disks in a single LUN, some corruption issues. Recovered from Veeam in like 20 minutes but unfun week. Memorial day weekend it took two days to get a replacement hard drive. No lost data, but getting the hot spare to unspare itself took some work.

I have a DC on each SAN, plus a physical. I mean, tossing 8 gigs of RAM and maybe 60 gigs of disk space is overkill. OS doesn't cost much, nothing if you're using DataCenter version.

[TechMan]

Sorta. He copied the values from the document rather than the credentials some other tool gave him.

I've seen that so many times, that I make sure to remove the credentials from screen shots.

[TechMan]

Mike is spot on!



Queue whistling pic.

We had a crap physical DC. I happened to order a very nifty HP cube server with all the bells and whistles for a subsidiary that I suspected would not work out. Long with a bunch of other things I thought they would need... But if things didn't work out, would be very beneficial to the main office.

My sympathy for having VNX's. I have one at the moment, and support has been absolutely horrible. EMC is charging platinum pricing for bronze service, to be generous. I'm angling for a NimbleStorage array, but likely will end up with an HP 3Par. Let's see. So far, EMC yanked out the good SP when attempting to replace the bad SP. Lost data, recovered from tape. Another time we lost two disks in a single LUN, some corruption issues. Recovered from Veeam in like 20 minutes but unfun week. Memorial day weekend it took two days to get a replacement hard drive. No lost data, but getting the hot spare to unspare itself took some work.

I have a DC on each SAN, plus a physical. I mean, tossing 8 gigs of RAM and maybe 60 gigs of disk space is overkill. OS doesn't cost much, nothing if you're using DataCenter version.

Did you see the HP acquired Nimble?

[Regolith]

RevDisk, not a word of that makes any sense at all.   laughing.  I guess I'm just not computer savvy.    :)

There's a post deep in the reddit thread that does a pretty good job of summing up:

https://np.reddit.com/r/cscareerquestions/comments/6ez8ag/accidentally_destroyed_production_database_on/dieko7y/

    I'm old and pretty technologically illiterate. I understand about 20% of what you guys are talking about here.

I'm bored, so let me explain to you. Not knowing which 20% you understand, let's go back to basics:

• A database is a piece of software that stores data used by an application. Reddit has a database that stores user accounts, threads, comments, everything.
• In order for your application to access a database, you need to input in your application its URL (its address), and a valid account's username and password.
• Some accounts can only read the data in the database, some can read and write, modify, and delete data in the database.
• A production environment is the real instance of the application and its database used by the company or the clients. The production database has all the real data.
• A development environment is an instance of the application and database used for development. The developer usually has, on his own computer, a database with fake data, and the code of the application. When he runs the application from his code, the application should use the test database.
• Tests will usually either create crap data in the database, or simply overwrite the database with fresh fake data every time they are run. So you really don't want your development application to connect to the production database.

So in this case, the new guy was told on his first day of work to set up his own development environment. He was provided a procedure to do it.

But when the time came to connect his development application to the development database, he made a mistake, and instead of using the url and account of his development database, he used those provided in the procedure, which were those of the production database.

When he ran tests, his development application overwrote the production data with fake test data.

Now let's look at who did what wrong. First the new guy:

• He made a small mistake when reading the procedure.

The company:

• They put the URL of the production database in the development setup guide. Not recommended.
• They put the username and password of an account with full access to the production database in that guide. Enormous mistake.
• They didn't prevent other computers from connecting to the production environment (the production database should refuse connections from any server which isn't the one running the production application, even if it provides a valid username/password). Big mistake.
• They have backups of their database, which is good, but seem unable to restore it. Restoring a database can be tricky indeed, that's why you make procedures, test them, and get people who know how to deal with databases. The company's fault if they don't.

The company deserves nearly all the blame. They violated basic security measures that would have easily prevented that from happening.

[Mannlicher]

Let me dumb it down for you.

Company make boo boo.

CTO make boo boo.

Junior developer make boo boo that turns Company and CTO boo boos into HUGE OH MY SWEET BABY JEEBUS!

Simple, really.

no need to be condescending.     I gathered that things went awry.   laughing

[HeroHog]

no need to be condescending.     I gathered that things went awry.   laughing

Condescending? No! Succinct!

[zxcvbob]

The CTO is making so much noise to deflect attention away from himself; blame his own screw-up on a subordinate.  He should be fired for that.

I've almost deleted a database on a production machine; it was about a year ago.  I caught myself just as I was about to hit <Enter>.  I thought I was on the development machine, and the terminal session said I was on the dev machine -- but I was telnet'd from there to the production machine.  I realized I had 2 terminals with the development system name (which is not that unusual by itself) and none with the production machine name.

And I knew better.  The folks who gave me access to the production machine even told me to be careful.  It would not have been a huge disaster, but it would have created hours of extra work for them to cleanup my mess.

My real screw-up was passing through from one system to the other instead of starting a new session.

[Firethorn]

There's a post deep in the reddit thread that does a pretty good job of summing up:

In a lot of cases you don't even give your development guys access to the production systems at all.

As a developer, I can guarantee that I'm going to crash the test environment several times a day.  That's okay, I actually have automated tools that amount to 'nuke and rebuild'.  I set that off, and I'm back in business in like 5 minutes or less.

Once I have whatever feature I was working on in working order, acceptably optimized, and tested, then somebody else looks at it, tests it, and transfers it over to production.

But that's because I'm working with a tiny representation of the real thing.  Restoring the primary system takes far longer, fortunately it can often be 'hidden' because of multiple redundant nodes that failover to each other, and only 1 updates at a time.  You just don't see the new stuff until your node is updated.

[TechMan]

The CTO is making so much noise to deflect attention away from himself; blame his own screw-up on a subordinate.  He should be fired for that.

I've almost deleted a database on a production machine; it was about a year ago.  I caught myself just as I was about to hit <Enter>.  I thought I was on the development machine, and the terminal session said I was on the dev machine -- but I was telnet'd from there to the production machine.  I realized I had 2 terminals with the development system name (which is not that unusual by itself) and none with the production machine name.

And I knew better.  The folks who gave me access to the production machine even told me to be careful.  It would not have been a huge disaster, but it would have created hours of extra work for them to cleanup my mess.

My real screw-up was passing through from one system to the other instead of starting a new session.

Good catch.  I don't like the session within a session method fro that very reason.

[RevDisk]

I'm always of the opinion that your production environment should be able to be set on fire, nuked by an intern, any cable could be ripped out.... And you should be back in business in whatever interval that is acceptable to the business. It's not easy. Over the years, I learned the best way to do so is work out the numbers for achieving that, put it in a very simple framework. I usually go with gold, silver, bronze. Sometimes platinum, sometimes lead. Obviously, I expect and usually want the brass to pick silver. But they're committed to the accompanying number afterwards.

If they fund for a 4 or 8 hour recovery, they can't scream at you for a 4 or 8 hour recovery. Well, they can. Then you ask if they're authorizing the extra bucks to bump up to mirrored production environment or whatnot. I'm a big believer in the boss should never be surprised. Our tape drive is acting up and I want to buy a replacement. But it's a capital expense. Instead we're relying on redundant far cheaper boxes of cheap hard drives. Which are faster anyways, but don't have the space for long term archiving. We only keep a month on disk.

Did you see the HP acquired Nimble?

Yep. EMC is bad, has been getting worse and I have zero faith that they will get better under Dell. I've seen view Dell products get better after acquisition. They made a new Nimble clone that we'd absolutely love if it wasn't made by EMC.

HPE/HP/whatever and Nimble... Is a gamble. Historically the products get better for a couple years after HP stripmines a new acquisition. Then you get a long slow downward spiral. Their enterprise support has generally been decent. (ie not stuck in India support hell for multi hundred thousand dollar device) 3Par is further along the acquisition path and still... decent.

Pure is too much smoke and not enough substance that I've seen.

Good catch.  I don't like the session within a session method fro that very reason.

Ditto. I actively try to avoid it.

[AJ Dual]

We're on a project to go almost 100% cloud over the next two years. Since everything is virtual in the VNX boxes, doesn't really matter where they are. Our datacenter is already 90% empty as everything virtualized.

We're not thrilled with EMC either, and Dell's acquisition is not inspiring confidence.

Datacenter as a utility, just like your gas, electric, and water is a thing now. And making it triply redundant across AWS regions is definitely appealing. And we'll be keeping an eye on MS Edge and Google's offerings, and might even make it sextuple-redundant someday.

Our biggest challenge has been maintaining three redundant fat pipes through three different vendors. They keep buying each other out. 

[K Frame]

"We're on a project to go almost 100% cloud over the next two years."

Ah, the cloud...

Always laughed at the concept.

All it means is that your data is now on off-site servers, which are out of your control. They're not electron bits floating freely around in "a cloud.'