Armed Polite Society

Main Forums => The Roundtable => Topic started by: RevDisk on June 12, 2017, 12:31:54 PM

Title: Junior dev manages to cripple/kill company on first day
Post by: RevDisk on June 12, 2017, 12:31:54 PM

https://np.reddit.com/r/cscareerquestions/comments/6ez8ag/accidentally_destroyed_production_database_on/
https://qz.com/999495/the-tech-world-is-rallying-around-a-young-developer-who-made-a-huge-embarrassing-mistake/

Obviously, not the kid's fault. CTO fired the kid probably to try to cover his own rear end.

Company managed to put read/write passwords in their setup guide.
They didn't bother helping a new dev set up their environment.
Scripts had no safety logic. This is akin to remove all shields or covers from any type of engine.
Backups? lolno.

I don't consider myself a real DBA, and I backup all production databases every 12 hours with Veeam to external big boxes of cheap hard drives. Two of them. Because. On each virtual server is the last two day's worth of backups. Oh, and we write weekly to tape, plus monthly tapes. I'd take more extensive procautions, but realistically anything that nukes all three backups probably means the end of the business anyways. We're talking about doing live replication across the country, mostly for performance rather than disaster management.
Title: Re: Junior dev manages to cripple/kill company on first day
Post by: Brad Johnson on June 12, 2017, 12:40:43 PM
From his description it appears he did exactly as instructed. Sounds like someone else's head needs to roll.

Brad
Title: Re: Junior dev manages to cripple/kill company on first day
Post by: RevDisk on June 12, 2017, 12:41:50 PM
From his description it appears he did exactly as instructed. Sounds like someone else's head needs to roll.

Brad

Sorta. He copied the values from the document rather than the credentials some other tool gave him.
Title: Re: Junior dev manages to cripple/kill company on first day
Post by: K Frame on June 12, 2017, 01:10:04 PM
I can't believe the number of incredible screw ups this company allowed to happen before this poor guy hired on. He needs a lawyer, because he really should consider suing for wrongful termination.


I've seen some similar screw ups happen in the past at my company, but NEVER in a production environment.

We had one guy nuke our sandbox SharePoint site almost to glass because one of the admins gave him basically god rights and the guy had no real experience with SharePoint.
Title: Re: Junior dev manages to cripple/kill company on first day
Post by: AJ Dual on June 12, 2017, 01:15:26 PM
If word leaks sideways in the org chart, about the litany and unbroken chain of process flaws and best-practices violations, the CTO is going to lose his job.
Title: Re: Junior dev manages to cripple/kill company on first day
Post by: griz on June 12, 2017, 01:24:05 PM
A company that hires a new guy and gives him all the tools he needs to erase EVERYTHING on his first day with no supervision?  Sounds like they were a disaster waiting to happen and he pulled the trigger.
Title: Re: Junior dev manages to cripple/kill company on first day
Post by: AJ Dual on June 12, 2017, 01:32:42 PM
Something similar happened here. Tech Services/Infrastructure/Ops manager allowed his team to have all three of the company Domain Controllers on only one of our twin EMC VNX clusters, and only one of the twin EMC SAN's that served them.

Despite some of his team warning him it wasn't a good idea, and that they should be spread out.

Apparently a particular run of HGST enterprise class drives had a problem where gas would "stick" to the platters and screw up the head's read/write ability. A few drives went down, more came online, except drives that had been idle had that problem worse.

And the HSGT supplied firmware "fix" to clean off the drives just wrecked them all en-masse. Oops.

Prod users didn't notice much, a few things ran slow as things flipped over to the mirrored disaster recovery datacenter, and along with the DC's, DNS got hosed for a bit... but IT sure knew. And we lost drives well past what the RAID could rebuild from, and with the damaged/missing virtual DC's, we lost the backup index too. So everything was backed up 10 ways come Sunday, but it was unsearchable.

Fortunately our team here is top notch despite oversights like that, and within 24 hours, 90% of the environment was restored. Within 72 hours 100% was. And that counted almost 250+ virtual servers for Stage, Test, and Dev which were set back up all manually, and only fed their data from backups.

EMC and HGST was onsite by the next day working with us, and while I'm not privy to the details, we got some even newer hardware out of whatever happened, and I don't think we paid, or paid much for it.

That guy was "soft fired", given 3 months to find something else and move on.

The team member most vocal about spreading the DC's and other critical infrastructure across both VNX's and both SAN's has his job now.

Title: Re: Junior dev manages to cripple/kill company on first day
Post by: K Frame on June 12, 2017, 01:34:00 PM
A company that hires a new guy and gives him all the tools he needs to erase EVERYTHING on his first day with no supervision?  Sounds like they were a disaster waiting to happen and he pulled the trigger.

Without a doubt.

It sounds as if they're a non-ITIL, non-ISSO and/or non-CMMI shop.

That's basic data continuity/protection.
Title: Re: Junior dev manages to cripple/kill company on first day
Post by: K Frame on June 12, 2017, 01:45:12 PM
"And the HSGT supplied firmware "fix" to clean off the drives just wrecked them all en-masse. Oops."

Reminds me of an incident with our local cable TV/high-speed internet provider maybe 10-12 years ago.

Cox Cable here in Northern Virginia had the high speed road runner internet service, and one evening it just up and died. Completely.

Turns out that Cox had rolled out a software upgrade that totally bricked something like 15,000 Toshiba cable modems. http://www.dslreports.com/shownews/Whoops-41697

They made everyone come in to one of their service centers to exchange modems for a newer model.

The CEO, Gary McCollum, was walking around and he and I got to chat. I had one very pointed question for him -- "Don't you guys have a test environment?"

Turns out that yeah, they have a test environment.

But what they didn't have was a test environment with full representation of all modems then in use. The modems that were killed by the upgrade weren't part of the test environment.

I was not particularly complimentary of that.
Title: Re: Junior dev manages to cripple/kill company on first day
Post by: K Frame on June 12, 2017, 01:58:09 PM
Just shared this with a bunch of people on my team.
Title: Re: Junior dev manages to cripple/kill company on first day
Post by: Mannlicher on June 12, 2017, 02:15:14 PM
RevDisk, not a word of that makes any sense at all.   laughing.  I guess I'm just not computer savvy.    :)
Title: Re: Junior dev manages to cripple/kill company on first day
Post by: K Frame on June 12, 2017, 02:24:32 PM
RevDisk, not a word of that makes any sense at all.   laughing.  I guess I'm just not computer savvy.    :)

Let me dumb it down for you.

Company make boo boo.

CTO make boo boo.

Junior developer make boo boo that turns Company and CTO boo boos into HUGE OH MY SWEET BABY JEEBUS!

Simple, really.

:rofl:
Title: Re: Junior dev manages to cripple/kill company on first day
Post by: HeroHog on June 12, 2017, 02:51:32 PM
Let me dumb it down for you.

Company make boo boo.

CTO make boo boo.

Junior developer make boo boo that turns Company and CTO boo boos into HUGE OH MY SWEET BABY JEEBUS!

Simple, really.

:rofl:

When I share this, I'm quoting this!
Title: Re: Junior dev manages to cripple/kill company on first day
Post by: RevDisk on June 12, 2017, 03:54:31 PM
Mike is spot on!


Something similar happened here. Tech Services/Infrastructure/Ops manager allowed his team to have all three of the company Domain Controllers on only one of our twin EMC VNX clusters, and only one of the twin EMC SAN's that served them.

Despite some of his team warning him it wasn't a good idea, and that they should be spread out.

Apparently a particular run of HGST enterprise class drives had a problem where gas would "stick" to the platters and screw up the head's read/write ability. A few drives went down, more came online, except drives that had been idle had that problem worse.

And the HSGT supplied firmware "fix" to clean off the drives just wrecked them all en-masse. Oops.

Prod users didn't notice much, a few things ran slow as things flipped over to the mirrored disaster recovery datacenter, and along with the DC's, DNS got hosed for a bit... but IT sure knew. And we lost drives well past what the RAID could rebuild from, and with the damaged/missing virtual DC's, we lost the backup index too. So everything was backed up 10 ways come Sunday, but it was unsearchable.

Fortunately our team here is top notch despite oversights like that, and within 24 hours, 90% of the environment was restored. Within 72 hours 100% was. And that counted almost 250+ virtual servers for Stage, Test, and Dev which were set back up all manually, and only fed their data from backups.

EMC and HGST was onsite by the next day working with us, and while I'm not privy to the details, we got some even newer hardware out of whatever happened, and I don't think we paid, or paid much for it.

That guy was "soft fired", given 3 months to find something else and move on.

The team member most vocal about spreading the DC's and other critical infrastructure across both VNX's and both SAN's has his job now.


Queue whistling pic.

We had a crap physical DC. I happened to order a very nifty HP cube server with all the bells and whistles for a subsidiary that I suspected would not work out. Long with a bunch of other things I thought they would need... But if things didn't work out, would be very beneficial to the main office.

My sympathy for having VNX's. I have one at the moment, and support has been absolutely horrible. EMC is charging platinum pricing for bronze service, to be generous. I'm angling for a NimbleStorage array, but likely will end up with an HP 3Par. Let's see. So far, EMC yanked out the good SP when attempting to replace the bad SP. Lost data, recovered from tape. Another time we lost two disks in a single LUN, some corruption issues. Recovered from Veeam in like 20 minutes but unfun week. Memorial day weekend it took two days to get a replacement hard drive. No lost data, but getting the hot spare to unspare itself took some work.

I have a DC on each SAN, plus a physical. I mean, tossing 8 gigs of RAM and maybe 60 gigs of disk space is overkill. OS doesn't cost much, nothing if you're using DataCenter version.
Title: Re: Junior dev manages to cripple/kill company on first day
Post by: TechMan on June 12, 2017, 04:22:37 PM
Sorta. He copied the values from the document rather than the credentials some other tool gave him.

I've seen that so many times, that I make sure to remove the credentials from screen shots.
Title: Re: Junior dev manages to cripple/kill company on first day
Post by: TechMan on June 12, 2017, 04:33:19 PM
Mike is spot on!



Queue whistling pic.

We had a crap physical DC. I happened to order a very nifty HP cube server with all the bells and whistles for a subsidiary that I suspected would not work out. Long with a bunch of other things I thought they would need... But if things didn't work out, would be very beneficial to the main office.

My sympathy for having VNX's. I have one at the moment, and support has been absolutely horrible. EMC is charging platinum pricing for bronze service, to be generous. I'm angling for a NimbleStorage array, but likely will end up with an HP 3Par. Let's see. So far, EMC yanked out the good SP when attempting to replace the bad SP. Lost data, recovered from tape. Another time we lost two disks in a single LUN, some corruption issues. Recovered from Veeam in like 20 minutes but unfun week. Memorial day weekend it took two days to get a replacement hard drive. No lost data, but getting the hot spare to unspare itself took some work.

I have a DC on each SAN, plus a physical. I mean, tossing 8 gigs of RAM and maybe 60 gigs of disk space is overkill. OS doesn't cost much, nothing if you're using DataCenter version.

Did you see the HP acquired Nimble?
Title: Re: Junior dev manages to cripple/kill company on first day
Post by: Regolith on June 12, 2017, 08:05:24 PM
RevDisk, not a word of that makes any sense at all.   laughing.  I guess I'm just not computer savvy.    :)

There's a post deep in the reddit thread that does a pretty good job of summing up:

https://np.reddit.com/r/cscareerquestions/comments/6ez8ag/accidentally_destroyed_production_database_on/dieko7y/
Quote from: hey01

Quote
    I'm old and pretty technologically illiterate. I understand about 20% of what you guys are talking about here.

I'm bored, so let me explain to you. Not knowing which 20% you understand, let's go back to basics:

  • A database is a piece of software that stores data used by an application. Reddit has a database that stores user accounts, threads, comments, everything.
  • In order for your application to access a database, you need to input in your application its URL (its address), and a valid account's username and password.
  • Some accounts can only read the data in the database, some can read and write, modify, and delete data in the database.
  • A production environment is the real instance of the application and its database used by the company or the clients. The production database has all the real data.
  • A development environment is an instance of the application and database used for development. The developer usually has, on his own computer, a database with fake data, and the code of the application. When he runs the application from his code, the application should use the test database.
  • Tests will usually either create crap data in the database, or simply overwrite the database with fresh fake data every time they are run. So you really don't want your development application to connect to the production database.

So in this case, the new guy was told on his first day of work to set up his own development environment. He was provided a procedure to do it.

But when the time came to connect his development application to the development database, he made a mistake, and instead of using the url and account of his development database, he used those provided in the procedure, which were those of the production database.

When he ran tests, his development application overwrote the production data with fake test data.

Now let's look at who did what wrong. First the new guy:

  • He made a small mistake when reading the procedure.

The company:
  • They put the URL of the production database in the development setup guide. Not recommended.
  • They put the username and password of an account with full access to the production database in that guide. Enormous mistake.
  • They didn't prevent other computers from connecting to the production environment (the production database should refuse connections from any server which isn't the one running the production application, even if it provides a valid username/password). Big mistake.
  • They have backups of their database, which is good, but seem unable to restore it. Restoring a database can be tricky indeed, that's why you make procedures, test them, and get people who know how to deal with databases. The company's fault if they don't.

The company deserves nearly all the blame. They violated basic security measures that would have easily prevented that from happening.
Title: Re: Junior dev manages to cripple/kill company on first day
Post by: Mannlicher on June 12, 2017, 09:15:02 PM
Let me dumb it down for you.

Company make boo boo.

CTO make boo boo.

Junior developer make boo boo that turns Company and CTO boo boos into HUGE OH MY SWEET BABY JEEBUS!

Simple, really.

:rofl:

no need to be condescending.     I gathered that things went awry.   laughing
Title: Re: Junior dev manages to cripple/kill company on first day
Post by: HeroHog on June 12, 2017, 09:52:40 PM
no need to be condescending.     I gathered that things went awry.   laughing

Condescending? No! Succinct!
Title: Re: Junior dev manages to cripple/kill company on first day
Post by: zxcvbob on June 12, 2017, 10:31:12 PM
The CTO is making so much noise to deflect attention away from himself; blame his own screw-up on a subordinate.  He should be fired for that.

I've almost deleted a database on a production machine; it was about a year ago.  I caught myself just as I was about to hit <Enter>.  I thought I was on the development machine, and the terminal session said I was on the dev machine -- but I was telnet'd from there to the production machine.  I realized I had 2 terminals with the development system name (which is not that unusual by itself) and none with the production machine name.

And I knew better.  The folks who gave me access to the production machine even told me to be careful.  It would not have been a huge disaster, but it would have created hours of extra work for them to cleanup my mess.

My real screw-up was passing through from one system to the other instead of starting a new session.
Title: Re: Junior dev manages to cripple/kill company on first day
Post by: Firethorn on June 12, 2017, 11:51:10 PM
There's a post deep in the reddit thread that does a pretty good job of summing up:

In a lot of cases you don't even give your development guys access to the production systems at all.

As a developer, I can guarantee that I'm going to crash the test environment several times a day.  That's okay, I actually have automated tools that amount to 'nuke and rebuild'.  I set that off, and I'm back in business in like 5 minutes or less.

Once I have whatever feature I was working on in working order, acceptably optimized, and tested, then somebody else looks at it, tests it, and transfers it over to production.

But that's because I'm working with a tiny representation of the real thing.  Restoring the primary system takes far longer, fortunately it can often be 'hidden' because of multiple redundant nodes that failover to each other, and only 1 updates at a time.  You just don't see the new stuff until your node is updated.

Title: Re: Junior dev manages to cripple/kill company on first day
Post by: TechMan on June 13, 2017, 07:01:01 AM
The CTO is making so much noise to deflect attention away from himself; blame his own screw-up on a subordinate.  He should be fired for that.

I've almost deleted a database on a production machine; it was about a year ago.  I caught myself just as I was about to hit <Enter>.  I thought I was on the development machine, and the terminal session said I was on the dev machine -- but I was telnet'd from there to the production machine.  I realized I had 2 terminals with the development system name (which is not that unusual by itself) and none with the production machine name.

And I knew better.  The folks who gave me access to the production machine even told me to be careful.  It would not have been a huge disaster, but it would have created hours of extra work for them to cleanup my mess.

My real screw-up was passing through from one system to the other instead of starting a new session.

Good catch.  I don't like the session within a session method fro that very reason.
Title: Re: Junior dev manages to cripple/kill company on first day
Post by: RevDisk on June 13, 2017, 07:50:53 AM

I'm always of the opinion that your production environment should be able to be set on fire, nuked by an intern, any cable could be ripped out.... And you should be back in business in whatever interval that is acceptable to the business. It's not easy. Over the years, I learned the best way to do so is work out the numbers for achieving that, put it in a very simple framework. I usually go with gold, silver, bronze. Sometimes platinum, sometimes lead. Obviously, I expect and usually want the brass to pick silver. But they're committed to the accompanying number afterwards.

If they fund for a 4 or 8 hour recovery, they can't scream at you for a 4 or 8 hour recovery. Well, they can. Then you ask if they're authorizing the extra bucks to bump up to mirrored production environment or whatnot. I'm a big believer in the boss should never be surprised. Our tape drive is acting up and I want to buy a replacement. But it's a capital expense. Instead we're relying on redundant far cheaper boxes of cheap hard drives. Which are faster anyways, but don't have the space for long term archiving. We only keep a month on disk.



Did you see the HP acquired Nimble?

Yep. EMC is bad, has been getting worse and I have zero faith that they will get better under Dell. I've seen view Dell products get better after acquisition. They made a new Nimble clone that we'd absolutely love if it wasn't made by EMC.

HPE/HP/whatever and Nimble... Is a gamble. Historically the products get better for a couple years after HP stripmines a new acquisition. Then you get a long slow downward spiral. Their enterprise support has generally been decent. (ie not stuck in India support hell for multi hundred thousand dollar device) 3Par is further along the acquisition path and still... decent.

Pure is too much smoke and not enough substance that I've seen.


Good catch.  I don't like the session within a session method fro that very reason.

Ditto. I actively try to avoid it.
Title: Re: Junior dev manages to cripple/kill company on first day
Post by: AJ Dual on June 13, 2017, 09:40:09 AM
We're on a project to go almost 100% cloud over the next two years. Since everything is virtual in the VNX boxes, doesn't really matter where they are. Our datacenter is already 90% empty as everything virtualized.

We're not thrilled with EMC either, and Dell's acquisition is not inspiring confidence.

Datacenter as a utility, just like your gas, electric, and water is a thing now. And making it triply redundant across AWS regions is definitely appealing. And we'll be keeping an eye on MS Edge and Google's offerings, and might even make it sextuple-redundant someday.

Our biggest challenge has been maintaining three redundant fat pipes through three different vendors. They keep buying each other out.  :P
Title: Re: Junior dev manages to cripple/kill company on first day
Post by: K Frame on June 13, 2017, 09:57:07 AM
"We're on a project to go almost 100% cloud over the next two years."

Ah, the cloud...

Always laughed at the concept.

All it means is that your data is now on off-site servers, which are out of your control. They're not electron bits floating freely around in "a cloud.'
Title: Re: Junior dev manages to cripple/kill company on first day
Post by: RevDisk on June 13, 2017, 09:59:51 AM

It all really depends on your business model for if cloud makes sense or not. Multi gig CAD drawings, can't cloud it. Lots of small word docs, pictures and whatnot? Can cloud it. Mixed cloud can be interesting, but is getting easier.



All of our web sites are rented dumb Linux VM, because it's so ridiculously cheap you'd have to be an idiot to self-host that.

Email is starting to look cheaper to rent than run. We're moving to O365 because Microsoft isn't giving people a choice about it. Some parts make me happy, some parts are enraging. My 'home email' I moved from a self-run postfix to Google Business. $5/month? *expletive deleted*it, I'd pay more than that for not having to deal with spam filtering alone. They throw in dozens if not hundreds of free extra stuff.

I just wish business wise we could go with Google over MS, for pricing/security/reliability/futureproofing/etc. Unfortunately, our users probably can't handle the complexity of using bog simple web pages used by a billion teenagers everyday, so O365 it is. Microsoft is expensive and they make incredibly stupid decisions that are openly hostile to their business users, on par with Apple. But they have a captive audience of millions of office workers barely competent at the desktop tools they are sort of familiar with and incapable of substantial change.

Cloud DB is nosebleedingly expensive unless you go with Amazon DB or mysql/FOSS. Pricing out 700GB azure SQL server was...  fun.

We're not at ubiquitous computing, datacenter/bandwidth is a long way from utility and cloud still has a very very long way to mature or become trustworthy. I give it another... three decades? Maybe. I'm optimistic.
Title: Re: Junior dev manages to cripple/kill company on first day
Post by: TechMan on June 13, 2017, 11:45:29 AM
I have a couple of more years on a few Equallogic boxes, that should give me enough time to see what HP does to Nimble.  We are looking at a PanZura device to put our Revit models out into the AWS cloud.
Title: Re: Junior dev manages to cripple/kill company on first day
Post by: AJ Dual on June 13, 2017, 02:31:32 PM
"We're on a project to go almost 100% cloud over the next two years."

Ah, the cloud...

Always laughed at the concept.

All it means is that your data is now on off-site servers, which are out of your control. They're not electron bits floating freely around in "a cloud.'

(rubs hands gleefully) Oh boy, this has suddenly become one of those threads where we flip a coin and see if Mike shows an ounce of humility and learns something, or just gets angrier and angrier until it's locked.  :angel:
Title: Re: Junior dev manages to cripple/kill company on first day
Post by: Fitz on June 13, 2017, 02:55:01 PM
People who think the cloud is nothing more than remote servers don't understand the cloud

It's not about the infrastructure , it's about the automation, commoditizing of hardware , and multi geo/ HADR that's transparent to the consuming service



The "hurr durr it's another computer " types not only don't understand the tech, but as is evident here, they don't even understand the origin of the term. The term cloud has never been about "electrons floating around". It was a term to describe the obfuscation of the underlying service fabric, and the notion that the underlying infrastructure didn't matter much because of the aforementioned commoditization


Considering the industry shift, anyone in IT who is still ignorant of the cloud would do well to ensure they actually make an effort to understand the tech, as the days of on prem services for many technologies are numbered


In fact, had the company involved been on my service, I would have had their database back in 15 minutes
Title: Re: Junior dev manages to cripple/kill company on first day
Post by: AJ Dual on June 13, 2017, 03:00:28 PM
You know... putting your stuff on "someone else's computer" that's massively parallel, distributed, fault-tolerant/redundant to a degree that even most Fortune 500 companies can't afford on their own, in a secure datacenter with 2-3 different kinds of infrastructure/power backup, and is actually spread out running in parallel over several U.S. states, different continents even... is "stupid", because reasons.  =D
Title: Re: Junior dev manages to cripple/kill company on first day
Post by: Brad Johnson on June 13, 2017, 03:00:53 PM

Considering the industry shift, anyone in IT who is still ignorant of the cloud would do well to ensure they actually make an effort to understand the tech, as the days of on prem services for many technologies are numbered

Bewaaaaare the cloud...  :old:

Brad
Title: Re: Junior dev manages to cripple/kill company on first day
Post by: RevDisk on June 13, 2017, 03:17:41 PM
"We're on a project to go almost 100% cloud over the next two years."

Ah, the cloud...

Always laughed at the concept.

All it means is that your data is now on off-site servers, which are out of your control. They're not electron bits floating freely around in "a cloud.'
(rubs hands gleefully) Oh boy, this has suddenly become one of those threads where we flip a coin and see if Mike shows an ounce of humility and learns something, or just gets angrier and angrier until it's locked.  :angel:

Angrier and angrier, obviously.

I'm not quite as leery about cloud as I used to be. One simple reason. The customers now collectively have enough lawyers to make the providers live up to expectations. It's also maturing. The weaker players have been shaken out. AWS and Azure will stick around. AWS is essentially larger than the next handful of cloud providers put together, but that will change because MS is 'forcing' a lot of people towards Azure if they want to use MS products. AWS is making money hand over fist. If you told me that Amazon's IT department was more profitable than the rest of the business combined, I wouldn't remotely be surprised.

That said, you can lose your shirt if you don't know what you are doing. It's just as dangerous as running your own internal resources if you don't have a clue. AWS and similar clouds are huge huge huge targets for rogue entities. State sponsored hacker groups like China, Russia or our own 'beloved' NSA are definitely threats against our core infrastructure. On the other hand, cloud providers have more niche security staff than even megacorps generally can hire full time. Renting your infrastructure vs owning can make economic sense. It's a thousand times more dangerous and complex than the 'cloud' marketing says, but that doesn't mean you shouldn't do it.

Again, tiny linux servers? You'd have to be flat out insane to self-host at current pricing. That's where "cloud" is the norm and self-hosting has become unusual.

Netflix, for example, generally runs none of their own infrastructure and they're essentially the largest content provider on the internet (possibly the entire world). Except for maybe Youtube. They don't run their cloud infrastructure like any person would run their domestic infrastructure. Nifty thing is, they open source huge amount of their tooling, which is flat out insane. Good, but insane. I'd explain chaos monkeys and dynamic load balancing and hadoop and all kinds of other stuff.

But it's not something to immediately write off. This is coming from a guy that firmly reminds people that there is no cloud, it's just renting a server from someone else.


People who think the cloud is nothing more than remote servers don't understand the cloud


Correct. Some routers and switches are involved as well.   =D


Considering the industry shift, anyone in IT who is still ignorant of the cloud would do well to ensure they actually make an effort to understand the tech, as the days of on prem services for many technologies are numbered

I worry people actually believe this as a real meaningful statement. I mean, if you REALLY know what you're doing, yes, kinda sorta not really. If you taking to the average manager, it's important to make eye contact and explain it's a marketing statement. IaaS and SaaS are tools. Potentially good tools. But they're just another tool in the bucket of other tools. Maybe a major tool that you use for a huge part of your daily job. Anyone trying to sell you a hammer that swears it will obsolete every other tool in the bucket is lying, because they're being paid to lie to you. I'm not bashing Fitz, he knows exactly what I mean. He also knows to beware of idiots with high spending authorization that stupidly listen to marketing drones over their own people.

It's equal stupid to swear a tool is never useful as it is to claim it is useful for absolutely everything. Bandwidth is not universal and not falling at predicted rates. And I mean for bloody everything. Home, cell or fiber. Because the telecoms own Congress. I don't see that changing. That's the fundamental limit on IaaS, SaaS and every other XaaS. Most of the country has one provider and very unpromising backup providers. If you DO have hardened multiply redundant interwebz connections at appropriate speeds, then yes, it's a good gamble with only moderate risk.

That said, and I tell anyone with cloudz fever, make sure you always have an exit strategy. Prices could go up, businesses close, quality and support change. People ALWAYS go pants on head retarded eventually.
Title: Re: Junior dev manages to cripple/kill company on first day
Post by: Fitz on June 13, 2017, 06:20:18 PM
I didn't say all, I said many. And it's true. And actually, most of the shift will be in the PaaS space.


PaaS, microservices, containers etc. That's where people are building their products because it's far and away the most stable, scalable, and secure place to do it.


Of all the people who decry the lack of security in the cloud, the "high" cost, etc. I've only actually met one who was doing a better job than the major providers.

Economy of scale doesn't just go away because some yokel thinks the cloud is dumb. and you KNOW i'm not talking about guys like you.


I'm talking about probably 70 percent of "professional" IT shops who are terrible. I'm talking about the legions of "professionals" who were vulnerable to wannacry after the fix had been in the wild for a month. I'm talking about guys who still say their data has to "stay with us! Not in someone else's server!" and then have essentially NO HADR plan.

Managed services absolutely will be taking over for the majority of enterprise grade applications. My application alone has had 3700% growth in 3 years. 3 years. It's getting to the point that MS is considering dropping on prem support entirely. Right now the on prem product is just a way to entice customers with "hybrid" features, or "please dip your toes in the cloud and OH LOOK HOW EASY IT IS" stuff.


People make big waves in the news about outages that cloud providers have, but they usually fail to mention that those "mnassive" outages end up bringing EOM reliability to 99.99 percent or so.

Show me someone that can consistently do 4 9's of availability and i'll show you one of two things: either a liar, or someone who's not actually tracking their service uptime
Title: Re: Junior dev manages to cripple/kill company on first day
Post by: TechMan on June 13, 2017, 09:29:16 PM
I love SaaS for almost everything.  I have had Office 365 since the end of 2011 and happily pay for it and won't consider moving to any other platform.  It just works and I don't have to be an Exchange expert to make it work.  I don't have to invest in backup software and I don't have to invest in new hardware.
Title: Re: Junior dev manages to cripple/kill company on first day
Post by: K Frame on June 14, 2017, 09:51:32 AM
"People who think the cloud is nothing more than remote servers don't understand the cloud

It's not about the infrastructure , it's about the automation, commoditizing of hardware , and multi geo/ HADR that's transparent to the consuming service"

You mean that there's actually other stuff involved, mechanisms, processes, procedures, blah blah blah that make "the cloud" work?????

Well DUH!

But at its very heart, it's still data stored on remote devices that are, as often as not, out of your direct control.

If, as you claim, the cloud isn't about the infrastructure, that infers that you can simply delete the infrastructure, right? Well then, where is your data stored? Where do your applications reside? Where's your active directory base?

The infrastructure is as critical to the operation of the cloud as the concept is, as without one, you don't have the other.

But, if it makes you happy, I'll ask the other members of the high availability-disaster recovery cloud group I've been sitting on for the last 18 months to sent me to a re-education camp...  ;/


Title: Re: Junior dev manages to cripple/kill company on first day
Post by: K Frame on June 14, 2017, 10:02:38 AM
"Angrier and angrier, obviously."

Actually more and more entertained by all of the people who are quickly jumping to the "defense" of the cloud to tell me that my base assessment -- that it's nothing more than data stored somewhere else instead of in internal data centers -- is wrong and that somehow I just don't have the understands because, you know... LEVELS OF TECHNOLOGY!


HURPDY DURPDY DOO! DOMAIN DEVICES! SCALERS! PROTOCOLS! MAGICAL DEVICES! YOU CAN ONLY APPRECIATE THE CLOUD IF YOUR ARCHITECT THE CLOUD!



Right.

Well no flying *expletive deleted*ck there's a lot of conceptualization and technology behind "the cloud" and how it works.

But it still boils down to data storage. You're either storing your data internally, in your own data centers, or your storing your data externally in "the cloud." Which is external data centers.

In some cases it makes a LOT of sense for an organization.

In other cases, no.



Title: Re: Junior dev manages to cripple/kill company on first day
Post by: Fitz on June 14, 2017, 10:34:18 AM
Heh

You should calm down
Title: Re: Junior dev manages to cripple/kill company on first day
Post by: AJ Dual on June 14, 2017, 10:47:19 AM
(https://armedpolitesociety.com/proxy.php?request=http%3A%2F%2Fi156.photobucket.com%2Falbums%2Ft33%2FAJ_Dual%2Fold_zps2e7s9bhm.jpg&hash=98e3b09d2946689464747357ae33cca2d7538a6d) (http://s156.photobucket.com/user/AJ_Dual/media/old_zps2e7s9bhm.jpg.html)
Title: Re: Junior dev manages to cripple/kill company on first day
Post by: K Frame on June 14, 2017, 10:57:14 AM
Heh

You should calm down

You should learn to understand derisive sarcasm and mockery.

I also mock those idiots where I work who think that the cloud is either:

1. The answer to ALL of the customer's problems, and everything should be on the external vendor cloud! (Yeah really? Even the top secret diplomatic *expletive deleted*it.... really?)

2. The most evil thing ever, and it's just an open door with a big welcome sign for the Chinese/Russians/North Koreans/whomever to steal our data.

I guess I need to add a third category...

Title: Re: Junior dev manages to cripple/kill company on first day
Post by: Fitz on June 14, 2017, 11:04:15 AM
"Grumpy dudes who fly into a profanity laden diatribe when someone suggests that their oversimplification of a technology is incorrect?"

I mean, I'm one too, but come on dude :-)

You either knew precisely what I meant when I said the cloud was about the infrastructure , and were deliberately being obtuse, or. Well I don't know actually

Offsite hosting has always been a thing, the cloud is notably different. And if it's a "no duh " situation, considering most of the denizens of this board are NOT privy to the details, maybe some more explanation is in order

Or, you know, you could just post some random oversimplification then be a toolbag when someone disagrees. That's way more entertaining at least !  =D
Title: Re: Junior dev manages to cripple/kill company on first day
Post by: Fitz on June 14, 2017, 11:09:53 AM
Woops
Title: Re: Junior dev manages to cripple/kill company on first day
Post by: K Frame on June 14, 2017, 11:11:07 AM
The concept for the cloud has been around since the first mainframe days of the 1950s. And you're still wrong. The cloud is not just about infrastructure. Infrastructure is part of it. Capabilities and capacities are also part of it. And data storage is at its very core.

And gee. I'm the only person here to every hyper simplify something. I'm such an evil, evil person.  ;/

Title: Re: Junior dev manages to cripple/kill company on first day
Post by: Fitz on June 14, 2017, 11:12:03 AM
The concept for the cloud has been around since the first mainframe days of the 1950s. And you're still wrong. The cloud is not just about infrastructure. Infrastructure is part of it. Capabilities and capacities are also part of it. And data storage is at its very core.

And gee. I'm the only person here to every hyper simplify something. I'm such an evil, evil person.  ;/



The cloud is not just about infrastructure. Wait a sec

You agreeing with me now?

What did you say here that differed from my first reply?

Better yet , what did you say here that's a bit different than YOUR first reply?

Title: Re: Junior dev manages to cripple/kill company on first day
Post by: K Frame on June 14, 2017, 11:15:21 AM
You miss this or something?

"You mean that there's actually other stuff involved, mechanisms, processes, procedures, blah blah blah that make "the cloud" work?Huh??

Well DUH!"

That's sarcasm and mockery, Sheldon.
Title: Re: Junior dev manages to cripple/kill company on first day
Post by: Fitz on June 14, 2017, 11:17:18 AM
No i didn't miss that, I already addressed it. I.e. It's different than your first reply in this thread, which was wrong


Title: Re: Junior dev manages to cripple/kill company on first day
Post by: Fitz on June 14, 2017, 11:18:36 AM
So you come in a thread, say something inaccurate, then later say something accurate while being a cock

Got it

  =D

As long as I'm clear on your tactics. I'll keep it in mind for the future
Title: Re: Junior dev manages to cripple/kill company on first day
Post by: BobR on June 14, 2017, 11:21:01 AM
I know nothing about the cloud except it is some place above where all the data goes to die(?).

(https://cloudaccesssecurity.files.wordpress.com/2012/10/skull.jpg)

;)  :)

bob
Title: Re: Junior dev manages to cripple/kill company on first day
Post by: K Frame on June 14, 2017, 11:22:59 AM
"All it means is that your data is now on off-site servers, which are out of your control. They're not electron bits floating freely around in "a cloud.'"

OK, so what's so inaccurate about these statements?


Is cloud data stored on off-site servers? Generally yes. It is. Unless you're running your own internal cloud. Then it's not really a cloud.

Are these servers out of your control? Unless you're running your own internal cloud (which again, isn't really a cloud) yes they are. You're taking it on faith that the vendors are doing their due diligence.

Is your data actually floating around freely in white puffy clouds? Are you an idiot?


So, explain in great detail, how ANY of that is inaccurate for the purpose of cloud computing/cloud storage?

EDUCATE ME, CLOUD MASTER!



Oh, wait, you mean I didn't acknowledge that there's an architecture behind the cloud? So that must mean I think all of that data, and all of those applications, are transported via Amish horse and buggy, right?
Title: Re: Junior dev manages to cripple/kill company on first day
Post by: Fitz on June 14, 2017, 11:30:02 AM
"All it means is that your data is now on off-site servers, which are out of your control. They're not electron bits floating freely around in "a cloud.'"

OK, so what's so inaccurate about these statements?


Is cloud data stored on off-site servers? Generally yes. It is. Unless you're running your own internal cloud. Then it's not really a cloud.

Are these servers out of your control? Unless you're running your own internal cloud (which again, isn't really a cloud) yes they are. You're taking it on faith that the vendors are doing their due diligence.

Is your data actually floating around freely in white puffy clouds? Are you an idiot?


So, explain in great detail, how ANY of that is inaccurate for the purpose of cloud computing/cloud storage?

EDUCATE ME, CLOUD MASTER!



Oh, wait, you mean I didn't acknowledge that there's an architecture behind the cloud? So that must mean I think all of that data, and all of those applications, are transported via Amish horse and buggy, right?


No,


AJ mentioned his company was moving into the cloud. You went into "lol the cloud" and "it's not just data floating around."

Which AJ didn't say was the case. You wanted to come into a thread and ridicule someone for using an industry colloquialism for... whatever reason.

you said "all it means"

Which is inaccurate. If you'll read my very first reply on the subject, you'll find nothing inaccurate there. Your post missed many key points, which mght be helpful for the large population of the board who aren't industry folks. Which is why it's silly to play the "lol it's just someone elses computer" card. It's misleading to people who aren't experts, and that's how you get dumb appliances with two hard drives selling for 800 bucks because they're a "personal cloud"


I really don't understand why you've turned this thread into a vehicle to call people idiots and be a jerk. Each successive post from you has been dripping with derision, and i'm not quite sure what anyone has done to earn it.

You should chill out
Title: Re: Junior dev manages to cripple/kill company on first day
Post by: Fitz on June 14, 2017, 11:33:17 AM
a private cloud CAN be a cloud. Being open to the public is irrelevant... go read my first post again
Title: Re: Junior dev manages to cripple/kill company on first day
Post by: K Frame on June 14, 2017, 11:46:54 AM
"Your post missed many key points,"

OMG! You mean that NO ONE at APS can use any sort of simplification for anything anymore -- that they must explain every concept and nuance in exacting, precise, detail -- because someone else might not get something?

I guess APS needs a new motto...

TO SIMPLIFY IS TO SIN

or maybe...

BABY JESUS CRIES WHEN THE EXPLANATION ISN'T DOGMATIC

Title: Re: Junior dev manages to cripple/kill company on first day
Post by: Fitz on June 14, 2017, 11:55:06 AM
Yeah, you're not mad at all.

you know what this thread needs? More sputtering, frothing caps lock and logical fallacies from mike Irwin.
Title: Re: Junior dev manages to cripple/kill company on first day
Post by: Fitz on June 14, 2017, 11:56:57 AM
Nobody said you needed to explain everything in detail, guy.

You explained nothing. You did it deliberately, so you could feel superior.
Title: Re: Junior dev manages to cripple/kill company on first day
Post by: K Frame on June 14, 2017, 12:06:40 PM
No. I just didn't feel like meeting your standards of anal exactitude by providing a 40,000 word essay on the cloud, its architecture, and its operation.

Here's your next data delivery from Amish Cloud Services, Inc. :rofl:

https://www.youtube.com/watch?v=RUFL6tWlf5A
Title: Re: Junior dev manages to cripple/kill company on first day
Post by: K Frame on June 14, 2017, 12:24:08 PM
I would be very remiss, though, in pointing out the Cloud's greatest benefit to society worldwide...

Virtually unlimited access to porn.
Title: Re: Junior dev manages to cripple/kill company on first day
Post by: zxcvbob on June 14, 2017, 12:50:02 PM
(https://armedpolitesociety.com/proxy.php?request=http%3A%2F%2Fi156.photobucket.com%2Falbums%2Ft33%2FAJ_Dual%2Fold_zps2e7s9bhm.jpg&hash=98e3b09d2946689464747357ae33cca2d7538a6d) (http://s156.photobucket.com/user/AJ_Dual/media/old_zps2e7s9bhm.jpg.html)
Title: Re: Junior dev manages to cripple/kill company on first day
Post by: K Frame on June 14, 2017, 12:50:44 PM
And I was wearing an onion on my belt, which was the style of the time...  :old:
Title: Re: Junior dev manages to cripple/kill company on first day
Post by: RevDisk on June 14, 2017, 02:47:32 PM
I really don't understand why you've turned this thread into a vehicle to call people idiots and be a jerk. Each successive post from you has been dripping with derision, and i'm not quite sure what anyone has done to earn it.

You should chill out

It's mike. Angrier and angrier is kinda his trademark.


(https://i.imgur.com/tjbAdUE.png)

(https://i.imgur.com/nUMfeU3.jpg)
Title: Re: Junior dev manages to cripple/kill company on first day
Post by: makattak on June 14, 2017, 02:53:04 PM
(https://armedpolitesociety.com/proxy.php?request=http%3A%2F%2Fi156.photobucket.com%2Falbums%2Ft33%2FAJ_Dual%2Fold_zps2e7s9bhm.jpg&hash=98e3b09d2946689464747357ae33cca2d7538a6d) (http://s156.photobucket.com/user/AJ_Dual/media/old_zps2e7s9bhm.jpg.html)

 :rofl:

Someone set this up on purpose. Right?  :laugh:
Title: Re: Junior dev manages to cripple/kill company on first day
Post by: TechMan on June 14, 2017, 04:56:50 PM
And this has run its course.