Oh crap. Heartbleed or why most servers in the world are in a bad way

[Phyphor]

http://arstechnica.com/security/2014/04/critical-crypto-bug-exposes-yahoo-mail-passwords-russian-roulette-style/

Lest readers think "catastrophic" is too exaggerated a description for the critical defect affecting an estimated two-thirds of the Internet's Web servers, consider this: at the moment this article was being prepared, the so-called Heartbleed bug was exposing end-user passwords, the contents of confidential e-mails, and other sensitive data belonging to Yahoo Mail and almost certainly countless other services.

The two-year-old bug is the result of a mundane coding error in OpenSSL, the world's most popular code library for implementing HTTPS encryption in websites, e-mail servers, and applications. The result of a missing bounds check in the source code, Heartbleed allows attackers to recover large chunks of private computer memory that handle OpenSSL processes. The leak is the digital equivalent of a grab bag that hackers can blindly reach into over and over simply by sending a series of commands to vulnerable servers. The returned contents could include something as banal as a time stamp, or it could return far more valuable assets such as authentication credentials or even the private key at the heart of a website's entire cryptographic certificate.

Underscoring the urgency of the problem, a conservatively estimated two-thirds of the Internet's Web servers use OpenSSL to cryptographically prove their legitimacy and to protect passwords and other sensitive data from eavesdropping. Many more e-mail servers and end-user computers rely on OpenSSL to encrypt passwords, e-mail, instant messages, and other sensitive data. OpenSSL developers have released version 1.0.1g that readers should install immediately on any vulnerable machines they maintain. But given the stakes and the time it takes to update millions of servers, the risks remain high.
Enter Yahoo Mail

For an idea of the type of information that remains available to anyone who knows how to use open source tools like this one, just consider Yahoo Mail, the world's most widely used Web mail service. The images below were recovered by Mark Loman, a malware and security researcher with no privileged access to Yahoo Mail servers. The plaintext passwords appearing in them have been obscured to protect the Yahoo Mail users they belong to, a courtesy not everyone exploiting this vulnerability is likely to offer. To retrieve them, Loman sent a series of requests to servers running Yahoo Mail at precisely the same time as the credentials just happened to be stored—Russian roulette-style—in Yahoo memory.

Now, this gets to hit the mainstream.

Of course, before you use a given server, you should check it for the vulnerability.  Do so here: http://filippo.io/Heartbleed

[RevDisk]

https://rhn.redhat.com/errata/RHSA-2014-0376.html
http://lists.centos.org/pipermail/centos-announce/2014-April/020249.html

For RHEL/Scientific/CentOS/etc:

 httpd              x86_64      2.2.15-30.el6.centos         updates      821 k
 httpd-tools        x86_64      2.2.15-30.el6.centos         updates       73 k
 mod_ssl            x86_64      1:2.2.15-30.el6.centos       updates       91 k
 openssl            x86_64      1.0.1e-16.el6_5.7            updates      1.5 M
 openssl-devel      x86_64      1.0.1e-16.el6_5.7            updates      1.2 M

[mtnbkr]

Check apps as well.  We run CentOS 5.9, which is OK, but Splunk 6 comes with its own OpenSSL binaries, which are vulnerable.  So, I have a couple systems which are affected even though the OS isn't.

Chris

[Nick1911]

Check apps as well.  We run CentOS 5.9, which is OK, but Splunk 6 comes with its own OpenSSL binaries, which are vulnerable.  So, I have a couple systems which are affected even though the OS isn't.

Chris

The splunk server, or agents?

[mtnbkr]

Server.  We don't use the agents, so no clue there.  No mention from Splunk on how to fix.   I opened a case with them.

It' a big deal for me because we're in the process of upgrading 50-odd Splunk boxes from 4.3.6 to 6.02. 

Chris

[Scout26]

For those of us that don't speak geek and use yahoo for mail.  What does mean for us?  How can we check?  Is this only servers or does it heit our computers as well?

[lee n. field]

NSA's fault.

[mtnbkr]

For those of us that don't speak geek and use yahoo for mail.  What does mean for us?  How can we check?  Is this only servers or does it heit our computers as well?

You're fooked if they use an affected version of OpenSSL. 

OpenSSL is what encrypts the traffic between your computer and the server.  This could affect any website where you use httpS in the URL.  Email, banking, web forums, etc.

Chris

[Balog]

You're fooked if they use an affected version of OpenSSL. 

OpenSSL is what encrypts the traffic between your computer and the server.  This could affect any website where you use httpS in the URL.  Email, banking, web forums, etc.

Chris

And there is nothing an end user can do, until they correctly patch their servers right?

[mtnbkr]

Don't log in?

Chris

[Nick1911]

Looks like the 6.0.2 agents use 1.0.1e openssl.

Aaannnddd... the agent listens to a port, which tls connections can be made to, and heartbeat can be initialized on....   

[mtnbkr]

and we're getting <crickets> from Splunk.

Chris

[RevDisk]

For those of us that don't speak geek and use yahoo for mail.  What does mean for us?  How can we check?  Is this only servers or does it heit our computers as well?

What does mean for us?
OpenSSL is the clue that binds a lot of the internet's security together. It's what powers the S in https. It's also used in lots of other packages. And not much directly, it's a server thing. For the last two years, folks may or may not have been able to read 64 kilobytes of data from a server's RAM surreptitiously. There's no way of knowing if a server was compromised or not, as there is no logging trail.

How can we check?
Make sure your programs are always up to date.

Is this only servers or does it heit our computers as well?
Yes.
It is only OpenSSL servers, but that could theoretically include your computer if you run software that acts as an OpenSSL server. Main concern is on what you traditionally think of as servers, however.

[RevDisk]

Looks like the 6.0.2 agents use 1.0.1e openssl.

Aaannnddd... the agent listens to a port, which tls connections can be made to, and heartbeat can be initialized on....  

Just a note to others (Nick obviously knows this), version number is not necessarily indicative of a problem or not. openssl-1.0.1g is patched so using that is fine, but some distributions patch existing versions to maintain consistency. For example, openssl-1.0.1e-16.el6_5.7.x86_64.rpm is the patched RPM for RHEL, CentOS, etc. Patched version for Ubuntu 13.10 and related is 1.0.1e-3ubuntu1.2.

If you use software that uses OpenSSL such as OpenVPN (or Barracuda appliances), apache, etc, contact your vendor.

List of major apps:
http://www.openssl.org/related/apps.html

[Balog]

Don't log in?

Chris

If a malicious entity has compromised my account and password, would that accomplish anything? Not being a smart ass here, I don't quite grok what you mean.

[RevDisk]

If a malicious entity has compromised my account and password, would that accomplish anything? Not being a smart ass here, I don't quite grok what you mean.

He's correct. This bug mostly revolves around reading a server's memory. Not logging in does lower your exposure.
But you're also correct. If a malicious entity compromised the entire server, it won't accomplish much by not logging in.

[Balog]

He's correct. This bug mostly revolves around reading a server's memory. Not logging in does lower your exposure.
But you're also correct. If a malicious entity compromised the entire server, it won't accomplish much by not logging in.

I was wondering if frequent password changes might be helpful since it appears to be blind chance on if they get your info.

[vaskidmark]

For those of us that don't speak geek and use yahoo for mail.  What does mean for us?  How can we check?  Is this only servers or does it heit our computers as well?

I talked with my geek friend earlier today while they were carrying him, babbling, out of the building after he had tried explaining the implications to his CIO.  From what I understood there is nothing to worry about as all your bank accounts have already been drained, there are at least three dozen Russian women who have proof you have offered to marry them, and enough Cialis has been shipped to you home to last you a couple of years - if you live that long.

Seriously, he took a break from trying to bring his CIO up to speed on what the information getting out to the public (in his case meaning all his company's clients) will do to overtime in soothing their furrowed brows.  I had never before seen a quadruple espresso Cuban coffee knocked back in one shot.  It was impressive.

I'm just waiting for one of the other shoes to be dropped - that it's not just Yahoo accounts that folks need to be worried about.

stay safe.

[Scout26]

Of course, before you use a given server, you should check it for the vulnerability.  Do so here: http://filippo.io/Heartbleed

So I just go and type in "mybank.com" or do I need the actual "342.67.58.0.0.1" address?

[Nick1911]

So I just go and type in "mybank.com" or do I need the actual "342.67.58.0.0.1" address?

Honestly, I think it's far more likely for the vulnerability to appear some minor API endpoint somewhere then the public facing webservers.  Or in our case, a third party app that's installed on the servers to monitor logs.  In other words - it's a storm that will need to blow over, and there isn't a lot the end user can do about it.

[Fitz]

Honestly, I think it's far more likely for the vulnerability to appear some minor API endpoint somewhere then the public facing webservers.  Or in our case, a third party app that's installed on the servers to monitor logs.  In other words - it's a storm that will need to blow over, and there isn't a lot the end user can do about it.

This. Especially with all the reverse proxies, federated auth, and stuff that makes up the modern web.

Watch your accounts closely, change passwords frequently. Get a password management tool and use long randoms for a while that you change often.

Additionally, if a particular service you have has 2 factor auth, enable it. FOr a LOT of stuff, (paypal, my bank, even google and whatnot), 2 factor is available. I get a text before i can log in

[mtnbkr]

Honestly, I think it's far more likely for the vulnerability to appear some minor API endpoint somewhere then the public facing webservers.  Or in our case, a third party app that's installed on the servers to monitor logs.  In other words - it's a storm that will need to blow over, and there isn't a lot the end user can do about it.

What sort of hardware do you build your Splunk boxes on?  My company runs a managed service based on Splunk and historically designed the boxes around storage rather than Splunk performance (each box had 27tb, but used slow 7200rpm drives in a RAID5 array).  I'm in the process of designing a new system that will be Splunk-foward for the customers who need less storage, but better performance for queries and reporting.  Splunk needs lots of disk IO for that.  

My indexers will be a Dell 720 with 48gb RAM and 16 1.2tb 10k rpm drives in a RAID 1+0 array.  The Search Heads will be the same box, but with 4x 800gb SSD in a RAID 1+0 array.  For long term storage, we're going to use an NFS mount with the original boxes (720s with 27tb storage and 24gb RAM), where raw disk IO isn't important.  The Search Head will talk to the indexers over 10gbBT, incoming logs for the indexers will come in via a 2nd 10gbBT, and user access to the Search Head is over 10gbBT.  Management access and indexer-to-archive will each have their own dedicated 1gbBT links.

Believe it or not, some of my Splunk-savvy customers need that sort of system to get the performance they demand.  The one customer driving this is excited about the initial design.  It'll actually be cheaper for them, hardware-wise, than the existing estate which requires more of the older boxes to keep up with their Splunk needs.

BTW, I'm going to start working on a Splunk Certified Architect cert in a few weeks.

ETA: The new design is modular.  If we need more archive, just add an archive server to the network and configure NFS, diddle some scripts, and away you go.  If more indexing capacity is needed, just add another indexer.  Previously, you had to replace and entire box and migrate Splunk the archive data, etc. 

Chris

[RevDisk]

Uh. Wow. That's a lot of data, mtnbkr. I'm assuming web traffic?

I know Splunk can be used to index damn near anything, but I've seen people use it for either log analytics or some specific niche data stream (sensors, usually). I was planning on using it to hack together basically a Solar Winds alternative (sorta). Basically mapping port to mac to ip to device to device characteristics to keep a map of what is where and when.

[mtnbkr]

Uh. Wow. That's a lot of data, mtnbkr. I'm assuming web traffic?

I know Splunk can be used to index damn near anything, but I've seen people use it for either log analytics or some specific niche data stream (sensors, usually). I was planning on using it to hack together basically a Solar Winds alternative (sorta). Basically mapping port to mac to ip to device to device characteristics to keep a map of what is where and when.

Just logs.  My customers send IDS/IPS, FW, routers, proxies, various server logs, etc.  They have retention periods running from 90 days to "forever".  That's just raw data, then you have the reporting/auditing requirements ("I need to see what Joe has done on the network for the last year" or "show me every time this IP connected to that IP and what type of traffic it sent") and the fact that some queries can take a day or more to run due to the volume.  Some of my customers turn Splunk into a lightweight SIEM or use it to monitor devices by output (ie send an email if you don't see logs for more than X min).  They do all kinds of crazy things with their data.  Splunk makes that possible and even easy though.  It's a framework and there's so much you can do with it, it isn't funny.  You can tie all kinds of things together (how about tying orders on a web commerce site to unsolicited feedback posted to twitter from the happy or unhappy customer?)

Splunk gets nosebleed expensive at these levels.  Several of my customers have 500gb and up to 1tb licenses (spread over multiple indexers).  That allows them to index that much per day.  And yes, they use every bit of the licensed capacity.

Chris

[Nick1911]

What sort of hardware do you build your Splunk boxes on?  

I don't have much insight into it, honestly.  This company is big enough that providing hardware is a whole different department, as is systems administration.  We get VM's handed to us generally.  There is a whole team that manages the splunk front end, but my team is responsible for the agents on our servers.