Don’t judge the risk by the logo

It’s been almost a year since the OpenSSL Heartbleed vulnerability, a flaw which started a trend of the branded vulnerability, changing the way security vulnerabilities affecting open-source software are being reported and perceived. Vulnerabilities are found and fixed all the time, and just because a vulnerability gets a name and a fancy logo doesn’t mean it is of real risk to users.

ven1

So let’s take a tour through the last year of vulnerabilities, chronologically, to see what issues got branded and which issues actually mattered for Red Hat customers.

“Heartbleed” (April 2014)CVE-2014-0160

Heartbleed was an issue that affected newer versions of OpenSSL. It was a very easy to exploit flaw, with public exploits released soon after the issue was public. The exploits could be run against vulnerable public web servers resulting in a loss of information from those servers. The type of information that could be recovered varied based on a number of factors, but in some cases could include sensitive information. This flaw was widely exploited against unpatched servers.

For Red Hat Enterprise Linux, only customers running version 6.5 were affected as prior versions shipped earlier versions of OpenSSL that did not contain the flaw.

Apache Struts 1 Class Loader RCE (April 2014) CVE-2014-0114

This flaw allowed attackers to manipulate exposed ClassLoader properties on a vulnerable server, leading to remote code execution. Exploits have been published but they rely on properties that are exposed on Tomcat 8, which is not included in any supported Red Hat products. However, some Red Hat products that ship Struts 1 did expose ClassLoader properties that could potentially be exploited.

Various Red Hat products were affected and updates were made available.

OpenSSL CCS Injection (June 2014) CVE-2014-0224

After Heartbleed, a number of other OpenSSL issues got attention. CCS Injection was a flaw that could allow an attacker to decrypt secure connections. This issue is hard to exploit as it requires a man in the middle attacker who can intercept and alter network traffic in real time, and as such we’re not aware of any active exploitation of this issue.

Most Red Hat Enterprise Linux versions were affected and updates were available.

glibc heap overflow (July 2014) CVE-2014-5119

A flaw was found inside the glibc library where an attacker who is able to make an application call a specific function with a carefully crafted argument could lead to arbitrary code execution. An exploit for 32-bit systems was published (although this exploit would not work as published against Red Hat Enterprise Linux).

Some Red Hat Enterprise Linux versions were affected, in various ways, and updates were available.

JBoss Remoting RCE (July 2014) CVE-2014-3518

A flaw was found in JBoss Remoting where a remote attacker could execute arbitrary code on a vulnerable server. A public exploit is available for this flaw.

Red Hat JBoss products were only affected by this issue if JMX remoting is enabled, which is not the default. Updates were made available.

“Poodle” (October 2014) CVE-2014-3566

Continuing with the interest in OpenSSL vulnerabilities, Poodle was a vulnerability affecting the SSLv3 protocol. Like CCS Injection, this issue is hard to exploit as it requires a man in the middle attack. We’re not aware of active exploitation of this issue.

Most Red Hat Enterprise Linux versions were affected and updates were available.

“ShellShock” (September 2014) CVE-2014-6271

The GNU Bourne Again shell (Bash) is a shell and command language interpreter used as the default shell in Red Hat Enterprise Linux. Flaws were found in Bash that could allow remote code execution in certain situations. The initial patch to correct the issue was not sufficient to block all variants of the flaw, causing distributions to produce more than one update over the course of a few days.

Exploits were written to target particular services. Later, malware circulated to exploit unpatched systems.

Most Red Hat Enterprise Linux versions were affected and updates were available.

RPM flaws (December 2014) CVE-2013-6435, CVE-2014-8118

Two flaws were found in the package manager RPM. Either could allow an attacker to modify signed RPM files in such a way that they would execute code chosen by the attacker during package installation. We know CVE-2013-6435 is exploitable, but we’re not aware of any public exploits for either issue.

Various Red Hat Enterprise Linux releases were affected and updates were available.

“Turla” malware (December 2014)

Reports surfaced of a trojan package targeting Linux, suspected as being part of an “advance persistent threat” campaign. Our analysis showed that the trojan was not sophisticated, was easy to detect, and unlikely part of such a campaign.

The trojan does not use any vulnerability to infect a system, it’s introduction onto a system would be via some other mechanism. Therefore it does not have a CVE name and no updates are applicable for this issue.

“Grinch” (December 2014)

An issue was reported which gained media attention, but was actually not a security vulnerability. No updates were applicable for this issue.

“Ghost” (January 2015) CVE-2015-0235

A bug was found affecting certain function calls in the glibc library. A remote attacker that is able to make an application call to an affected function could execute arbitrary code. While a proof of concept exploit is available, not many applications were found to be vulnerable in a way that would allow remote exploitation.

Red Hat Enterprise Linux versions were affected and updates were available.

“Freak” (March 2015) CVE-2015-0204

It was found that OpenSSL clients accepted EXPORT-grade (insecure) keys even when the client had not initially asked for them. This could be exploited using a man-in-the-middle attack, which could downgrade to a weak key, factor it, then decrypt communication between the client and the server. Like Poodle and CCS Injection, this issue is hard to exploit as it requires a man in the middle attack. We’re not aware of active exploitation of this issue.

Red Hat Enterprise Linux versions were affected and updates were available.

Other issues of customer interest

We can also get a rough guide of which issues are getting the most attention by looking at the number of page views on the Red Hat CVE pages. While the top views were for the  issues above, also of increased interest was:

  • A kernel flaw (May 2014) CVE-2014-0196, allowing local privilege escalation. A public exploit exists for this issue but does not work as published against Red Hat Enterprise Linux.
  • “BadIRET”, a kernel flaw (December 2014) CVE-2014-9322, allowing local privilege escalation. Details on how to exploit this issue have been discussed, but we’re not aware of any public exploits for this issue.
  • A flaw in BIND (December 2014), CVE-2014-8500. A remote attacker could cause a denial of service against a BIND server being used as a recursive resolver.  Details that could be used to craft an exploit are available but we’re not aware of any public exploits for this issue.
  • Flaws in NTP (December 2014), including CVE-2014-9295. Details that could be used to craft an exploit are available.  These serious issues had a reduced impact on Red Hat Enterprise Linux.
  • A flaw in Samba (February 2015) CVE-2015-0240, where a remote attacker could potentially execute arbitrary code as root. Samba servers are likely to be internal and not exposed to the internet, limiting the attack surface. No exploits that lead to code execution are known to exist, and some analyses have shown that creation of such a working exploit is unlikely.

Conclusion

ven2

We’ve shown in this post that for the last year of vulnerabilities affecting Red Hat products the issues that matter and the issues that got branded do have an overlap, but they certainly don’t closely match. Just because an issue gets given a name, logo, and press attention does not mean it’s of increased risk. We’ve also shown there were some vulnerabilities of increased risk that did not get branded.

At Red Hat, our dedicated Product Security team analyse threats and vulnerabilities against all our products every day, and provide relevant advice and updates through the customer portal. Customers can call on this expertise to ensure that they respond quickly to address the issues that matter, while avoiding being caught up in a media whirlwind for those that don’t.

Reactive Product Security at Red Hat

The goal of Product Security at Red Hat is “to help protect customers from meaningful security concerns when using Red Hat products and services.” What does that really mean and how do we go about it? In this blog, we take a look at how Red Hat handles security vulnerabilities and what we do to reduce risk to our customers.

In 2001, we founded a dedicated security team within Red Hat to handle product security. Back then, we really had just one product line, the Red Hat® Linux® distribution. Now, 14 years later, we support well over 100 different products and versions from Red Hat Enterprise Linux, to OpenStack®, to Docker. In addition to handling the reaction to vulnerabilities found in our products, we also proactively work on improving security for the future. An upcoming blog post will highlight some of those activities.

All software, no matter what the license, provenance, or supply-chain involved, have bugs–mistakes in the code which introduce errors. Some of those errors may cause a program to behave differently than what is expected, others may cause a program to crash. Of these errors, a small proportion are classified as vulnerabilities if they pose a security risk where an attacker can deliberately cause a program to fail.

Our products are generally made up of many different open source components; for example, Red Hat Enterprise Linux 7 is composed of several thousand different packages and each one can be a separate open source project. Red Hat Product Security is accountable for knowing every component used in every product so we can keep track of the security issues. This has become an area of expertise for us and is recognized by the industry as handling vulnerabilities in third party software is not a trivial task.

It all starts with a team which monitors a number of sources to find out about security issues in such third party components. In a previous blog post, we gave some metrics for a years worth of vulnerabilities and showed that in nearly half of the vulnerabilities we fixed, we were aware in advance of the issue being made public. The biggest source of information regarding non-public issues was through the two-way relationships we have with upstream open source projects and our peer vendors. Additionally, 17% of all the issues we fixed were found internally by Red Hat through security audits by our Quality Engineering team or by the product engineering teams themselves.

The next step is to assess whether these issues affect any of our products and determine the severity of each one. We do this based on a technical assessment from our team of skilled researchers. In addition to the nature of the vulnerability itself and the types of exploits likely to operate against it, other considerations include which specific pieces of code are impacted, the sensitivity of the applications they support, and their potential degree of exposure. For any given a vulnerability in an Open Source component, different products across different vendors could be affected in different ways depending on the versions being used, what patches are included, and even how the package is compiled.

In order to manage this workload, the Product Security team makes use of a number of tools and workflow processes all built around the principles of GTD.

Depending on the severity of the issue and the life-cycle for the product, patches get created and updates prepared. For many of our products, our policy is to back-port fixes an approach that significantly reduces the potential for compatibility issues and the introduction of additional vulnerabilities while making it easier for customers to consume updates. These updates, together with our advisory text explaining the issue, make their way to customers as security errata.

We actively monitor the time it takes for vulnerabilities to pass through this entire process. For example, Red Hat Enterprise Linux 5, since its release in 2007, has had 98% of all critical flaw fixes available to customers either the same day or next calendar day, once the issue was known to the public. We make all of our data on this available so customers can determine metrics for their particular environment.

In practice, what this means is that a Red Hat subscription provides customers with guidance, stability, and security that can only come from Red Hat. For a given product, there is a single mechanism to get updates for security issues across all components and technologies included, no matter which open source project they came from. Products are supported with long life cycles, and we maintain security updates for open source components included even beyond their upstream end of life.

We’ve briefly shown that we have well-established processes to effectively manage vulnerabilities in open source software, and that we are effective in getting fixes for these issues to customers, but there’s more that we do on the reactive side of handling security events.

2014 will be remembered for a number of high profile vulnerabilities, including several in widely used open source components: Heartbleed, ShellShock, and Poodle. Where these affected Red Hat products, we provided fast updates to correct them. However, getting fast fixes out was only part of the value.

In September last year, serious issues were found in the UNIX-like shell, Bash, called ShellShock. During this incident, Red Hat customers also received:

  • Timely advice. By the time the issue went public, we had specific knowledge base articles on the Customer Portal explaining how products were affected, how to get and install the fixes, and how to determine if you were vulnerable to the issue. Our article, linked above, was the definitive source of information about the vulnerability–being cited by most news articles, Wikipedia, and even US-CERT. The knowledge base and blog were continually updated with the latest knowledge and best practices.
  • Industry-leading security expertise. After the original flaw in Bash was identified and fixed, a second issue was discovered in public. It was a Red Hat Product Security engineer who designed and wrote the comprehensive patch used by most vendors in fully addressing this issue.
  • Immediate support. The Red Hat Customer Portal had an alert on every page, with notifications, and our support staff had access to the technical information. We were ready to provide immediate support to customers.
  • Proactive notifications. For customers with products affected by the issue, we sent email notifications within the first few hours. This email provided a call to action and linked to our specific knowledge and fixes for this issue. Posts on our Red Hat Support social media channels also directed customers to our knowledge base articles and fixes.
  • A self-detection tool: We also released a self-detection tool via Red Hat Access Labs to allow customers to easily identify whether their environment was vulnerable.

We’d like customers to hear about these major security issues from us first and then be able to install the fix for the issue. When a significant security event occurs, customers can can come to Red Hat first, safe in the knowledge that we’ll be on top of the situation and be able to give specific, timely, calm, and technically-accurate advice on how the issues affect all of our products and services.

The Source of Vulnerabilities, How Red Hat finds out about vulnerabilities.

Red Hat Product Security track lots of data about every vulnerability affecting every Red Hat product. We make all this data available on our Measurement page and from time to time write various blog posts and reports about interesting metrics or trends.

One metric we’ve not written about since 2009 is the source of the vulnerabilities we fix. We want to answer the question of how did Red Hat Product Security first hear about each vulnerability?

Every vulnerability that affects a Red Hat product is given a master tracking bug in Red Hat bugzilla. This bug contains a whiteboard field with a comma separated list of metadata including the dates we found out about the issue, and the source. You can get a file containing all this information already gathered for every CVE. A few months ago we updated our ‘daysofrisk’ command line tool to parse the source information allowing anyone to quickly create reports like this one.

So let’s take a look at some example views of recent data: every vulnerability fixed in every Red Hat product in the 12 months up to 30th August 2014 (a total of 1012 vulnerabilities).

Firstly a chart just giving the breakdown of how we first found out about each issue: Sources of issues

  • CERT: Issues reported to us from a national cert like CERT/CC or CPNI, generally in advance of public disclosure
  • Individual: Issues reported to Red Hat Product Security directly by a customer or researcher, generally in advance of public disclosure
  • Red Hat: Issues found by Red Hat employees
  • Relationship: Issues reported to us by upstream projects, generally in advance of public disclosure
  • Peer vendors: Issues reported to us by other OS distributions, through relationships
    or a shared private forum
  • Internet: For issues not disclosed in advance we monitor a number of mailing lists and security web pages of upstream projects
  • CVE: If we’ve not found out about an issue any other way, we can catch it from the list of public assigned CVE names from Mitre

Next a breakdown of if we knew about the issue in advance. For the purposes of our reports we count knowing the same day of an issue as not knowing in advance, even though we might have had a few hours notice: Known in advanceThere are few interesting observations from this data:

  • Red Hat employees find a lot of vulnerabilities. We don’t just sit back and wait for others to find flaws for us to fix, we actively look for issues ourselves and these are found by engineering, quality assurance, as well as our security teams. 17% of all the issues we fixed in the year were found by Red Hat employees. The issues we find are shared back in advance where possible to upstream and other peer vendors (generally via the ‘distros’ shared private forum).
  • Relationships matter. When you are fixing vulnerabilities in third party software, having a relationship with the upstream makes a big difference. But
    it’s really important to note here that this should never be a one-way street, if an upstream is willing to give Red Hat information about flaws in advance,
    then we need to be willing to add value to that notification by sanity checking the draft advisory, checking the patches, and feeding back the
    results from our quality testing. A recent good example of this is the OpenSSL CCS Injection flaw; our relationship with OpenSSL gave us advance
    notice of the issue and we found a mistake in the advisory as well as a mistake in the patch which otherwise would have caused OpenSSL to have to have
    done a secondary fix after release. Only two of the dozens of companies prenotified about those OpenSSL issues actually added value back to OpenSSL.
  • Red Hat can influence the way this metric looks; without a dedicated security team a vendor could just watch what another vendor does and copy them,
    or rely on public feeds such as the list of assigned CVE names from Mitre. We can make the choice to invest to find more issues and build upstream relationships.

Enterprise Linux 5.10 to 5.11 risk report

Red Hat Enterprise Linux 5.11 was released this month (September 2014), eleven months since the release of 5.10 in October 2013. So, as usual, let’s use this opportunity to take a look back over the vulnerabilities and security updates made in that time, specifically for Red Hat Enterprise Linux 5 Server.

Red Hat Enterprise Linux 5 is in Production 3 phase, being over seven years since general availability in March 2007, and will receive security updates until March 31st 2017.

Errata count

The chart below illustrates the total number of security updates issued for Red Hat Enterprise Linux 5 Server if you had installed 5.10, up to and including the 5.11 release, broken down by severity. It’s split into two columns, one for the packages you’d get if you did a default install, and the other if you installed every single package.

Note that during installation there actually isn’t an option to install every package, you’d have to manually select them all, and it’s not a likely scenario. For a given installation, the number of package updates and vulnerabilities that affected your systems will depend on exactly what you selected during installation and which packages you have subsequently installed or removed.

Security errata 5.10 to 5.11 Red Hat Enterprise Linux 5 ServerFor a default install, from release of 5.10 up to and including 5.11, we shipped 41 advisories to address 129 vulnerabilities. 8 advisories were rated critical, 11 were important, and the remaining 22 were moderate and low.

For all packages, from release of 5.10 up to and including 5.11, we shipped 82 advisories to address 298 vulnerabilities. 12 advisories were rated critical, 29 were important, and the remaining 41 were moderate and low.

You can cut down the number of security issues you need to deal with by carefully choosing the right Red Hat Enterprise Linux variant and package set when deploying a new system, and ensuring you install the latest available Update release.

Critical vulnerabilities

Vulnerabilities rated critical severity are the ones that can pose the most risk to an organisation. By definition, a critical vulnerability is one that could be exploited remotely and automatically by a worm. However we also stretch that definition to include those flaws that affect web browsers or plug-ins where a user only needs to visit a malicious (or compromised) website in order to be exploited. Most of the critical vulnerabilities we fix fall into that latter category.

The 12 critical advisories addressed 33 critical vulnerabilities across just three different projects:

  • An update to NSS/NSPR: RHSA-2014:0916(July 2014). A race condition was found in the way NSS verified certain certificates which could lead to arbitrary code execution with the privileges of the user running that application.
  • Updates to PHP, PHP53: RHSA-2013:1813, RHSA-2013:1814
    (December 2013). A flaw in the parsing of X.509 certificates could allow scripts using the affected function to potentially execute arbitrary code. An update to PHP: RHSA-2014:0311
    (March 2014). A flaw in the conversion of strings to numbers could allow scripts using the affected function to potentially execute arbitrary code.
  • Updates to Firefox, RHSA-2013:1268 (September 2013), RHSA-2013:1476 (October 2013), RHSA-2013:1812 (December 2013), RHSA-2014:0132 (February 2014), RHSA-2014:0310 (March 2014), RHSA-2014:0448 (Apr 2014), RHSA-2014:0741 (June 2014), RHSA-2014:0919 (July 2014) where a malicious web site could potentially run arbitrary code as the user running Firefox.

Updates to correct 32 of the 33 critical vulnerabilities were available via Red Hat Network either the same day or the next calendar day after the issues were public.

Overall, for Red Hat Enterprise Linux 5 since release until 5.11, 98% of critical vulnerabilities have had an update available to address them available from the Red Hat Network either the same day or the next calendar day after the issue was public.

Other significant vulnerabilities

Although not in the definition of critical severity, also of interest are other remote flaws and local privilege escalation flaws:

  • A flaw in glibc, CVE-2014-5119, fixed by RHSA-2014:1110 (August 2014). A local user could use this flaw to escalate their privileges. A public exploit is available which targets the polkit application on 32-bit systems although polkit is not shipped in Red Hat Enterprise Linux 5. It may be possible to create an exploit for Red Hat Enterprise Linux 5 by targeting a different application.
  • Two flaws in squid, CVE-2014-4115, and CVE-2014-3609, fixed by RHSA-2014:1148 (September 2014). A remote attacker could cause Squid to crash.
  • A flaw in procmail, CVE-2014-3618, fixed by RHSA-2014:1172 (September 2014). A remote attacker could send an email with specially crafted headers that, when processed by formail, could cause procmail to crash or, possibly, execute arbitrary code as the user running formail.
  • A flaw in Apache Struts, CVE-2014-0114, fixed by RHSA-2014:0474 (April 2014). A remote attacker could use this flaw to manipulate the ClassLoader used by an application server running Stuts 1 potentially leading to arbitrary code execution under some conditions.
  • A flaw where yum-updatesd did not properly perform RPM signature checks, CVE-2014-0022, fixed by RHSA-2014:1004 (Jan 2014). Where yum-updatesd was configured to automatically install updates, a remote attacker could use this flaw to install a malicious update on the target system using an unsigned RPM or an RPM signed with an untrusted key.
  • A flaw in the kernel floppy driver, CVE-2014-1737, fixed by RHSA-2014:0740 (June 2014). A local user who has write access to /dev/fdX on a system with floppy drive could use this flaw to escalate their privileges. A public exploit is available for this issue. Note that access to /dev/fdX is by default restricted only to members of the floppy group.
  • A flaw in libXfont, CVE-2013-6462, fixed by RHSA-2014:0018 (Jan 2014). A local user could potentially use this flaw to escalate their privileges to root.
  • A flaw in xorg-x11-server, CVE-2013-6424, fixed by RHSA-2013:1868 (Dec 2013). An authorized client could potentially use this flaw to escalate their privileges to root.
  • A flaw in the kernel QETH network device driver, CVE-2013-6381, fixed by RHSA-2014:0285 (March 2014). A local, unprivileged user could potentially use this flaw to escalate their privileges. Note this device is only found on s390x architecture systems.

Note that Red Hat Enterprise Linux 5 was not affected by the OpenSSL issue, CVE-2014-0160, “Heartbleed”.

Previous update releases

We generally measure risk in terms of the number of vulnerabilities, but the actual effort in maintaining a Red Hat Enterprise Linux system is more related to the number of advisories we released: a single Firefox advisory may fix ten different issues of critical severity, but takes far less total effort to manage than ten separate advisories each fixing one critical PHP vulnerability.

To compare these statistics with previous update releases we need to take into account that the time between each update release is different. So looking at a default installation and calculating the number of advisories per month gives the following chart:

Security Errata per month to 5.10This data is interesting to get a feel for the risk of running Enterprise Linux 5 Server, but isn’t really useful for comparisons with other major versions, distributions, or operating systems — for example, a default install of Red Hat Enterprise Linux 4AS did not include Firefox, but 5 Server does. You can use our public security measurement data and tools, and run your own custom metrics for any given Red Hat product, package set, time scales, and severity range of interest.

See also:
5.10, 5.9, 5.8, 5.7, 5.6, 5.5, 5.4, 5.3, 5.2, and 5.1 risk reports.