Reactive Product Security at Red Hat

The goal of Product Security at Red Hat is “to help protect customers from meaningful security concerns when using Red Hat products and services.” What does that really mean and how do we go about it? In this blog, we take a look at how Red Hat handles security vulnerabilities and what we do to reduce risk to our customers.

In 2001, we founded a dedicated security team within Red Hat to handle product security. Back then, we really had just one product line, the Red Hat® Linux® distribution. Now, 14 years later, we support well over 100 different products and versions from Red Hat Enterprise Linux, to OpenStack®, to Docker. In addition to handling the reaction to vulnerabilities found in our products, we also proactively work on improving security for the future. An upcoming blog post will highlight some of those activities.

All software, no matter what the license, provenance, or supply-chain involved, have bugs–mistakes in the code which introduce errors. Some of those errors may cause a program to behave differently than what is expected, others may cause a program to crash. Of these errors, a small proportion are classified as vulnerabilities if they pose a security risk where an attacker can deliberately cause a program to fail.

Our products are generally made up of many different open source components; for example, Red Hat Enterprise Linux 7 is composed of several thousand different packages and each one can be a separate open source project. Red Hat Product Security is accountable for knowing every component used in every product so we can keep track of the security issues. This has become an area of expertise for us and is recognized by the industry as handling vulnerabilities in third party software is not a trivial task.

It all starts with a team which monitors a number of sources to find out about security issues in such third party components. In a previous blog post, we gave some metrics for a years worth of vulnerabilities and showed that in nearly half of the vulnerabilities we fixed, we were aware in advance of the issue being made public. The biggest source of information regarding non-public issues was through the two-way relationships we have with upstream open source projects and our peer vendors. Additionally, 17% of all the issues we fixed were found internally by Red Hat through security audits by our Quality Engineering team or by the product engineering teams themselves.

The next step is to assess whether these issues affect any of our products and determine the severity of each one. We do this based on a technical assessment from our team of skilled researchers. In addition to the nature of the vulnerability itself and the types of exploits likely to operate against it, other considerations include which specific pieces of code are impacted, the sensitivity of the applications they support, and their potential degree of exposure. For any given a vulnerability in an Open Source component, different products across different vendors could be affected in different ways depending on the versions being used, what patches are included, and even how the package is compiled.

In order to manage this workload, the Product Security team makes use of a number of tools and workflow processes all built around the principles of GTD.

Depending on the severity of the issue and the life-cycle for the product, patches get created and updates prepared. For many of our products, our policy is to back-port fixes an approach that significantly reduces the potential for compatibility issues and the introduction of additional vulnerabilities while making it easier for customers to consume updates. These updates, together with our advisory text explaining the issue, make their way to customers as security errata.

We actively monitor the time it takes for vulnerabilities to pass through this entire process. For example, Red Hat Enterprise Linux 5, since its release in 2007, has had 98% of all critical flaw fixes available to customers either the same day or next calendar day, once the issue was known to the public. We make all of our data on this available so customers can determine metrics for their particular environment.

In practice, what this means is that a Red Hat subscription provides customers with guidance, stability, and security that can only come from Red Hat. For a given product, there is a single mechanism to get updates for security issues across all components and technologies included, no matter which open source project they came from. Products are supported with long life cycles, and we maintain security updates for open source components included even beyond their upstream end of life.

We’ve briefly shown that we have well-established processes to effectively manage vulnerabilities in open source software, and that we are effective in getting fixes for these issues to customers, but there’s more that we do on the reactive side of handling security events.

2014 will be remembered for a number of high profile vulnerabilities, including several in widely used open source components: Heartbleed, ShellShock, and Poodle. Where these affected Red Hat products, we provided fast updates to correct them. However, getting fast fixes out was only part of the value.

In September last year, serious issues were found in the UNIX-like shell, Bash, called ShellShock. During this incident, Red Hat customers also received:

  • Timely advice. By the time the issue went public, we had specific knowledge base articles on the Customer Portal explaining how products were affected, how to get and install the fixes, and how to determine if you were vulnerable to the issue. Our article, linked above, was the definitive source of information about the vulnerability–being cited by most news articles, Wikipedia, and even US-CERT. The knowledge base and blog were continually updated with the latest knowledge and best practices.
  • Industry-leading security expertise. After the original flaw in Bash was identified and fixed, a second issue was discovered in public. It was a Red Hat Product Security engineer who designed and wrote the comprehensive patch used by most vendors in fully addressing this issue.
  • Immediate support. The Red Hat Customer Portal had an alert on every page, with notifications, and our support staff had access to the technical information. We were ready to provide immediate support to customers.
  • Proactive notifications. For customers with products affected by the issue, we sent email notifications within the first few hours. This email provided a call to action and linked to our specific knowledge and fixes for this issue. Posts on our Red Hat Support social media channels also directed customers to our knowledge base articles and fixes.
  • A self-detection tool: We also released a self-detection tool via Red Hat Access Labs to allow customers to easily identify whether their environment was vulnerable.

We’d like customers to hear about these major security issues from us first and then be able to install the fix for the issue. When a significant security event occurs, customers can can come to Red Hat first, safe in the knowledge that we’ll be on top of the situation and be able to give specific, timely, calm, and technically-accurate advice on how the issues affect all of our products and services.

The Source of Vulnerabilities, How Red Hat finds out about vulnerabilities.

Red Hat Product Security track lots of data about every vulnerability affecting every Red Hat product. We make all this data available on our Measurement page and from time to time write various blog posts and reports about interesting metrics or trends.

One metric we’ve not written about since 2009 is the source of the vulnerabilities we fix. We want to answer the question of how did Red Hat Product Security first hear about each vulnerability?

Every vulnerability that affects a Red Hat product is given a master tracking bug in Red Hat bugzilla. This bug contains a whiteboard field with a comma separated list of metadata including the dates we found out about the issue, and the source. You can get a file containing all this information already gathered for every CVE. A few months ago we updated our ‘daysofrisk’ command line tool to parse the source information allowing anyone to quickly create reports like this one.

So let’s take a look at some example views of recent data: every vulnerability fixed in every Red Hat product in the 12 months up to 30th August 2014 (a total of 1012 vulnerabilities).

Firstly a chart just giving the breakdown of how we first found out about each issue: Sources of issues

  • CERT: Issues reported to us from a national cert like CERT/CC or CPNI, generally in advance of public disclosure
  • Individual: Issues reported to Red Hat Product Security directly by a customer or researcher, generally in advance of public disclosure
  • Red Hat: Issues found by Red Hat employees
  • Relationship: Issues reported to us by upstream projects, generally in advance of public disclosure
  • Peer vendors: Issues reported to us by other OS distributions, through relationships
    or a shared private forum
  • Internet: For issues not disclosed in advance we monitor a number of mailing lists and security web pages of upstream projects
  • CVE: If we’ve not found out about an issue any other way, we can catch it from the list of public assigned CVE names from Mitre

Next a breakdown of if we knew about the issue in advance. For the purposes of our reports we count knowing the same day of an issue as not knowing in advance, even though we might have had a few hours notice: Known in advanceThere are few interesting observations from this data:

  • Red Hat employees find a lot of vulnerabilities. We don’t just sit back and wait for others to find flaws for us to fix, we actively look for issues ourselves and these are found by engineering, quality assurance, as well as our security teams. 17% of all the issues we fixed in the year were found by Red Hat employees. The issues we find are shared back in advance where possible to upstream and other peer vendors (generally via the ‘distros’ shared private forum).
  • Relationships matter. When you are fixing vulnerabilities in third party software, having a relationship with the upstream makes a big difference. But
    it’s really important to note here that this should never be a one-way street, if an upstream is willing to give Red Hat information about flaws in advance,
    then we need to be willing to add value to that notification by sanity checking the draft advisory, checking the patches, and feeding back the
    results from our quality testing. A recent good example of this is the OpenSSL CCS Injection flaw; our relationship with OpenSSL gave us advance
    notice of the issue and we found a mistake in the advisory as well as a mistake in the patch which otherwise would have caused OpenSSL to have to have
    done a secondary fix after release. Only two of the dozens of companies prenotified about those OpenSSL issues actually added value back to OpenSSL.
  • Red Hat can influence the way this metric looks; without a dedicated security team a vendor could just watch what another vendor does and copy them,
    or rely on public feeds such as the list of assigned CVE names from Mitre. We can make the choice to invest to find more issues and build upstream relationships.

Enterprise Linux 5.10 to 5.11 risk report

Red Hat Enterprise Linux 5.11 was released this month (September 2014), eleven months since the release of 5.10 in October 2013. So, as usual, let’s use this opportunity to take a look back over the vulnerabilities and security updates made in that time, specifically for Red Hat Enterprise Linux 5 Server.

Red Hat Enterprise Linux 5 is in Production 3 phase, being over seven years since general availability in March 2007, and will receive security updates until March 31st 2017.

Errata count

The chart below illustrates the total number of security updates issued for Red Hat Enterprise Linux 5 Server if you had installed 5.10, up to and including the 5.11 release, broken down by severity. It’s split into two columns, one for the packages you’d get if you did a default install, and the other if you installed every single package.

Note that during installation there actually isn’t an option to install every package, you’d have to manually select them all, and it’s not a likely scenario. For a given installation, the number of package updates and vulnerabilities that affected your systems will depend on exactly what you selected during installation and which packages you have subsequently installed or removed.

Security errata 5.10 to 5.11 Red Hat Enterprise Linux 5 ServerFor a default install, from release of 5.10 up to and including 5.11, we shipped 41 advisories to address 129 vulnerabilities. 8 advisories were rated critical, 11 were important, and the remaining 22 were moderate and low.

For all packages, from release of 5.10 up to and including 5.11, we shipped 82 advisories to address 298 vulnerabilities. 12 advisories were rated critical, 29 were important, and the remaining 41 were moderate and low.

You can cut down the number of security issues you need to deal with by carefully choosing the right Red Hat Enterprise Linux variant and package set when deploying a new system, and ensuring you install the latest available Update release.

Critical vulnerabilities

Vulnerabilities rated critical severity are the ones that can pose the most risk to an organisation. By definition, a critical vulnerability is one that could be exploited remotely and automatically by a worm. However we also stretch that definition to include those flaws that affect web browsers or plug-ins where a user only needs to visit a malicious (or compromised) website in order to be exploited. Most of the critical vulnerabilities we fix fall into that latter category.

The 12 critical advisories addressed 33 critical vulnerabilities across just three different projects:

  • An update to NSS/NSPR: RHSA-2014:0916(July 2014). A race condition was found in the way NSS verified certain certificates which could lead to arbitrary code execution with the privileges of the user running that application.
  • Updates to PHP, PHP53: RHSA-2013:1813, RHSA-2013:1814
    (December 2013). A flaw in the parsing of X.509 certificates could allow scripts using the affected function to potentially execute arbitrary code. An update to PHP: RHSA-2014:0311
    (March 2014). A flaw in the conversion of strings to numbers could allow scripts using the affected function to potentially execute arbitrary code.
  • Updates to Firefox, RHSA-2013:1268 (September 2013), RHSA-2013:1476 (October 2013), RHSA-2013:1812 (December 2013), RHSA-2014:0132 (February 2014), RHSA-2014:0310 (March 2014), RHSA-2014:0448 (Apr 2014), RHSA-2014:0741 (June 2014), RHSA-2014:0919 (July 2014) where a malicious web site could potentially run arbitrary code as the user running Firefox.

Updates to correct 32 of the 33 critical vulnerabilities were available via Red Hat Network either the same day or the next calendar day after the issues were public.

Overall, for Red Hat Enterprise Linux 5 since release until 5.11, 98% of critical vulnerabilities have had an update available to address them available from the Red Hat Network either the same day or the next calendar day after the issue was public.

Other significant vulnerabilities

Although not in the definition of critical severity, also of interest are other remote flaws and local privilege escalation flaws:

  • A flaw in glibc, CVE-2014-5119, fixed by RHSA-2014:1110 (August 2014). A local user could use this flaw to escalate their privileges. A public exploit is available which targets the polkit application on 32-bit systems although polkit is not shipped in Red Hat Enterprise Linux 5. It may be possible to create an exploit for Red Hat Enterprise Linux 5 by targeting a different application.
  • Two flaws in squid, CVE-2014-4115, and CVE-2014-3609, fixed by RHSA-2014:1148 (September 2014). A remote attacker could cause Squid to crash.
  • A flaw in procmail, CVE-2014-3618, fixed by RHSA-2014:1172 (September 2014). A remote attacker could send an email with specially crafted headers that, when processed by formail, could cause procmail to crash or, possibly, execute arbitrary code as the user running formail.
  • A flaw in Apache Struts, CVE-2014-0114, fixed by RHSA-2014:0474 (April 2014). A remote attacker could use this flaw to manipulate the ClassLoader used by an application server running Stuts 1 potentially leading to arbitrary code execution under some conditions.
  • A flaw where yum-updatesd did not properly perform RPM signature checks, CVE-2014-0022, fixed by RHSA-2014:1004 (Jan 2014). Where yum-updatesd was configured to automatically install updates, a remote attacker could use this flaw to install a malicious update on the target system using an unsigned RPM or an RPM signed with an untrusted key.
  • A flaw in the kernel floppy driver, CVE-2014-1737, fixed by RHSA-2014:0740 (June 2014). A local user who has write access to /dev/fdX on a system with floppy drive could use this flaw to escalate their privileges. A public exploit is available for this issue. Note that access to /dev/fdX is by default restricted only to members of the floppy group.
  • A flaw in libXfont, CVE-2013-6462, fixed by RHSA-2014:0018 (Jan 2014). A local user could potentially use this flaw to escalate their privileges to root.
  • A flaw in xorg-x11-server, CVE-2013-6424, fixed by RHSA-2013:1868 (Dec 2013). An authorized client could potentially use this flaw to escalate their privileges to root.
  • A flaw in the kernel QETH network device driver, CVE-2013-6381, fixed by RHSA-2014:0285 (March 2014). A local, unprivileged user could potentially use this flaw to escalate their privileges. Note this device is only found on s390x architecture systems.

Note that Red Hat Enterprise Linux 5 was not affected by the OpenSSL issue, CVE-2014-0160, “Heartbleed”.

Previous update releases

We generally measure risk in terms of the number of vulnerabilities, but the actual effort in maintaining a Red Hat Enterprise Linux system is more related to the number of advisories we released: a single Firefox advisory may fix ten different issues of critical severity, but takes far less total effort to manage than ten separate advisories each fixing one critical PHP vulnerability.

To compare these statistics with previous update releases we need to take into account that the time between each update release is different. So looking at a default installation and calculating the number of advisories per month gives the following chart:

Security Errata per month to 5.10This data is interesting to get a feel for the risk of running Enterprise Linux 5 Server, but isn’t really useful for comparisons with other major versions, distributions, or operating systems — for example, a default install of Red Hat Enterprise Linux 4AS did not include Firefox, but 5 Server does. You can use our public security measurement data and tools, and run your own custom metrics for any given Red Hat product, package set, time scales, and severity range of interest.

See also:
5.10, 5.9, 5.8, 5.7, 5.6, 5.5, 5.4, 5.3, 5.2, and 5.1 risk reports.

Enterprise Linux 6.4 to 6.5 risk report

Red Hat Enterprise Linux 6.5 was released last week (November 2013), nine months since the release of 6.4 in February 2013. In this report we take a look back over the vulnerabilities and security updates since that last update, specifically for Red Hat Enterprise Linux 6 Server.

Red Hat Enterprise Linux 6 is in its fourth year since release, and will receive security updates until November 30th 2020.

Errata count

The chart below illustrates the total number of security updates issued for Red Hat Enterprise Linux 6 Server if you had installed 6.4, up to and including the 6.5 release, broken down by severity. The chart is split into two columns, one for the packages you’d get if you did a default install, and the other if you installed every single package.

During installation, there actually isn’t an option to install every package, you’d have to manually select them all, which is not a likely scenario. For a given installation, the number of package updates and vulnerabilities that affected you will depend on exactly what you selected during installation and which packages you have subsequently installed or removed.

 

20131121a

 

For a default install, from the release of 6.4 up to and including 6.5, we shipped 54 advisories to address 228 vulnerabilities. 3 advisories were rated critical, 18 were important, and the remaining 33 were moderate and low.

For all packages, from the release of 6.4 up to and including 6.5, we shipped 127 advisories to address 399 vulnerabilities. 17 advisories were rated critical, 40 were important, and the remaining 70 were moderate and low.

You can cut down the number of security issues you need to deal with by carefully choosing the right Red Hat Enterprise Linux variant and package set when deploying a new system, and ensuring you install the latest available Update release.

 

Critical vulnerabilities

Vulnerabilities rated with a critical severity are the ones that can pose the most risk to an organisation. By definition, a critical vulnerability is one that could be exploited remotely and automatically by a worm. However, we also stretch that definition to include those flaws that affect web browsers or plug-ins where a user only needs to visit a malicious (or compromised) website in order to be exploited. Most of the critical vulnerabilities we fix fall into that latter category.

The 19 critical advisories addressed 83 critical vulnerabilities across four different projects:

  1. An update to PHP: RHSA-2013:1049 (Jul 2013). If a PHP application used a particular function to parse untrusted XML content, a remote attacker able to supply specially crafted XML could use this flaw to crash the application or, possibly, execute arbitrary code.
  2. Updates to the OpenJDK 7 Java Runtime: RHSA-2013:1451 (Oct 2013), RHSA-2013:0957 (Jun 2013), RHSA-2013:0751 (Apr 2013), RHSA-2013:0602 (Mar 2013), and OpenJDK 6 Java Runtime: RHSA-2013:0605 (Mar 2013), RHSA-2013:0273 (Feb 2013), RHSA-2013:0245 (Feb 2013), where a malicious web site presenting a Java applet could potentially run arbitrary code as the user running the web browser.
  3. Updates to Firefox/XULRunner: RHSA-2013:1476 (Oct 2013), RHSA-2013:1268 (Sep 2013), RHSA-2013:1140 (Aug 2013), RHSA-2013:0981 (Jun 2013), RHSA-2013:0820 (May 2013), RHSA-2013:0696 (Apr 2013), RHSA-2013:0614 (Mar 2013), RHSA-2013:0271 (Feb 2013), where a malicious web site could potentially run arbitrary code as the user running Firefox.
  4. An update to Thunderbird: RHSA-2013:0272 (Feb 2013), where a malicious email message could potentially run arbitrary code as the user running Thunderbird.

Updates to correct 60 of the 83 critical vulnerabilities (72%) were available via Red Hat Network either the same day or the next calendar day after the issues were public.

Not counting updates to OpenJDK, this number would have been 100%. Two OpenJDK updates both took 7 days due to a number of problems with the packages, including requiring respins after failing QA, and broken fixes and regressions from upstream.

Overall, for Red Hat Enterprise Linux 6, from the release of 6.0 up to and including 6.5, 90% of the critical vulnerabilities have had an update that addressed these issues available from the Red Hat Network either the same day or the next calendar day after the issue was public.

Other significant vulnerabilities

Although not in the definition of critical severity, also of interest are remote denial of service flaws and local privilege escalation flaws:

  • A flaw in dbus-glibc: CVE-2013-0292 fixed by RHSA-2013:0568 (Feb 2013). A local attacker could use this flaw to escalate their privileges to root. A public exploit exists for this issue.
  • A flaw in the Tomcat init scripts: CVE-2013-1976 fixed by RHSA-2013:0869, where a local user who could deploy a Tomcat application could gain root privileges. This issue is considered to be easily exploitable.
  • A race condition was found in PolicyKit. Packages that used PolicyKit authorization could have intended authorizations bypassed, or in certain conditions users could escalate their privileges: CVE-2013-4288 fixed by RHSA-2013:1270 in polkit, CVE-2013-4311 fixed by RHSA-2013:1272 in libvirt, CVE-2013-4324 fixed by RHSA-2013:1273 in spice-gtk, CVE-2013-4325 fixed by RHSA-2013:1274 in hplip, CVE-2013-4326 fixed by RHSA-2013:1282 in rtkit. A proof of concept for this flaw is available.
  • Various flaws in the kernel:
    • CVE-2013-2094, fixed by RHSA-2013:0830, (May 2013) could allow a local unprivileged user to gain root privileges; a public exploit exists for this issue.
    • CVE-2013-0268, fixed by RHSA-2013:0630, (Mar 2013) could allow a local root user who has limited capabilities the ability to further escalate their privileges; a public exploit exists for this issue.
    • CVE-2013-0871, fixed by RHSA-2013:0567, (Feb 2013) could allow a local unprivileged user to gain root privileges. In practice, our testing showed the involved race condition is hard to win, even with the publicly available exploit.
    • CVE-2013-0913, fixed by RHSA-2013:0744, (Apr 2013) could allow a local unprivileged user to gain root privileges. We have not seen a public exploit for this issue, but it was supposedly used in a competition as part of an exploit against Google Chrome OS.
    • CVE-2013-1935, fixed by RHSA-2013:0911, where an unprivileged KVM guest user could potentially crash the host.
    • Other kernel flaws were fixed that could potentially allow escalation of privileges, but either were hard to exploit, or currently have no public exploits available, such as CVE-2013-0228, CVE-2013-1773, CVE-2013-1796, CVE-2013-1797, CVE-2013-1943, and CVE-2013-2224.
  • Two flaws in BIND: CVE-2013-2266 fixed by RHSA-2013:0689 (Mar 2013), and CVE-2013-4854 fixed by RHSA-2013:1114 (Jul 2013). A remote attacker could use either of these flaws to cause BIND to crash. We are not aware of a specific exploits for these issues, but they could be easily created.
  • A flaw in the 389 Directory Server: CVE-2013-4283 fixed by RHSA-2013:1182, which could cause the Directory Server to crash. We are not aware of a public exploit for this issue.
  • A flaw in the GnuTLS library: CVE-2013-2116 fixed by RHSA-2013:0883. A remote attacker could use this flaw to crash a server application that uses GnuTLS. We are not aware of a public exploit for this issue.

Previous update releases

We generally measure risk in terms of the number of vulnerabilities, but the actual effort in maintaining a Red Hat Enterprise Linux system is more related to the number of advisories we released: a single Firefox advisory may fix ten different issues of critical severity, but takes far less total effort to manage than ten separate advisories each fixing one critical PHP vulnerability.

To compare these statistics with previous update releases, we need to take into account that the time between each update release is different. So looking at a default installation and calculating the number of advisories per month gives the following chart:20131121b

This data is interesting to get a feel for the risk of running Enterprise Linux 6 Server, but is not really useful for comparisons with other major versions, distributions, or operating systems — for example, a default install of Red Hat Enterprise Linux 6 Server does not include Firefox, but 5 Server does. You can use our public security measurement data and tools to run your own custom metrics for any given Red Hat product, package set, timescales, and severity range of interest.

See also: 6.4, 6.3, 6.2, and 6.1 risk reports.