Security improvements in Red Hat Enterprise Linux 7

Each new release of Red Hat® Enterprise Linux® is not only built on top of the previous version, but a large number of its components incorporate development from the Fedora distribution. For Red Hat Enterprise Linux 7, most components are aligned with Fedora 19, and with select components coming from Fedora 20. This means that users benefit from new development in Fedora, such as firewalld which is described below. While preparing the next release of Red Hat Enterprise Linux, we review components for their readiness for an enterprise-class distribution. We also make sure that we address known vulnerabilities before the initial release. And we review new components to check that they meet our standards regarding security and general suitability for enterprise use.

One of the first things that happens is a review of the material going into a new version of Red Hat Enterprise Linux. Each release includes new packages that Red Hat has never shipped before and anything that has never been shipped in a Red Hat product receives a security review. We look for various problems – from security bugs in the actual software to packaging issues. It’s even possible that some packages won’t make the cut if they prove to have issues that cannot be resolved in a manner we decide is acceptable. It’s also possible that a package was once included as a dependency or feature that is no longer planned for the release. Rather than leave those in the release, we do our best to remove the unneeded packages as they could result in security problems later down the road.

Previously fixed security issues are also reviewed to ensure nothing has been missed since the last version. While uncommon, it is possible that a security fix didn’t make it upstream, or was somehow dropped from a package at some point during the move between major releases. We spend time reviewing these to ensure nothing important was missed that could create problems later.

Red Hat Product Security also adds several new security features in order to better protect the system.

Before its 2011 revision, the C++ language definition was ambiguous as to what should happen if an integer overflow occurs during the size computation of an array allocation. The C++ compiler in Red Hat Enterprise Linux 7 will perform a size check (and throw std::bad_alloc on failure) if the size (in bytes) of the allocated array exceeds the width of a register, even in C++98 mode. This change affects the code generated by the compiler–it is not a library-level correction. Consequently, we compiled all of Red Hat Enterprise Linux 7 with a compiler version that performs this additional check.

When we compiled Red Hat Enterprise Linux 7, we also tuned the compiler to add “stack protector” instrumentation to additional functions. The GCC compiler in Red Hat Enterprise Linux 6 used heuristics to determine whether a function warrants “stack protector” instrumentation. In contrast, the compiler in Red Hat Enterprise Linux 7 uses precise rules that add the instrumentation to only those functions that need it. This allowed us to instrument additional functions with minimal performance impact, extending this probabilistic defense against stack-based buffer overflows to an even larger part of the code base.

Red Hat Enterprise Linux 7 also includes firewalld. firewalld allows for centralized firewall management using high-level concepts, such as zones. It also extends spoofing protection based on reverse path filters to IPv6, where previous Red Hat Enterprise Linux versions only applied anti-spoofing filter rules to IPv4 network traffic.

Every version of Red Hat Enterprise Linux is the result of countless hours of work from many individuals. Above we highlighted a few of the efforts that the Red Hat Product Security team assisted with in the release of Red Hat Enterprise Linux 7. We also worked with a number of other individuals to see these changes become reality. Our job doesn’t stop there, though. Once Red Hat Enterprise Linux 7 was released, we immediately began tracking new security issues and deciding how to fix them. We’ll further explain that process in an upcoming blog post about fixing security issues in Red Hat Enterprise Linux 7.

Reactive Product Security at Red Hat

The goal of Product Security at Red Hat is “to help protect customers from meaningful security concerns when using Red Hat products and services.” What does that really mean and how do we go about it? In this blog, we take a look at how Red Hat handles security vulnerabilities and what we do to reduce risk to our customers.

In 2001, we founded a dedicated security team within Red Hat to handle product security. Back then, we really had just one product line, the Red Hat® Linux® distribution. Now, 14 years later, we support well over 100 different products and versions from Red Hat Enterprise Linux, to OpenStack®, to Docker. In addition to handling the reaction to vulnerabilities found in our products, we also proactively work on improving security for the future. An upcoming blog post will highlight some of those activities.

All software, no matter what the license, provenance, or supply-chain involved, have bugs–mistakes in the code which introduce errors. Some of those errors may cause a program to behave differently than what is expected, others may cause a program to crash. Of these errors, a small proportion are classified as vulnerabilities if they pose a security risk where an attacker can deliberately cause a program to fail.

Our products are generally made up of many different open source components; for example, Red Hat Enterprise Linux 7 is composed of several thousand different packages and each one can be a separate open source project. Red Hat Product Security is accountable for knowing every component used in every product so we can keep track of the security issues. This has become an area of expertise for us and is recognized by the industry as handling vulnerabilities in third party software is not a trivial task.

It all starts with a team which monitors a number of sources to find out about security issues in such third party components. In a previous blog post, we gave some metrics for a years worth of vulnerabilities and showed that in nearly half of the vulnerabilities we fixed, we were aware in advance of the issue being made public. The biggest source of information regarding non-public issues was through the two-way relationships we have with upstream open source projects and our peer vendors. Additionally, 17% of all the issues we fixed were found internally by Red Hat through security audits by our Quality Engineering team or by the product engineering teams themselves.

The next step is to assess whether these issues affect any of our products and determine the severity of each one. We do this based on a technical assessment from our team of skilled researchers. In addition to the nature of the vulnerability itself and the types of exploits likely to operate against it, other considerations include which specific pieces of code are impacted, the sensitivity of the applications they support, and their potential degree of exposure. For any given a vulnerability in an Open Source component, different products across different vendors could be affected in different ways depending on the versions being used, what patches are included, and even how the package is compiled.

In order to manage this workload, the Product Security team makes use of a number of tools and workflow processes all built around the principles of GTD.

Depending on the severity of the issue and the life-cycle for the product, patches get created and updates prepared. For many of our products, our policy is to back-port fixes an approach that significantly reduces the potential for compatibility issues and the introduction of additional vulnerabilities while making it easier for customers to consume updates. These updates, together with our advisory text explaining the issue, make their way to customers as security errata.

We actively monitor the time it takes for vulnerabilities to pass through this entire process. For example, Red Hat Enterprise Linux 5, since its release in 2007, has had 98% of all critical flaw fixes available to customers either the same day or next calendar day, once the issue was known to the public. We make all of our data on this available so customers can determine metrics for their particular environment.

In practice, what this means is that a Red Hat subscription provides customers with guidance, stability, and security that can only come from Red Hat. For a given product, there is a single mechanism to get updates for security issues across all components and technologies included, no matter which open source project they came from. Products are supported with long life cycles, and we maintain security updates for open source components included even beyond their upstream end of life.

We’ve briefly shown that we have well-established processes to effectively manage vulnerabilities in open source software, and that we are effective in getting fixes for these issues to customers, but there’s more that we do on the reactive side of handling security events.

2014 will be remembered for a number of high profile vulnerabilities, including several in widely used open source components: Heartbleed, ShellShock, and Poodle. Where these affected Red Hat products, we provided fast updates to correct them. However, getting fast fixes out was only part of the value.

In September last year, serious issues were found in the UNIX-like shell, Bash, called ShellShock. During this incident, Red Hat customers also received:

  • Timely advice. By the time the issue went public, we had specific knowledge base articles on the Customer Portal explaining how products were affected, how to get and install the fixes, and how to determine if you were vulnerable to the issue. Our article, linked above, was the definitive source of information about the vulnerability–being cited by most news articles, Wikipedia, and even US-CERT. The knowledge base and blog were continually updated with the latest knowledge and best practices.
  • Industry-leading security expertise. After the original flaw in Bash was identified and fixed, a second issue was discovered in public. It was a Red Hat Product Security engineer who designed and wrote the comprehensive patch used by most vendors in fully addressing this issue.
  • Immediate support. The Red Hat Customer Portal had an alert on every page, with notifications, and our support staff had access to the technical information. We were ready to provide immediate support to customers.
  • Proactive notifications. For customers with products affected by the issue, we sent email notifications within the first few hours. This email provided a call to action and linked to our specific knowledge and fixes for this issue. Posts on our Red Hat Support social media channels also directed customers to our knowledge base articles and fixes.
  • A self-detection tool: We also released a self-detection tool via Red Hat Access Labs to allow customers to easily identify whether their environment was vulnerable.

We’d like customers to hear about these major security issues from us first and then be able to install the fix for the issue. When a significant security event occurs, customers can can come to Red Hat first, safe in the knowledge that we’ll be on top of the situation and be able to give specific, timely, calm, and technically-accurate advice on how the issues affect all of our products and services.

Update on Red Hat Enterprise Linux 6 and FIPS 140 validations

Red Hat achieved its latest successful FIPS 140 validation back in April 2013. Since then, a lot has happened. There have been well publicized attacks on cryptographic protocols, weaknesses in implementations, and changing government requirements. With all of these issues in play, we want to explain what we are doing about it.

One of the big changes was that we enabled support of Elliptic Curve Cryptography (ECC) and Elliptic Curve Diffie Hellman (ECDH) in Red Hat Enterprise Linux to meet the National Institute of Standards and Technology’s (NIST’s) “Suite B” requirements taking effect this year. Because we added new ciphers, we knew we needed to re-certify. Re-certification brings many advantages to our government customers, who not only benefit from the re-certification, but they also maintain coverage from our last FIPS 140 validation effort. One advantage of re-certification is that we have picked up fixes for BEAST, Lucky 13, Heartbleed, Poodle, and some lesser known vulnerabilities around certificate validation. It should be noted that these attacks are against higher level protocols that are not part of any crypto primitives covered by a FIPS validation. But, knowing the fixes are in the packages under evaluation should give customers additional peace of mind.

The Red Hat Enterprise Linux 6 re-certification is now under way. It includes reworked packages to meet all the updated requirements that NIST has put forth taking effect Jan. 1, 2014, such as a new Deterministic Random Bit Generator (DRGB) as specified in SP 800-90A (PDF); an updated RSA key generation technique as specified in FIPS 186-4 (PDF); and updated key sizes and algorithms as specified in SP 800-131A (PDF).

Progress on the certification is moving along – we’ve completed review and preliminary testing and are now applying for Cryptographic Algorithm Validation System (CAVS) certificates. After that, we’ll submit validation paperwork to NIST. All modules being re-certified are currently listed on NIST’s Modules in Process page, except Volume Encryption (dm-crypt). Its re-certification is taking a different route because the change is so minor thus not needing CAVS testing. We are expecting the certifications to be completed early this year.

Before you initiate a “docker pull”

In addition to the general challenges that are inherent to isolating containers, Docker brings with it an entirely new attack surface in the form of its automated fetching and installation mechanism, “docker pull”. It may be counter-intuitive, but “docker pull” both fetches and unpacks a container image in one step. There is no verification step and, surprisingly, malformed packages can compromise a system even if the container itself is never run. Many of the CVE’s issues against Docker have been related to packaging that can lead to install-time compromise and/or issues with the Docker registry.

One, now resolved, way such malicious issues could compromise a system was by a simple path traversal during the unpack step. By simply using a tarball’s capacity to unpack to paths such as “../../../” malicious images were able to override any part of a host file system they desired.

Thus, one of the most important ways you can protect yourself when using Docker images is to make sure you only use content from a source you trust and to separate the download and unpack/install steps. The easiest way to do this is simply to not use “docker pull” command. Instead, download your Docker images over a secure channel from a trusted source and then use the “docker load” command. Most image providers also serve images directly over a secure, or at least verifiable, connection. For example, Red Hat provides a SSL-accessible “Container Images”.  Fedora also provides Docker images with each release as well.

While Fedora does not provide SSL with all mirrors, it does provide a signed checksum of the Docker image that can be used to verify it before you use “docker load”.

Since “docker pull” automatically unpacks images and this unpacking process itself is often compromised, it is possible that typos can lead to system compromises (e.g. a malicious “rel” image downloaded and unpacked when you intended “rhel”). This typo problem can also occur in Dockerfiles. One way to protect yourself is to prevent accidental access to index.docker.io at the firewall-level or by adding the following /etc/hosts entry:

127.0.0.1 index.docker.io

This will cause such mistakes to timeout instead of potentially downloading unwanted images. You can still use “docker pull” for private repositories by explicitly providing the registry:

docker pull registry.somewhere.com/image

And you can use a similar syntax in Dockerfiles:

from registry.somewhere.com/image

Providing a wider ecosystem of trusted images is exactly why Red Hat began its certification program for container applications. Docker is an amazing technology, but it is neither a security nor interoperability panacea. Images still need to come from sources that certify their security, level-of-support, and compatibility.