What should we learn from Shellshock?

The recent Bash vulnerability, known as Shellshock, provides an excellent opportunity to discuss security. Shellshock is somewhat unique as it was not a new type of bug nor did it require complex steps to remediate, despite its widespread impact. Members of the security and IT community can use this time to be reminded of a few valuable lessons about security and systems best practices in their response.

Background

On September 24^th, a disclosure was made that the popular Bash shell had an arbitrary command execution bug (https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2014-6271). This means that Bash would parse specially crafted commands in a way which was not anticipated. The bug left many workstations, servers and devices open to attackers.

The vulnerability stems from a Bash deficiency that improperly checks for the end of a command inside a shell function. Bash would continue to execute beyond a function definition when the offending strings were passed to a new shell.

$ export a='() { echo “Hello world”; }; echo “Arbitrary command”‘

$ bash

Arbitrary command

$

Alone this is only a minor bug, but the Bash developers made an unfortunate design decision to not differentiate between functions and environment variables. The format of the text within the variable was used to differentiate whether it was a function or not.

This means that each of the following commands is an acceptable way to create a function:

$ a () { echo “Hello world”; }

$ a='() { echo “Hello world”; }’

This design choice created an attack vector where attackers could execute arbitrary commands on any system to which they could pass environment variables to Bash.  Unfortunately, many Unix programs are designed to permit the passing of environment variables in this manner as standard behavior.

Attack Vectors

Immediately following the disclosure the most discussed vectors were the Secure Shell daemon (sshd) used for remote logins and the Hyper Text Transfer Protocol daemon (httpd) used for serving web content.

sshd

The sshd vector is fairly obvious, as most often remote logins to systems execute a shell. If that shell is Bash and an environment variable can be passed to the shell, then the remote host can be forced to execute arbitrary commands upon login. The sshd_config file supports the AcceptEnv configuration command that may be used to limit which environment variables can be passed, however few operating system implementations fully restrict passing these variables.

Here are a few examples:

CentOS 7:

AcceptEnv LANG LC_CTYPE LC_NUMERIC LC_TIME LC_COLLATE LC_MONETARY LC_MESSAGES

AcceptEnv LC_PAPER LC_NAME LC_ADDRESS LC_TELEPHONE LC_MEASUREMENT

AcceptEnv LC_IDENTIFICATION LC_ALL LANGUAGE

AcceptEnv XMODIFIERS

Ubuntu 12.04:

AcceptEnv LANG LC_*

FreeBSD 10.0:

Fully restricts environment variable passing (not vulnerable by default)

MacOS 10.9:

AcceptEnv LANG LC_*

This means if the test string is inserted in one of the AcceptEnv supported variables, that an arbitrary command may be executed.

Using CentOS 7 as an example:

$ export LC_NAME='() { echo “Hello world”; }; echo “Arbitrary command”‘

$ ssh localhost

xxx@localhost’s password:

Last login: xxx from xxx

Arbitrary command

$

It is important to note in the previous example using ssh, a valid login to the system is a requirement.  Because of this, an attacker must piggyback on a valid session from an existing user. This creates a high barrier of exploitation for this particular vector.

httpd

Unfortunately for web server administrators, there is a much lower barrier for exploitation through httpd. Plenty of people over the last twenty years have used shell scripts as a quick way to write CGI for their websites. If you are familiar with such scripts, you know that web servers pass environment variables to the script during execution. These variables contain information such as the specific URL requested, what browser version is being used, etc..

Of course the environment variable is the way to pass arbitrary commands in this bug, leaving Bash-based CGI scripts vulnerable to remote command execution. Since so many web servers contain these scripts and most are open to access from the internet, they provide an easily exploitable vector.

As web based tools, Google, and other search engines are even generous enough to provide a large initial index for attackers: https://www.google.com/search?q=filetype%3Ash+inurl%3Acgi-bin

Even operating systems such as FreeBSD, which do not include Bash as a default shell, are vulnerable when Bash is specified as the script parser. The following CGI example uses FreeBSD 10.0 with Apache 2.4:

$ cat test.sh

#!/usr/local/bin/bash

echo “Content-type: text/plain”

echo

echo “Hello web user”

echo

env

The env command outputs the current environment, which will display the variables passed to the script from Apache. The output below is a subset of the variables for succinctness:

$ fetch -o- -q http://localhost/cgi-bin/test.sh | head -n 5

Hello web user

SERVER_SIGNATURE=

HTTP_USER_AGENT=fetch libfetch/2.0

SERVER_PORT=80

This script can now be called with a user agent configured to cause remote command execution.

$ id -g

1001

$ fetch -o- -q –user-agent='() { echo ; }; echo “Content-type: text/plain”; echo “”; echo “Arbitrary command ran as `/usr/bin/id`”;’ http://localhost/cgi-bin/test.sh | head -n8

Arbitrary command ran as uid=80(www) gid=80(www) groups=80(www)

Content-type: text/plain

Hello web user

SERVER_SIGNATURE=

SERVER_PORT=80

HTTP_HOST=localhost

dhclient / qmail / others

After the initial thoughts around sshd and httpd, the security community, and attackers, began to scour Unix software for ways to remotely pass environment variables. It is very common for systems to use a shell and environment variables in Unix, and while vulnerable software has been identified, the list is no doubt incomplete. It is worth mentioning that the Shellshock bug was introduced more than twenty years ago. Finding every piece of software bundled with a Bash shell or still making use of Bash, over that time would be an implausible task. Therefore understanding further risk should be secondary to resolving the problem itself.

The Fix

As part of the public disclosure, patches were provided for Bash all the way back to Bash 3.0, which was released more than fourteen years ago. This was a great benefit to the community as it allows in-place patching of the majority of systems running Bash. Unfortunately, this initial patch only patched the original arbitrary execution bug. It did not fix the root of the problem, differentiating between variables and functions. Parsing of commands in a secure way is not a simple exercise, and naturally within a few hours another method to bypass the fix was found (http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2014-7169):

$ ls file

ls: cannot access file: No such file or directory

$ env var='() { (func)=>\’ bash -c “file uname -o”

bash: var: line 1: syntax error near unexpected token `=’

bash: var: line 1: `’

bash: error importing function definition for `var’

$ cat file

GNU/Linux

$

Over the days following the initial patch, an inquisitive security community found a number of other parser problems:

https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2014-6277

https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2014-6278

https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2014-7186

https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2014-7187

Finally a fix to the root of the problem was applied to the Bash source repository late on September 27^th .

The fix differentiates a variable definition from a function definition, and the first example provided earlier no longer works:

$ abc='() { echo “Hello world”; }’

$ abc

bash: abc: command not found

$

$ abc () { echo “Hello world”; }

$ abc

Hello world

$

With the new functionality when functions are passed to Bash, they now receive a specific format that can no longer conflict with variables that are passed to sub-shells:

$ export -f abc

$ env | grep -A1 ^BASH

BASH_FUNC_abc%%=() {  echo “Hello world”

}

$

This means the examples above which used LC_NAME and HTTP_USER_AGENT would no longer function as the names are not named in the format BASH_FUNC_<function>%%.

The Data

The release of any new remote vulnerability is immediately met with public scans for those hosts that are vulnerable.  The scans originate from both attackers and researchers seeking knowledge about exposures.  Level 3’s broad view of the internet gives us an opportunity to see what is occurring during these periods of wide spread scanning.

With httpd hosted bash CGI scripts being the most widely deployed risk, looking at trends in http/https (ports 80 and 443) traffic during the time period the bug was released can provide visibility to how widespread scanning was.  This graph shows internet http and https traffic starting the day prior to the disclosure, and ending the day after the function differentiation patch was applied.

As you can see by the lack of variations in trend this shows how even large-scale security events pale in the face of overall internet traffic on web servers.

The same data displayed over the weeks preceding and following the public release confirms the trend during that time period is in line with internet behavior overall.

In order to find evidence of an increase in web traffic looking for the shellshock vulnerability this same data can be filtered through our reputation data. Black Lotus Labs maintains situational risk data around potential threats in the internet. This data is used to protect our customers, our infrastructure, and perform threat analysis such as this. When we take the same web traffic, but filter it through a lens of those hosts that pose a risk we can see a distinct set of spikes in traffic during the days following the release of the bug, confirming that there was interest in this bug amongst attackers.

Response

Bugs like Shellshock help to reinforce the reality that the community needs to be maintaining best practices. The bug itself, along with the resolution to it, was actually relatively simple in nature.

The security community

Best practices demand that command parsing of user input should be avoided where possible and sanitized if unavoidable. As can be seen by the many ways that Bash’s parser was attacked, handling of this input in the right way is not always a simple exercise. While placing blame on a twenty-year-old design decision is unfair, the Bash maintainers still refuse to resolve the problem in its entirety. Removing the conflict with namespace between passed variables and functions was a step in the right direction, but not allowing functions to pass at all by default would have been a far more secure decision. Any further parser problems found can still be passed through variables names in the format BASH_FUNC_<name>%% and continue to be exploited.

This brings us to another best practice of restricting environment variable passing. This is also a well-known best practice, and if the environmental variable  BASH_FUNC_* is not able to be passed, then the above problem will not exist at all. Continued restrictions on what may be passed to Bash and how these passed values are defined will only be a positive change to combat against the next potential security vulnerability.

Our customers and the IT community

However wide spread vulnerabilities such as Shellshock may be, they can help to reinforce patching best practices for everyone from home Linux users through the largest IT teams in the world. Having a comprehensive inventory of systems and software along with an automated patching capability would have given Shellshock a limited window of risk for most systems. This bug came with a need to only patch a single binary, and patches covering over a decade of backdated shell versions. Not many solutions are so accommodating.

The last best common practice item to highlight here is maintaining awareness of security issues and steps to remediate them. Patches were released along with the disclosure, and those security and systems administrator teams who took the time to maintain awareness of current risks to their environment were able to patch within hours. Taking the time to fully understand the bug has the additional benefit of allowing these teams block further parsing problems through their IPS, load balancers, WAF, or even Apache mod_security configuration to monitor traffic for and potentially block the offending strings.

So what should we learn?

Implementation of best common practices, being aware of the situation at hand, and having security/systems knowledge can quickly put Shellshock in the rearview mirror. Evaluating these controls, their execution, and overall response to bugs like Shellshock can help ensure minimized risk in the future. For developers and security professionals, ensuring that known risks are mitigated in future development as well as audited against past designs will provide further security against the risks that exist on our systems.