How To Detect and Mitigate the Log4Shell Vulnerability (CVE-2021-44228 & CVE-2021-45046)
A few days ago, a serious new vulnerability was identified in Apache log4j v2 and published as CVE-2021-44228. We were one of the first security companies to write about it, and we named it "Log4Shell".
This guide will help you:
- Find trusted sources for Log4Shell information
- Determine if you are impacted by Log4Shell
- Mitigate the Issue
If you're just trying to understand the Log4Shell vulnerability and the impact of it, please refer to our earlier blog post.
Be careful what Log4Shell advice you trust online
Because of the severe impact from this vulnerability, there has been a lot of discussion on the internet about it. Some of this information is outdated or wrong and will leave you vulnerable if you follow it!
In contrast, this guide has been written by a team of professional Security Engineers at LunaSec. Everything here has been peer-reviewed by multiple security experts, and where possible our sources will be linked for other Security professionals to verify against. This post links to many other guides and how-tos that we believe are trustworthy.
The full list of common bad advice is at the bottom of this post. If you believe you've already mitigated Log4Shell, or you believe you're not vulnerable, please double-check your current information is up-to-date.
Determine if you are impacted by Log4Shell
This vulnerability affects anybody who's using the log4j packages. That means it's primarily Java, but other languages like Scala, Groovy, or Clojure are also impacted.
Automatically Scanning Your Package
We've built a command line utility that can check .jar and .war files in your project directory and report if any are vulnerable. It works by scanning for hashes of known vulnerable log4j classes. If you have a vulnerable version of a log4j in your built Java project, the hash will match a one of the hashes in the list.
Make sure you download the right version for your Operating System and CPU architecture. Once downloaded, you can extract it and run the log4shell command in your terminal. The tool can scan individual files or whole directories.
OSX or Linux
Windows
Example Output
Here is the help text:
Scanning a Java JAR file (vulnerable log4j detected)
Installing the Package on *nix systems (optional)
NOTE
Please make sure that you're running this command on your fully built .jar or .war file. If you are using vendor software that you think might be vulnerable, but you can't get the .jar or .war files to scan yourself (or it is obfuscated), then you'll need to check out the section on vendor software advisories instead.
The source code for this is available on our GitHub here.
Manually Scanning Your Dependencies
The above CLI tool automatically looks for hashes of vulnerable classes. Read this if you'd like to build your own tool or scan manually.
Scanning for vulnerable .class files
Our automated tool above implements this functionality, but if you need to do this yourself then our Go source code has a list of hashes that you can use to scan with. (Thank you, hillu!)
Scanning for the log4j JAR file
You may want to simply scan the filesystem for vulnerable copies of the log4j .jar file. We wrote a small shell script to accomplish that before writing the above CLI.
This is a less accurate method of detection versus our automated scanner tool above because rather an inspecting inside your code, it requires the log4j .jar file to be present on your filesystem (not inside your built package). It does not recursively unpack .jar files. This works best if your dependencies are committed into your Repo, or if you're using a tool like Maven that downloads the .jar files for you.
If you're using Maven: The default directory that .jar files are downloaded to is ~/.m2. You may want to clear your cache, and then rebuild your project in order to limit false positives.
Setup
Run Scan
Checking Package Version
If you can check what versions of log4j2 are being used, you can check for any below the recently published 2.16.0.
LIMITED VULNERABILITY IN 2.15.0
As of Tuesday, Dec 14, version 2.15.0 was found to still have a possible vulnerability in some apps. We recommend updating to 2.16.0 which removes the message lookup feature entirely.
log4j v2
Almost all versions of log4j version 2 are affected.
2.0-beta9 <= Apache log4j <= 2.14.1
In other words, if you're using any version of log4j that is older than 2.15.0, you are most likely vulnerable, and under very specific situations, still possibly vulnerable on 2.15.0.
log4j v1
Version 1 of log4j is vulnerable to other RCE attacks (like CVE-2019-17571), and if you're using it you need to migrate to 2.16.0.
Checking Vendor Software Versions
The above scanning tool might not work for vendor's packages because of obfuscation, and in any case, you'll likely need to contact the vendor for mitigation.
Luckily, many vendors have created their own documents to explain the impact of Log4Shell on their products, and an extensive list of those advisories is being compiled here.
If a vendor has not created an advisory for this, there currently does not exist a succinct list of which Vendor software has been affected. There is an effort by Kevin Beaumont to create a spreadsheet that attempts to capture this being worked on, but at this time of this post that effort is still a work in progress.
Scanning Remote Endpoints
Please see our instructions to identify vulnerable remote servers in our original Log4Shell post.
How to Mitigate the Issue
Now that you know where you're vulnerable, the following sections will help you to figure out how to patch it.
This diagram created by the Swiss Government is an excellent visualization of the Log4Shell exploit. Take note of the possible solutions (shown in red) as we go over mitigation strategies.
Option 1: Upgrading to 2.16.0
Apache log4j has released a version that fixes the Log4Shell vulnerability as of version 2.16.0. This version disables JNDI by default and removes the message lookup feature.
We recommend you upgrade, if possible. For most people, this is the final and correct solution to the issue.
VERSION 2.15.0 STILL MAY BE VULNERABLE
Log4j version 2.15.0 which was previously thought to be secure has been found to still have a limited vulnerability, that could result in a DOS (but not RCE), users must update to 2.16.0.
Option 2: Enable formatMsgNoLookups
THIS FLAG DOES NOT PREVENT ALL VULNERABILITIES
As of Dec 14, it's been found that this flag is ineffective at stopping RCE in some situations, explained here by log4j and in CVE-2021-45046. We found the CVE wording confusing and are still investigating this vulnerability.
You must update to 2.16.0 or use the JNDI patches below.
The above release of log4j hardcodes the formatMsgNoLookups flag to true, preventing the attack. If you are using log4j version 2.10.0 to version 2.14.0 and can't yet update, you can still set the flag manually.
Set formatMsgNoLookups=true when you configure log4j by performing one of the following:
Pass as a JVM Flag
You can pass this as an argument when you invoke java.
java -Dlog4j2.formatMsgNoLookups=true ...
Set Environment Variable
Alternatively, this feature may be set via Environment Variable.
LOG4J_FORMAT_MSG_NO_LOOKUPS=true java ...
Or you can set this using the JVM arguments environment variable.
JAVA_OPTS=-Dlog4j2.formatMsgNoLookups=true
Option 3: JNDI patch
It's possible to modify the JNDI in place to stop the attack at the language level. It can even be done while the server is running. Please note this is a last resort, and should only be done if the above options aren't possible.
The easy-to-use tool Log4jHotPatch will apply the JNDI patch automatically.
For those using Kubernetes that can't perform any of the above mitigations, a new feature called "Ephemeral Containers" allows applying the hot patch to a running container. This could be useful for containerized vendor software. This guide explains how to apply the patch.
Option 4: Remote hot patch / LogOut4Shell
Because of the extensive control Log4Shell gives an attacker, it's actually possible to use the bug against itself to patch a running server. This isn't the recommended strategy for various reasons, but it could be a last resort for systems that you can't easily restart or modify. Note that doing this on a system you don't have permission to is most likely illegal. The fix will only work until the server (or the JVM) is restarted.
How to accomplish this is explained in this guide.
We are currently adding this functionality to our CLI. Subscribe below to be alerted when it's live.
How to protect yourself from future 0-days
It's becoming increasingly apparent that Log4Shell is not going to be the last vulnerability of its kind. Any trusted dependency can have security flaws or be malicious, and there is always the risk of accidentally introducing vulnerabilities into your own system.
The only way to implement software that is truly resilient to future security vulnerabilities like Log4Shell is to implement a "Secure by Default" architecture. This is what companies and the federal government are migrating to now because it's the only strategy that protects you long-term.
What is "Secure by Default"?
We've written about this before in our post on Why Data Breaches Happen, but the short version is: Accept that you're going to be hacked, and that the outer walls of your system will eventually be breached.
Build security into the parts of your system that specifically need it. Secure by Default software is designed to fail predictably under attack, so that the most sensitive data remains secure.
We've made implementing that as easy as possible with our Open Source security framework LunaSec. It works inside web apps, embedding an additional layer of isolation around the most sensitive data. Please leave us a star on GitHub.
Known Bad Advice
The following are all pieces of advice we've seen thrown around online that are misguided and dangerous. If you see advice online that contains any of the following, we please ask you to share this post with the authors to help limit the fallout from Log4Shell.
Updating Java is insufficient
There are many reports online that only certain Java versions are affected and that you're safe if you're on a newer Java version. Even on newer versions, it's still possible for an attacker to instantiate local classes on the server to trigger an exploit. And, even if no exploits are found right away, it still enables a Denial-of-Service attack when you're using a vulnerable version of log4j.
We believe it's likely only a matter of time before all current Java versions are impacted when running a vulnerable version of log4j. Just upgrading your Java version is insufficient, and you should not rely on this as a long-term defense against exploitation.
A WAF will not save you from Log4Shell
The Log4Shell vulnerability can not be entirely mitigated by using a WAF (Web Application Firewall) because it does not require your usage of it to be publicly accessible. Internal Data Pipelines, like Hadoop and Spark, and Desktop apps like the NSA's Ghidra will still be vulnerable.
In addition, there is no simple way to "filter out" malicious requests with a simple WAF rule because Log4Shell payloads may be nested. (See this GitHub for examples)
If you are using a vulnerable version of log4j, the only secure way to mitigate Log4Shell is through one of the strategies detailed above.
Updating the log statement format with %m{nolookupzz} is not advisable
This only applies to log4j versions >= 2.7.0, older versions don't support it
WARNING
In addition to the below issues with this strategy, this mitigation may be bypassed in certain scenarios and still allows for RCE (see our dedicated post about it), and this has been entirely removed in log4j 2.16.0 and newer.
Some people online are suggesting updating your logging statements from %m to %m{nolookupzz} to mitigate this**.
The string is intentionally wrong here to prevent blind copy-pasting.
We do not recommend you follow this strategy. Even if you manage to patch your application 100% today, you will still likely accidentally add a %m again in the future and then you will be vulnerable again.
In addition, it's possible to miss a line in your logging statements, or have a dependency that is using log4j with %m without you realizing. If either happens you will still be vulnerable.
We're strong advocates of a "Secure by Default" mentality with software, and we recommend you follow one of the other mitigations instead.
Resources
OWASP Application Security Verification Standard
This is a design framework that's being developed by the Open Web Application Security Project (OWASP) to help standardize the levels of security that applications achieve. It provides a solid baseline for setting up a security roadmap for your application.
LunaSec Secure by Default Framework
Spoiler: Maintained by us.
An Open Source development framework that enables you to easily add a "Secure by Default" architecture to your existing software with only a few lines of code. It's loosely based on the OWASP Application Security Verification Standard (above) and is also designed to be implemented in multiple levels.
Acra Database Protection
An Open Source database encryption library that helps you prevent data leaks by adding a layer of encryption to your data. It's designed and maintained by Cossack Lab's, which also offers it as a commercial offering.
FullHunt log4j Scanner
This is a CLI tool to help you identify vulnerable log4j endpoints in your infrastructure. (Note: It's a new tool, so it might have bugs. Please file an issue if you find any!)
We're also currently offering a free 30-minute consultation with one of our Security Engineers. If you're interested, please book some time with us here.
Stay Strong!
The last few days have been a painful experience for nearly every tech company out there. We hope that this guide helps your day be a little better.
If this post helped you, please share it with others to help them with Log4Shell too.
Social Media Links
Feel free to join the discussion on this post on any of the following websites:
Updates
INFO
We're continuously keeping this post up-to-date as new information comes out. If you have any questions, or you're confused about our advice, please file an Issue on GitHub.
If you would like to contribute, or notice any errors, this post is an Open Source Markdown file on GitHub.
- Fixed some weird grammar.
- Added social links.
- Reworked some content. Added more options for mitigation.
- Add warnings about limited vuln in 2.15 / noMsgFormatLookups
- Add additional disclaimer about %m.
- Added link to 2nd CVE info.
