At 10:47 EST today, December 10th, 2021, cloud.gov started our incident response process for log4shell, CVE-2021-44228[1], as it posed an “imminent threat” to our platform and to our customers. We’ve concluded our initial investigation and have taken immediate steps to mitigate the risk and impact of this vulnerability, and are now planning our next steps to continue working toward complete remediation.
Current status
The entire platform, including customer applications, have web application firewall rules that block traffic targeting this vulnerability. This is a partial mitigation, a temporary measure, and could impact customer applications. If you run Java applications you should follow Apache log4j guidance, per CISA, to either update log4j2 to 2.15.0 or mitigate this vulnerability by setting "log4j2.formatMsgNoLookups" to “true” (if your log4j2 version is greater than 2.10). We are also continuing to assess the impact of this vulnerability.
Assessment of impact
Our initial investigation into the risk of exposure to the log4j vulnerability showed that some of our components are using an affected version of log4j (between versions 2.0-beta9 and 2.14.1). Most of these components are already not accessible publicly, but there are a few that are.
Based on this assessment, we determined that there were two immediate things we could do to mitigate the risk of this vulnerability while we work toward patching the log4j versions in the affected components:
1. Implementing web application firewall rules in our staging and production environments to block any incoming requests attempting the exploit.
2. Adjusting the log4j properties in the affected components to make sure the log4j2.formatMsgNoLookups property is set to true.
Current state of remediation
We have enabled the web application firewall (WAF) rules in both our staging and production environments to cover the platform across the board, which includes all customer applications. All requests that come in are now filtered through the WAF and checked for the log4j vulnerability; any that are found to match are immediately blocked and do not make it to their destination. The WAF maintains a sample of blocked requests and we have also enabled logging so that we can review and monitor any activity related to external entities attempting to exploit this vulnerability.
We are actively making adjustments to the log4j configuration properties in our affected components so that our JVMs know to start with the log4j2.formatMsgNoLookups property set to true. This is the recommended way[2] to remediate this vulnerability in instances where you cannot immediately update log4j to version 2.15.0, as recommended by CISA[3].
Customer impact
As mentioned above, the new web application firewall rules we implemented work across the entire platform, which includes all customer applications. Ideally this should provide immediate protection to our customers without any disruption, but if any customer notices degraded or impacted service of their application we encourage them to report this to us at
support@cloud.gov and let us know the details so that we can assist them.
We also encourage all of our customers to take a look at all of their Java-based applications to check for any log4j dependencies and take the necessary recommended actions[2] to mitigate this vulnerability, ideally by updating log4j to version 2.15.0 or by at least specifying the log4j2.formatMsgNoLookups configuration property and setting it to true, then redeploying their application(s).
We will continue to monitor platform stability as we deploy our own updates to address this vulnerability in the platform and encourage our customers to reach out to us at
support@cloud.gov if they believe their applications are impacted by any of our fixes.
Next steps
We implemented our own web application firewall rules to mitigate this issue as quickly as possible to protect the cloud.gov platform and our customers, but we are waiting to see when AWS adds their own rule sets to the GovCloud region and plan to either switch to those or use them in addition to our own to make sure all requests are properly checked and covered for this log4j vulnerability. AWS has already provided these new rules in the commercial regions.
Furthermore, once we finish updating our platform components that rely on log4j with the new configuration property setting, we will start researching what it will take to update log4j (and the JDK itself if necessary) to the latest version(s) and make plans to do so as soon as possible.
We will continue to provide updates to this incident as we uncover more details or make other changes to the platform to address this vulnerability. Once we are confident that we have fully patched everything and mitigated the risk, we will provide a final write-up and close this incident out.
Please reach out to us if you have any concerns or questions about this incident at
support@cloud.gov and we will be happy to help.
[1]
https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-44228[2]
https://logging.apache.org/log4j/2.x/security.html[3]
https://www.cisa.gov/uscert/ncas/current-activity/2021/12/10/apache-releases-log4j-version-2150-address-critical-rce