In compliance with directives from the FedRAMP JAB, cloud.gov has completed our ED-22-01 response per the CISA-provided template. The report will be available soon to our authorized customers on the FedRAMP Secure Repository in Max.gov, or by request to support@cloud.gov.

We're continuing to monitor the situation and developments with the Log4j vulnerability and have made several other platform changes over the last few days as updates have become available for cloud.gov's platform components.

For more information, please see our most recent news post here: https://cloud.gov/2021/12/22/log4j_vulnerability_bod_22-02_update/

We have posted a news update:

https://cloud.gov/2021/12/22/log4j_vulnerability_bod_22-02_update/

regarding log4j and BOD 22-02. The post summarizes customer responsibilities, steps we have taken to protect our customers, and the status of the platform with respect to log4j.

We’ve been continuing to perform updates, patch internal components, and implement remediations to ensure we are staying ahead of the Log4j2 vulnerability that was first reported last week. As of today, December 17, 2021, we have taken the following additional steps since our last update:

We deployed several additional updates to platform components that had received secondary fixes with the release of log4j 2.16.0.

We applied a mitigation fix to our logging platform that removes the JndiLookup class entirely to mitigate the vulnerability. We’ve already been tracking a separate work item to upgrade our entire logging platform stack in the coming weeks, which will include a full patch and fix of those components.

We have also updated the Java and PHP buildpacks within the platform to their latest releases, both of which address the log4j2 vulnerability. These have been applied ahead of the next general Cloud Foundry platform update to make sure customers have access to the fixes as quickly as possible for their applications. Customers will need to restage their Java and/or PHP applications to make use of the new buildpacks and will be receiving an automated notice as a reminder to do this. Please note that the PHP buildpack is not vulnerable in the cloud.gov platform, but we have updated it regardless as an additional precaution. Our blog post entitled “log4j customer responsibility”[1] has more information.

In addition to these recent updates and changes, we are also continuing to adjust our Web Application Firewall (WAF) rules now that AWS has made additional rules available in the service to specifically help address the log4j vulnerability. We plan to roll those updates out over the course of the next few business days as we continue to assess their potential impact on the platform and customer applications. As a reminder, we already have our initial WAF rules in place to provide additional protection from this vulnerability.

Lastly, we are still actively coordinating with AWS for any other actions that need to be taken as AWS continues to update their infrastructure and services.

We will post more updates as we continue to monitor the state of the log4j2 vulnerability and apply any other necessary fixes or mitigations to the cloud.gov platform. If anyone has any questions or concerns, please reach out to us at support@cloud.gov.

[1] https://cloud.gov/2021/12/14/log4j-buildpack-updates/#applicability-to-the-php-buildpack

At 11:32 AM ET today, December 13, 2021, we put several additional mitigations in place on the cloud.gov platform for the log4j2 vulnerability that was disclosed last Friday.

The Cloud Foundry community has released updated versions of several platform components, including Credhub and UAA[1], that include the latest version of log4j2 that addresses this critical vulnerability. We have spent the morning updating these components and rolling them out through all of our environments. They will be fully deployed in production later today.

We also took an additional step of setting a system-wide environment variable to set the correct log4j2 log message configuration property to mitigate this issue as well. Operators running a Cloud Foundry platform are able to set system-wide variables across the whole platform[2], which then become available and a part of all customer workloads running within the platform. In this case, we set the following system-wide environment variable: {"LOG4J_FORMAT_MSG_NO_LOOKUPS":"true"}

This environment variable will appear when you run cf env . Customers may already see this set in their applications as the platform updates we applied roll through today, but others will need to restage their application(s) in order for this environment variable to take effect. We will be sending a separate notice out to customers directly about this as well.

Should the specific need arise to change this setting, a customer can override this environment variable by using the cf set-env command or setting this variable in their application manifest. If there are any questions about this, please contact us at support@cloud.gov.

Please note that CISA is also tracking this log4j2 vulnerability and recommending that this be remediated by December 24, 2021[3], which means all customers should update any of their own log4j2 dependencies within their applications by this date.

Lastly, we’re tracking updates that the Cloud Foundry community is making to other platform components, including the PHP and Java buildpacks, which are also affected. Once these updates are released and we’re able to roll them out to the platform, we’ll post another update and communicate directly with customers running Java and/or PHP applications on the next steps as well.

We are continuing to monitor the platform for any adverse activity and communicate with our information security and incident response teams in case there are any new developments on this vulnerability.

For more information about the log4j2 vulnerability in general, please see the CISA notice here: https://www.cisa.gov/uscert/ncas/current-activity/2021/12/10/apache-releases-log4j-version-2150-address-critical-rce

Amazon Web Services has also released a security bulletin for what steps they are taking with their services: https://aws.amazon.com/security/security-bulletins/AWS-2021-006/

If anyone has any questions or concerns about this information or anything else related to the log4j2 vulnerability, please do not hesitate to reach out to us at support@cloud.gov.

[1] https://www.cloudfoundry.org/blog/log4j-vulnerability-cve-2021-44228-impact-on-cloud-foundry-products/
[2] https://docs.cloudfoundry.org/devguide/deploy-apps/environment-variable.html#evgroups
[3] https://www.cisa.gov/uscert/ncas/current-activity/2021/12/10/cisa-adds-thirteen-known-exploited-vulnerabilities-catalog

We've put multiple mitigations in place at this time. Please note that some of these mitigations may affect customers' ability to search logs.fr.cloud.gov for strings that might indicate exploit attempts. As always, contact cloud.gov support if you require assistance.
We'll provide further updates Monday.

At 10:47 EST today, December 10th, 2021, cloud.gov started our incident response process for log4shell, CVE-2021-44228[1], as it posed an “imminent threat” to our platform and to our customers. We’ve concluded our initial investigation and have taken immediate steps to mitigate the risk and impact of this vulnerability, and are now planning our next steps to continue working toward complete remediation.

Current status

The entire platform, including customer applications, have web application firewall rules that block traffic targeting this vulnerability. This is a partial mitigation, a temporary measure, and could impact customer applications. If you run Java applications you should follow Apache log4j guidance, per CISA, to either update log4j2 to 2.15.0 or mitigate this vulnerability by setting "log4j2.formatMsgNoLookups" to “true” (if your log4j2 version is greater than 2.10). We are also continuing to assess the impact of this vulnerability.

Assessment of impact

Our initial investigation into the risk of exposure to the log4j vulnerability showed that some of our components are using an affected version of log4j (between versions 2.0-beta9 and 2.14.1). Most of these components are already not accessible publicly, but there are a few that are.

Based on this assessment, we determined that there were two immediate things we could do to mitigate the risk of this vulnerability while we work toward patching the log4j versions in the affected components:

1. Implementing web application firewall rules in our staging and production environments to block any incoming requests attempting the exploit.

2. Adjusting the log4j properties in the affected components to make sure the log4j2.formatMsgNoLookups property is set to true.
Current state of remediation

We have enabled the web application firewall (WAF) rules in both our staging and production environments to cover the platform across the board, which includes all customer applications. All requests that come in are now filtered through the WAF and checked for the log4j vulnerability; any that are found to match are immediately blocked and do not make it to their destination. The WAF maintains a sample of blocked requests and we have also enabled logging so that we can review and monitor any activity related to external entities attempting to exploit this vulnerability.

We are actively making adjustments to the log4j configuration properties in our affected components so that our JVMs know to start with the log4j2.formatMsgNoLookups property set to true. This is the recommended way[2] to remediate this vulnerability in instances where you cannot immediately update log4j to version 2.15.0, as recommended by CISA[3].

Customer impact

As mentioned above, the new web application firewall rules we implemented work across the entire platform, which includes all customer applications. Ideally this should provide immediate protection to our customers without any disruption, but if any customer notices degraded or impacted service of their application we encourage them to report this to us at support@cloud.gov and let us know the details so that we can assist them.

We also encourage all of our customers to take a look at all of their Java-based applications to check for any log4j dependencies and take the necessary recommended actions[2] to mitigate this vulnerability, ideally by updating log4j to version 2.15.0 or by at least specifying the log4j2.formatMsgNoLookups configuration property and setting it to true, then redeploying their application(s).

We will continue to monitor platform stability as we deploy our own updates to address this vulnerability in the platform and encourage our customers to reach out to us at support@cloud.gov if they believe their applications are impacted by any of our fixes.

Next steps

We implemented our own web application firewall rules to mitigate this issue as quickly as possible to protect the cloud.gov platform and our customers, but we are waiting to see when AWS adds their own rule sets to the GovCloud region and plan to either switch to those or use them in addition to our own to make sure all requests are properly checked and covered for this log4j vulnerability. AWS has already provided these new rules in the commercial regions.

Furthermore, once we finish updating our platform components that rely on log4j with the new configuration property setting, we will start researching what it will take to update log4j (and the JDK itself if necessary) to the latest version(s) and make plans to do so as soon as possible.

We will continue to provide updates to this incident as we uncover more details or make other changes to the platform to address this vulnerability. Once we are confident that we have fully patched everything and mitigated the risk, we will provide a final write-up and close this incident out.

Please reach out to us if you have any concerns or questions about this incident at support@cloud.gov and we will be happy to help.

[1] https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-44228
[2] https://logging.apache.org/log4j/2.x/security.html
[3] https://www.cisa.gov/uscert/ncas/current-activity/2021/12/10/apache-releases-log4j-version-2150-address-critical-rce