As of late Monday, no further effect on platform stability or availability persists from these attacks.
Posted Mar 04, 2020 - 18:14 EST
Update
We are continuing to actively defend the platform. We have further blocked more malicious traffic and are continuing to refine our firewall rules to ensure allowed traffic is not blocked. We are still seeing malicious traffic attempts and will continue to provide updates while we believe there is still an ongoing threat.
While we are actively defending the platform, it is possible that we may be accidentally blocking valid traffic. Please contact us if you believe we are blocking valid traffic.
Posted Mar 03, 2020 - 13:06 EST
Update
We are continuing to actively defend the platform. We have further blocked more malicious traffic and are continuing to refine our firewall rules to ensure allowed traffic is not blocked. While we have recovered degraded component performance, we are still seeing malicious traffic attempts and will continue to provide updates while we believe there is still an ongoing threat.
While we are actively defending the platform, it is possible that we may be accidentally blocking valid traffic. Please contact us if you believe we are blocking valid traffic.
Posted Mar 03, 2020 - 11:35 EST
Update
We have rolled out geographic IP restrictions to all CDN instances and will continue to monitor the situation.
While we are actively defending the platform, it is possible that we may be accidentally blocking valid traffic. Please contact us if you believe we are blocking valid traffic.
Posted Mar 02, 2020 - 21:03 EST
Update
We are currently rolling out geographic IP restrictions to all CDN instances.
While we are actively defending the platform, it is possible that we may be accidentally blocking valid traffic. Please contact us if you believe we are blocking valid traffic.
Posted Mar 02, 2020 - 19:49 EST
Update
We have implemented geographic IP restrictions on the core cloud.gov platform and are continuing to work on adding geographic IP restrictions on our existing CDN services to help alleviate malicious requests further.
While we are actively defending the platform, it is possible that we may be accidentally blocking valid traffic. Please contact us if you believe we are blocking valid traffic.
Posted Mar 02, 2020 - 18:36 EST
Update
We have continuously implemented more extensive firewall rules blocking foreign geographic IP ranges and are continuing to monitor the traffic load on the platform. We are continuing to actively work with internal security teams to ensure our firewall configurations are current and we are appropriately blocking proper ranges. Customers should start to see lower error rates in their applications, as well.
We are in the process of implementing geographic IP restrictions on our existing CDN services to help alleviate malicious requests further.
While we are actively defending the platform, it is possible that we may be accidentally blocking valid traffic. Please contact us if you believe we are blocking valid traffic.
Posted Mar 02, 2020 - 17:36 EST
Monitoring
The cloud.gov Operations team has implemented more extensive firewall rules blocking a large amount of foreign geographic IP ranges and are continuing to monitor the traffic load on the platform. We are actively working with internal security teams to ensure our firewall configurations are current and we are appropriately blocking proper ranges.
While we are actively defending the platform, it is possible that we may be accidentally blocking valid traffic. Please contact us if you believe we are blocking valid traffic.
Posted Mar 02, 2020 - 16:08 EST
Identified
The cloud.gov Operations team has determined the cloud.gov platform is being attacked by a set of both known and unknown entities. We've determined over the last 24 hours there have been over 20 million attempted attacks on the cloud.gov control plane. This attack currently seems to be a generalized attack and does not seem to be targeting specific endpoints, applications, or areas of the platform.
We are currently seeing minimal performance degradation. The performance impacts are currently limited to our authentication and logging subsystems. We are actively monitoring for additional impacts.
We have implemented a Web Application Firewall with restrictive rules, and are continuing to tighten the rules as we see fit. We are being aggressive in defending the platform on behalf of customers and are in the process of implementing network-based L4 blocking in addition to our L7 blocking. As the traffic has been coming mostly from foreign geographical entities, we are going to block known IP ranges from foreign geographical IP ranges.
While we are actively defending the platform, it is possible that we may be accidentally blocking valid traffic. Please contact us if you believe we are blocking valid traffic.
Posted Mar 02, 2020 - 15:03 EST
Update
We are continuing to monitor for any further issues.
Posted Mar 02, 2020 - 13:37 EST
Update
We are continuing to monitor incoming traffic to the platform for potentially malicious traffic.
Posted Mar 02, 2020 - 13:35 EST
Monitoring
We have performed a rolling reboot of the control plane components and are continuing to monitor error rates. We have also implemented a Web Application Firewall ruleset against potentially malicious traffic.
Posted Mar 02, 2020 - 12:45 EST
Investigating
Customers have reported an increased error rate when interacting with our authentication and logging subsystems. We are currently investigating the issue and will provide further updates as we have them.
Posted Mar 02, 2020 - 11:46 EST
This incident affected: cloud.gov customer access (Logs front end, Login) and cloud.gov customer applications (Logs intake and storage).