Introduction

In recent weeks, the cloud.gov platform encountered a series of distributed denial-of-service (DDoS) attacks. These incidents temporarily impacted service availability. We are committed to maintaining the highest levels of service reliability and transparency, and this report provides a summary of the events and our responses.

Timeline of Events

Early April 2024: The platform experienced several brief disruptions due to significant increases in network traffic, which were identified as DDoS attacks. These incidents were effectively managed with minimal service disruption.

Analysis

The analysis revealed that the primary challenge was the saturation of our network infrastructure, leading to temporary service degradation. Our team was able to quickly identify and mitigate the attacks, minimizing their impact on service availability.

Response and Improvements

Following the incidents, we implemented several enhancements to strengthen our infrastructure and improve our response capabilities:

Infrastructure Scaling: We have increased the capacity of our network infrastructure to handle larger volumes of traffic.
Enhanced Monitoring: Improved monitoring tools are now in place to detect unusual traffic patterns more quickly.
Advanced Traffic Management: We have refined our traffic management policies to better identify and segregate malicious traffic.
Load Distribution: Adjustments have been made to optimize load distribution across the platform, reducing the potential impact of traffic spikes.

Ongoing Actions

Router Performance: Continuous improvements are being made to enhance the performance and resiliency of our routers.
Advanced Protection Implementation: Plans are in place to integrate additional advanced protective measures to further shield our services from malicious traffic.

Conclusion

With the implemented changes, we have observed improved platform stability and resilience against DDoS attacks. Our team remains vigilant and dedicated to further enhancing the security and reliability of cloud.gov. We appreciate the trust our customers place in us and are here to support any needs or questions at support@cloud.gov.

Posted May 01, 2024 - 13:49 EDT

Resolved

While we have detected additional DDoS attacks against the platform over the last week, there have been no additional platform outages, so we are resolving this incident.

As per our usual process, in the next few days the cloud.gov team plans to hold a retrospective on all of the DDoS incidents for the platform over the past two weeks. Once the retrospective is complete, we will publish our post-mortem analysis of the incidents, including lessons learned and planned improvements to the platform.

As always, thank you for being a cloud.gov customer!

Posted Apr 23, 2024 - 14:13 EDT

Update

From 3:51 PM ET to 3:56 PM ET, we detected another large-scale DDoS attack against the platform. Thanks to the currently deployed mitigations, the platform did not experience a full outage, but customers may have experienced elevated error rates from their applications.

Posted Apr 15, 2024 - 16:22 EDT

Monitoring

Incident Summary

Throughout the past week, our platform has been subject to ongoing Distributed Denial of Service (DDoS) attacks, as evidenced by the previous StatusPage updates.

Thanks to our security measures and platform automation, the platform was able to recover from those attacks in under five minutes, so we immediately marked all of those previous incidents as “Resolved”. To be clear, even though we considered those incidents resolved, our investigation into their causes and how to mitigate them more effectively remains ongoing.

Since these attacks are still ongoing and varying in scale, it is possible there could be further disruptions to our platform. To centralize and to improve our communications on these incidents, we will leave this particular incident open and will update it with announcements of any further outages or implemented mitigations.

Incident Details - DDoS attack

Time Detected: 4/15/2024 8:45 AM ET
Duration: Around 2 minutes
Impact: Users may have experienced slow response times or elevated rates of 502 error responses during the attack.
Resolution: Our automated DDoS protection systems quickly identified and mitigated the attack, restoring normal service operations without significant impact.

Actions Taken

In response to these ongoing attacks, we have implemented changes to the scaling of our platform infrastructure and the way that malicious traffic is intercepted.

Since these measures are being deployed actively in response to ongoing attacks, we cannot specify exactly what they are, but hopefully once these attacks subside we can provide further clarity.

Next Steps

We will continue to monitor our systems closely and adjust our security measures as needed. We will keep our users updated on any relevant developments or preventive measures being implemented.

Once the attacks have subsided or have been sufficiently mitigated, our team will conduct a post-mortem analysis of these incidents in order to identify any potential improvements to our security posture or our incident response techniques and processes.

We will publish a summary of our post-mortem with the findings of our investigation once it is complete.

Acknowledgment

We appreciate your understanding and patience during this incident. The swift resolution of this DDoS attack underscores our commitment to providing a secure and reliable platform. If you have any concerns or questions, please do not hesitate to contact our support team at support@cloud.gov.

Thank you for your continued trust in cloud.gov.

Posted Apr 15, 2024 - 11:45 EDT