Throughout this week, our platform has been subject to massive, coordinated Distributed Denial of Service (DDoS) attacks.
Today, on 4/12/2024, our platform experienced another DDoS attack that took our platform down for around 5 minutes.
Thanks to our security measures and platform automation, we were able to fully recover and mitigate the effects of the DDoS attack in under five minutes.
Unfortunately, while deploying additional mitigations for the underlying source of the DDoS attacks, there was an interruption to all traffic coming into our platform from a CDN, including traffic for cloud.gov Pages customers.
Incident Details - DDoS outage
Time Detected: 4/12/2024 1:54 PM ET Duration: Around 5 minutes Impact: Users may have experienced slow response times or temporary inability to access our services during the attack. Resolution: Our automated DDoS protection systems quickly identified and mitigated the attack, restoring normal service operations without significant impact.
Incident Details - CDN-based traffic outage
Time Detected: 4/12/2024 2:25 PM ET Duration: Around 5 minutes Impact: All customers whose traffic passes through a CDN, including cloud.gov Pages customers or users of brokered CDN services, experienced a full outage of their services. Resolution: We manually reverted the change which caused CDN-based traffic to be rejected and also reverted the change in the infrastructure source code, so that the change will not be deployed again.
Actions Taken
Our DDoS mitigation tools were activated to rate limit malicious traffic, allowing the platform to recover from the initial DDoS attack.
Our security team is conducting a thorough investigation into the attack to understand its origins and to prevent similar incidents in the future.
Next Steps
We will continue to monitor our systems closely and adjust our security measures as needed. An in-depth review of this incident is being conducted to identify any potential improvements to our security posture. We will keep our users updated on any relevant developments or preventive measures being implemented.
We will publish a post-mortem with the findings of our investigation in the coming days.
Acknowledgment
We appreciate your understanding and patience during this incident. The swift resolution of this DDoS attack underscores our commitment to providing a secure and reliable platform. If you have any concerns or questions, please do not hesitate to contact our support team.