Resolved -
Update, August 27, 2025 - Upon further investigation, the Cloud.gov team found that **there was no gap in database log ingestion**. While doing further querying to assess the impact of the apparent failed database log ingestion, a Cloud.gov engineer discovered that the query that was used to search for database logs was incorrect, which made it appear that database logs were missing and thus had failed to ingest. After fixing the query that is used to search for database logs, the engineer found that the database logs were ingested successfully with **no gap from July 22 to August 26**. Database logs are continuing to ingest successfully now. We apologize for any confusion.
From roughly July 22, 2025 1:30 PM ET to August 26, 2025 1 PM ET , the ingestion of brokered database logs to https://logs.fr.cloud.gov was failing.
While the database log ingestion seems to have recovered and be working as of August 26, 2025, the Cloud.gov team needs to investigate what caused this incident and how to prevent it from recurring. We will conduct a post-mortem analysis of this incident in the coming days and share our findings.
Aug 26, 13:00 EDT
Resolved -
From approximately 11:22 AM ET to 11:45 AM ET, some customers may have experienced unexpected 403 responses for requests to anything hosted on the cloud.gov platform. The effect was not widespread, as only a small amount of all traffic to cloud.gov overall received 403 responses.
The temporary 403 response rate spike was the result of a change to AWS WAF rules which was intended to block malicious traffic, but also had the side effect of blocking some legitimate traffic. As a result, we have disabled the WAF rule which caused the 403 errors and we will not re-enable it.
Thanks for your patience. if you have any questions, please contact us at support@cloud.gov.
Aug 21, 11:00 EDT
Resolved -
After adjusting some data node configuration, log ingestion has recovered and has stayed up to date.
As always, the Cloud.gov team takes these incidents very seriously. We will conduct a post-mortem analysis of this incident in the coming days and publish our findings.
Aug 14, 17:56 EDT
Identified -
We are again having issues with our log ingestion rate for https://logs.fr.cloud.gov. Consequently, there may be a delay before customer application logs appear on the system.
We are actively investigating the cause of the slow log ingestion and working towards a solution.
Aug 14, 15:51 EDT
Resolved -
After scaling up our logging infrastructure, we have seen log ingestion rates return to normal. Going forward, we are going to keep the infrastructure scaling at the new levels.
Identified -
We are having issues with our log ingestion rate for https://logs.fr.cloud.gov. Consequently, there may be a delay before customer application logs appear on the system.
We are are investigating the cause of the slow log ingestion and working on a solution. In the short term, we are scaling up our log ingestion infrastructure to increase our log ingestion rate.
Aug 14, 10:23 EDT