Brokered database logs not ingested

Incident Report for cloud.gov

Postmortem

Upon further investigation, the Cloud.gov team found that there was no gap in database log ingestion, and thus no incident.

While doing further querying to assess the impact of the apparent failed database log ingestion, a Cloud.gov engineer discovered that the query that was used to search for database logs was incorrect, which made it appear that database logs were missing and thus had failed to ingest. After fixing the query that is used to search for database logs, the engineer found that the database logs were ingested successfully with no gap from July 22 to August 26.

Database logs are continuing to ingest successfully now.

We apologize for any confusion. Please reach out to support@cloud.gov if you have any questions.

Posted Aug 27, 2025 - 16:55 EDT

Resolved

Update, August 27, 2025 - Upon further investigation, the Cloud.gov team found that **there was no gap in database log ingestion**. While doing further querying to assess the impact of the apparent failed database log ingestion, a Cloud.gov engineer discovered that the query that was used to search for database logs was incorrect, which made it appear that database logs were missing and thus had failed to ingest. After fixing the query that is used to search for database logs, the engineer found that the database logs were ingested successfully with **no gap from July 22 to August 26**. Database logs are continuing to ingest successfully now. We apologize for any confusion.

From roughly July 22, 2025 1:30 PM ET to August 26, 2025 1 PM ET , the ingestion of brokered database logs to https://logs.fr.cloud.gov was failing.

While the database log ingestion seems to have recovered and be working as of August 26, 2025, the Cloud.gov team needs to investigate what caused this incident and how to prevent it from recurring. We will conduct a post-mortem analysis of this incident in the coming days and share our findings.

Posted Aug 26, 2025 - 13:00 EDT