Partial log ingestion delay and failures - Resolved

Incident Report for cloud.gov

Postmortem

From November 19, 2025, 11:33 AM ET to December 15, 20925, 6 PM ET, Cloud.gov Logs experienced elevated ingestion failures for a small subset of application logs. Based on ingestion metrics collected during the incident window, approximately 0.0042% of total log events failed ingestion.

The issue was caused by a change to JSON log parsing combined with gaps in monitoring and test coverage. The parsing change was rolled back on December 15, and ingestion rates returned to normal. Log ingestion failures stopped by 6 PM ET.

There is no required customer action. We are adding monitoring and alerts for ingestion failures and improving test coverage for JSON log parsing scenarios.

Next update: No further updates are planned.

Posted Dec 19, 2025 - 16:42 EST

Resolved

Update, December 19, 2025 - The initial announcement described this incident as having only lasted for a few hours on December 15, 2025, but further analysis revealed that the incident really began on November 19, 2025 and continued until December 15, 2025. See the post-mortem analysis for fuller details.

Around 3:30 PM ET Monday, we detected that log ingestion into the Cloud.gov logging service was delayed by up to an hour, impacting the availability of recent logs.

Upon investigation, the team quickly identified that newly deployed log parsing code was contributing to the slowdown. By 4:30 PM ET Monday, we began a targeted rollback of that code, and ingestion throughput began trending back to normal. Over the following hours, ingestion returned to near real-time performance and full service availability.

During the event, we also observed an elevated rate of ingestion failures affecting only a subset of log streams, meaning some customers may have experienced partial or no log ingestion loss. We continue reviewing those failure patterns to ensure no residual impact remains.

If you have any questions or need assistance, please contact support@cloud.gov.

Posted Dec 15, 2025 - 15:30 EST