Interruption in customer application log availability
Incident Report for cloud.gov
Resolved
Log intake is normal and up to date.
Posted about 2 months ago. May 24, 2019 - 09:22 EDT
Monitoring
We've scaled out our logging cluster and are now processing logs faster than they're coming in. We're continuing to monitor.
Posted about 2 months ago. May 23, 2019 - 21:45 EDT
Identified
We've determined that a massive influx of logs has caused our log ingestion system to back up. Logs are still being ingested, but at a slower rate than normal. We are working to scale out our log aggregation to accommodate the increased load.
Posted about 2 months ago. May 23, 2019 - 20:48 EDT
Update
We are continuing to investigate this issue.
Posted about 2 months ago. May 23, 2019 - 17:23 EDT
Investigating
Customers are reporting that logs.fr.cloud.gov (Kibana) is not showing application logs since about 6:40 am ET today. Our engineering team is investigating why this is happening, and we will provide updates as we research it.
Posted about 2 months ago. May 23, 2019 - 17:23 EDT
This incident affected: cloud.gov customer applications (Logs intake and storage).