Interruption in customer application log availability

Incident Report for cloud.gov

Resolved

Log intake is normal and up to date.

Posted May 24, 2019 - 09:22 EDT

Monitoring

We've scaled out our logging cluster and are now processing logs faster than they're coming in. We're continuing to monitor.

Posted May 23, 2019 - 21:45 EDT

Identified

We've determined that a massive influx of logs has caused our log ingestion system to back up. Logs are still being ingested, but at a slower rate than normal. We are working to scale out our log aggregation to accommodate the increased load.

Posted May 23, 2019 - 20:48 EDT

Update

We are continuing to investigate this issue.

Posted May 23, 2019 - 17:23 EDT

Investigating

Customers are reporting that logs.fr.cloud.gov (Kibana) is not showing application logs since about 6:40 am ET today. Our engineering team is investigating why this is happening, and we will provide updates as we research it.

Posted May 23, 2019 - 17:23 EDT

This incident affected: cloud.gov customer applications (Logs intake and storage).