Unable to view logs dashboard

Incident Report for cloud.gov

Resolved

The incident has been resolved. We will add additional capacity on the afternoon of September 15, 2020.
Posted Sep 15, 2020 - 13:15 EDT

Update

We are continuing to monitor for any further issues. We will be adding capacity to the cluster this afternoon.
Posted Sep 15, 2020 - 13:08 EDT

Update

The log system is up, queries are working, and the system is still updating indexes. We expect the index updates to complete this morning and then an additional update will be performed to add additional capacity.
Posted Sep 15, 2020 - 07:38 EDT

Monitoring

Query functionality has been restored. The system is continuing to update. An additional update will be performed on September 15 to add additional capacity.
Posted Sep 14, 2020 - 19:07 EDT

Update

We have identified remediation steps to address the scale issue and are working to roll them out.
Posted Sep 14, 2020 - 17:29 EDT

Update

We believe we have correlated the issue in querying to increased load on the system. We are evaluating the best way to mitigate this issue and plan to take action shortly.
Posted Sep 14, 2020 - 16:33 EDT

Update

We are continuing to investigate this issue.
Posted Sep 14, 2020 - 15:19 EDT

Investigating

We are now seeing severely degraded performance on the logs frontend causing most or all queries to fail.
Posted Sep 14, 2020 - 14:15 EDT

Monitoring

We identified a server limit preventing queries from completing. We updated the configuration for this limit and are testing to confirm this has resolved the issue.
Posted Sep 14, 2020 - 12:57 EDT

Investigating

Users are currently unable to view logs at logs.fr.cloud.gov. Currently this appears to be only an issue viewing logs, not storing or shipping them.
Posted Sep 14, 2020 - 12:38 EDT
This incident affected: cloud.gov customer access (Logs front end).