The incident has been resolved. We will add additional capacity on the afternoon of September 15, 2020.
Posted Sep 15, 2020 - 13:15 EDT
Update
We are continuing to monitor for any further issues. We will be adding capacity to the cluster this afternoon.
Posted Sep 15, 2020 - 13:08 EDT
Update
The log system is up, queries are working, and the system is still updating indexes. We expect the index updates to complete this morning and then an additional update will be performed to add additional capacity.
Posted Sep 15, 2020 - 07:38 EDT
Monitoring
Query functionality has been restored. The system is continuing to update. An additional update will be performed on September 15 to add additional capacity.
Posted Sep 14, 2020 - 19:07 EDT
Update
We have identified remediation steps to address the scale issue and are working to roll them out.
Posted Sep 14, 2020 - 17:29 EDT
Update
We believe we have correlated the issue in querying to increased load on the system. We are evaluating the best way to mitigate this issue and plan to take action shortly.
Posted Sep 14, 2020 - 16:33 EDT
Update
We are continuing to investigate this issue.
Posted Sep 14, 2020 - 15:19 EDT
Investigating
We are now seeing severely degraded performance on the logs frontend causing most or all queries to fail.
Posted Sep 14, 2020 - 14:15 EDT
Monitoring
We identified a server limit preventing queries from completing. We updated the configuration for this limit and are testing to confirm this has resolved the issue.
Posted Sep 14, 2020 - 12:57 EDT
Investigating
Users are currently unable to view logs at logs.fr.cloud.gov. Currently this appears to be only an issue viewing logs, not storing or shipping them.
Posted Sep 14, 2020 - 12:38 EDT
This incident affected: cloud.gov customer access (Logs front end).