We have resolved the issue with unallocated shards. Log intake and storage have been fully restored.
Posted Jun 23, 2020 - 13:48 EDT
Update
We are working to resolve an issue with unallocated shards in the cluster.
Posted Jun 23, 2020 - 13:23 EDT
Update
We in the process of adding resources to the cluster to increase capacity.
Posted Jun 23, 2020 - 11:57 EDT
Update
We are continuing to investigate this issue.
Posted Jun 23, 2020 - 11:05 EDT
Update
After restarting one of the Elasticsearch data nodes the Kibana front-end (the search interface at https://logs.fr.cloud.gov) seems to be operating normally for customers. Queued logs are being ingested and indexed. Not all of the nodes are 100% healthy yet, and we're continuing to investigate how to fully return to normal operations.
Posted Jun 23, 2020 - 10:10 EDT
Investigating
We are reviewing our production log search platform for errors on accessing customer log data and log searching.
Posted Jun 23, 2020 - 09:35 EDT
This incident affected: cloud.gov customer access (Logs front end) and cloud.gov customer applications (Logs intake and storage).