The kubernetes cluster scale-out is complete and customer Redis and Elasticsearch instances are back online. Some apps may need to be restarted via the dashboard or CLI. We are completing a DNS update deployment and will close the maintenance window once that is done.
Posted Oct 23, 2019 - 13:52 EDT
Update
A few Redis and Elasticsearch instances remain in an offline or degraded state. The team has repaired the internal DNS mismatch and is working to repair any remaining instances that are in an offline or degraded state.
Posted Oct 23, 2019 - 13:00 EDT
Update
Some Redis and Elasticsearch instances remain in an offline or degraded state. We are working to address the offline and degraded instances. The team has identified the current cause of diminished availability (related to Redis sentinel quorum) and continue to make progress towards restoration.
Posted Oct 23, 2019 - 10:52 EDT
Update
The majority of Redis and Elasticsearch instances are back online at this time, but there are still many offline or in a degraded state. We are working to address the offline and degraded instances.
Posted Oct 23, 2019 - 07:50 EDT
Update
Elasticsearch and Redis client instances continue to experience outages related to prior scheduled maintenance activities. Engineering team members continue to address this outage. We have successfully scaled the compute cluster and are actively working towards customer Elasticsearch and Redis instances online.
Posted Oct 23, 2019 - 05:26 EDT
Update
Elasticsearch and Redis client instances continue to experience outages related to prior scheduled maintenance activities. Engineering team members continue to address this outage. The configuration changes for the compute cluster have succeeded, we are working on scaling the compute cluster so all customer data instances can be scheduled accordingly in order to be brought online.
Posted Oct 23, 2019 - 04:23 EDT
Update
Elasticsearch and Redis client instances continue to experience outages related to prior scheduled maintenance activities. Engineering team members continue to address this outage. We have progressed passed the core control plane configuration and are moving through updating the compute nodes.
Posted Oct 23, 2019 - 02:52 EDT
Update
Elasticsearch and Redis client instances continue to experience outages related to prior scheduled maintenance activities. Engineering team members continue to address this outage. We have worked with AWS Support to determine the root cause and have implemented a short-term workaround, unblocking us and allowing us to continue with planned maintenance and the restoration of customer access to affected services.
Posted Oct 23, 2019 - 01:34 EDT
Update
Elasticsearch and Redis client instances continue to experience outages related to prior scheduled maintenance activities. Engineering team members continue to address this outage. We are still experiencing issues with dependent AWS services and are working with AWS Support to work through the remaining issues.
Posted Oct 23, 2019 - 00:46 EDT
Update
Elasticsearch and Redis client instances continue to experience outages related to prior scheduled maintenance activities. Engineering team members continue to address this outage. At 23:12 ET cloud.gov attempted to continue a production rollout, but we were affected by an AWS DNS outage which affects our ability to fetch S3 resources, causing an inability to continue rolling out changes. We have filed an issue with AWS Support and are continuing to monitor the situation.
Posted Oct 22, 2019 - 23:33 EDT
Update
Elasticsearch and Redis client instances continue to experience outages related to prior scheduled maintenance activities. Engineering team members continue to address this outage. The cluster datastore restored successfully, we do not expect any customer data loss and are working on restoring access to customer instances.
Posted Oct 22, 2019 - 22:22 EDT
Update
Elasticsearch and Redis client instances continue to experience outages related to prior scheduled maintenance activities. Engineering team members continue to address this outage. We have restored a backup and are working on replicating the cluster datastore (etcd) to restore dependent services.
Posted Oct 22, 2019 - 21:17 EDT
Update
Elasticsearch and Redis client instances are currently experiencing outages related to scheduled maintenance activities. We are currently testing restoration of a backup that should enable availability of the cluster datastore (etcd) and dependent services.
Posted Oct 22, 2019 - 17:30 EDT
Update
Elasticsearch and Redis client instances are currently experiencing outages related to scheduled maintenance activities. We are currently troubleshooting the cluster state datastore (etcd). Once completed, services will begin automatic recovery at that time.
Posted Oct 22, 2019 - 16:47 EDT
Update
Some Elasticsearch and Redis client instances are currently experiencing outages related to scheduled maintenance activities. The cluster controller update is expected to complete within 30 minutes and affected instances will begin restoration at that time.
Posted Oct 22, 2019 - 15:38 EDT
In progress
Scheduled maintenance is currently in progress. We will provide updates as necessary.
Posted Oct 22, 2019 - 11:00 EDT
Scheduled
As part of our continued commitment to enhance and improve operational reliability for cloud.gov, we have identified the need for increasing resources available to our Elasticsearch and Redis cluster. The cloud.gov team will begin maintenance on the Redis / Elasticsearch environment that is likely to impact the operation of these services. Throughout the maintenance window, impacts to Redis / Elasticsearch may include slow operation and downtime. Customers who are not currently using the Redis or Elasticsearch services will not be impacted.
During the maintenance window, we will perform the following operations: -- Add additional servers to the cluster (scale up) to improve capacity -- Redistribute Redis and Elasticsearch workloads across the cluster
Our team will provide updates on the operational availability of Redis / Elasticsearch services throughout the maintenance period
Posted Oct 11, 2019 - 14:33 EDT
This scheduled maintenance affected: cloud.gov customer applications (Redis, Elasticsearch).