Command line interface commands and dashboard features will return errors (GovCloud environment)
Scheduled Maintenance Report for cloud.gov
Postmortem

What happened

We regularly upgrade cloud.gov platform components to their latest versions in order to deliver new features and security patches, as well as stability and operational improvements, as routine maintenance without customer application downtime. In the process of performing a scheduled upgrade on January 6, 2018, we caused some applications hosted on cloud.gov to be unavailable between 1:32 and 1:42 PM EST.

After completing an upgrade of a component that hosts customer applications, we started removing the previous version of this component before fully migrating customer applications to the new version. This error happened because we performed a manual step to migrate applications with the intention of preventing any momentary downtime, which didn’t successfully prevent downtime.

Once the previous version of the component was removed, all customer applications were automatically migrated to the new version and became available again.

What we’re doing

In the future the cloud.gov operations team will:

  • Better communicate expected behavior during scheduled maintenance. If momentary downtime of customer applications is likely to happen during a planned maintenance step, we will communicate this proactively, instead of taking on additional technical complexity to try to prevent the downtime that may not be successful.
  • Improve our processes so that all steps in our changes are automated and fully tested before being taken in production.
Posted about 1 year ago. Feb 20, 2018 - 13:22 EST

Completed
We have completed this scheduled maintenance.

Note: some applications may have been briefly unavailable between 1:32 and 1:42 PM EST. We apologize for the inconvenience and will write up a detailed post-mortem as soon as possible.
Posted over 1 year ago. Jan 06, 2018 - 14:38 EST
In progress
Scheduled maintenance is currently in progress. We will provide updates as necessary.
Posted over 1 year ago. Jan 06, 2018 - 10:00 EST
Scheduled
cloud.gov API actions (including command line interface commands and dashboard features) in the GovCloud environment will be intermittently unavailable and return errors while we do scheduled maintenance for the platform.

During this time, you should not run command line interface commands or take actions on the dashboard. For example, if you try to restage or restart an app, the deploy may return errors and fail, causing an outage for your application with no way to restart it until the maintenance is complete.
Posted over 1 year ago. Jan 05, 2018 - 13:28 EST