What happened

We regularly upgrade cloud.gov platform components to their latest versions in order to deliver new features and security patches, as well as stability and operational improvements, as routine maintenance without customer application downtime. In the process of performing a scheduled upgrade on January 6, 2018, we caused some applications hosted on cloud.gov to be unavailable between 1:32 and 1:42 PM EST.

After completing an upgrade of a component that hosts customer applications, we started removing the previous version of this component before fully migrating customer applications to the new version. This error happened because we performed a manual step to migrate applications with the intention of preventing any momentary downtime, which didn’t successfully prevent downtime.

Once the previous version of the component was removed, all customer applications were automatically migrated to the new version and became available again.

What we’re doing

In the future the cloud.gov operations team will:

• Better communicate expected behavior during scheduled maintenance. If momentary downtime of customer applications is likely to happen during a planned maintenance step, we will communicate this proactively, instead of taking on additional technical complexity to try to prevent the downtime that may not be successful.
• Improve our processes so that all steps in our changes are automated and fully tested before being taken in production.