As of 8pm Eastern time, the cloud.gov operations team is no longer seeing arbitrary HTTP 5xx errors on application operations.
Posted Sep 24, 2019 - 20:00 EDT
The Amazon Web Services capacity issues have been resolved. We are continuing to watch the production rollout.
Posted Sep 24, 2019 - 19:16 EDT
As of 3pm Eastern Time, the cloud.gov operations team was notified of a production rollout failure. This production rollout was tested to be a non-breaking, non-disruptive change. During the production rollout, Amazon Web Services EC2 in GovCloud no longer had the capacity to provision virtual machines due to lack of available capacity, causing the deployment to error and fail.
As a result of this deployment failure, our production system was in an inconsistent state. This inconsistent state may potentially cause HTTP 5xx errors when performing operations on customer applications, such as redeploying, scaling, or updating.
We are actively monitoring the issue and are currently attempting a redeployment of the production platform.
Posted Sep 24, 2019 - 16:34 EDT
This incident affected: Services cloud.gov depends on (AWS EC2-us-west-1, AWS EC2-us-west-2) and cloud.gov customer access (API).