Applications on the 18f.gov domain were unavailable from 5:35pm ET Saturday December 24th to 12:40am Sunday December 25th
Incident Report for cloud.gov
Postmortem

What happened

When we received the first report from our colleague, we helped them identify that there was no route to the application, and they recreated it. We soon realized that other applications on 18f.gov had similarly lost their routes. We audited and recreated routes for all of the applications that should be accessible via the 18f.gov domain.

What went wrong

When we received a similar second report, a cloud.gov operator realized that the sudden absence of routes was related to their attempt to remove access to the 18f.gov domain for a single organization. Instead of removing access from the intended organization, the operator had removed the domain from the system altogether, causing the deletion of all routes based on 18f.gov across organizations.

Steps we're taking as a result

  • We'll work with the upstream Cloud Foundry CLI team to ensure the cf delete-shared-domain command warns when there are existing routes and requests confirmation before actually performing the deletion.
Posted Dec 30, 2016 - 18:52 EST

Resolved
Colleagues in 18F reported that applications hosted on the 18f.gov domain (developed and maintained by 18F!) were inaccessible. We found that routes to the applications had disappeared, resulting in 404 errors. We recreated the routes, which restored access.
Posted Dec 30, 2016 - 18:31 EST