Applications on the domain were unavailable from 5:35pm ET Saturday December 24th to 12:40am Sunday December 25th
Incident Report for

What happened

When we received the first report from our colleague, we helped them identify that there was no route to the application, and they recreated it. We soon realized that other applications on had similarly lost their routes. We audited and recreated routes for all of the applications that should be accessible via the domain.

What went wrong

When we received a similar second report, a operator realized that the sudden absence of routes was related to their attempt to remove access to the domain for a single organization. Instead of removing access from the intended organization, the operator had removed the domain from the system altogether, causing the deletion of all routes based on across organizations.

Steps we're taking as a result

  • We'll work with the upstream Cloud Foundry CLI team to ensure the cf delete-shared-domain command warns when there are existing routes and requests confirmation before actually performing the deletion.
Posted Dec 30, 2016 - 18:52 EST

Colleagues in 18F reported that applications hosted on the domain (developed and maintained by 18F!) were inaccessible. We found that routes to the applications had disappeared, resulting in 404 errors. We recreated the routes, which restored access.
Posted Dec 30, 2016 - 18:31 EST