Some account actions are temporarily unavailable
Incident Report for cloud.gov
Postmortem

What happened

The application that provides account-related functions (self-signup, inviting other users, and password management for cloud.gov [non-SSO] accounts) was unavailable due to the failure of a Redis instance that the application depends upon.

Once we identified the issue, we repaired the failing instance, and the application returned to normal operation.

The dashboard relies on Redis as well, so it was also affected by a failing Redis instance for a short time. We received an alert for this and fixed it.

What we’re doing

We are reviewing and updating the configuration of our alerting systems. This will ensure that when similar failures occur in the future, we will be able to respond quickly and restore service before any users are impacted.

Posted Jun 16, 2017 - 09:59 EDT

Resolved
This incident has been resolved.
Posted Jun 12, 2017 - 16:13 EDT
Monitoring
A fix has been implemented and we are monitoring the results.
Posted Jun 12, 2017 - 15:09 EDT
Update
The dashboard was unavailable as well from 12:36 pm ET to 1:09 pm ET; it is available again. We continue to work on making the account actions available again.
Posted Jun 12, 2017 - 13:22 EDT
Investigating
As of 10:07 am ET, the following actions are unavailable (giving error messages):

Signing up for cloud.gov access (https://account.fr.cloud.gov/signup)
Inviting new users (https://account.fr.cloud.gov/invite)
For cloud.gov accounts (non-SSO login): changing password (https://account.fr.cloud.gov/change-password)
For cloud.gov accounts (non-SSO login): forgot password (https://account.fr.cloud.gov/forgot-password)

We’ve identified the issue, and we’re in the process of fixing it.
Posted Jun 12, 2017 - 12:34 EDT