Email service from cloud.gov interrupted
Incident Report for cloud.gov
Postmortem

What Happened

Summary

For approximately 22 hours, the cloud.gov platform was unable to send emails for password resets and user invites.

Timeline

On September 12, as part of a routine task, we pushed data to an approved software service that we use. In this data, we inadvertently included credentials for a service account that we use to send email from cloud.gov to customers (such as password resets and user invites). Within 20 minutes, we recognized the issue and started our security incident response process, which included deleting the data from the service, invalidating the credentials, and reviewing our audit logs to verify that the credentials were not used.

To prioritize security, we chose to first invalidate all the credentials, then later update systems that use them, knowing that this would cause password resets and user invites to be unavailable in the meantime.

On September 13, we updated the email service account credentials, which re-enabled emails for customers.

We also completed auditing the logs for signs of unauthorized use of credentials, and we found no evidence of unauthorized use.

What We’re Doing

Preventing more secrets from being exposed

We use an open-source tool (git-seekret) to help prevent secrets from being accidentally exposed. We were able to determine that this tool was in place, scanned the files committed and found no secrets in this case. We’re working to refine the scanning patterns we use to help prevent this type of failure in the future.

Posted Sep 17, 2019 - 17:43 EDT

Resolved
Email services are now operational.
Posted Sep 13, 2019 - 17:06 EDT
Update
We are continuing to work on a fix for this issue and expect to resolve it today.
Posted Sep 13, 2019 - 12:16 EDT
Update
We are continuing to work on a fix for this issue.
Posted Sep 12, 2019 - 16:56 EDT
Identified
Emails sent from cloud.gov to customers is currently impacted, including password resets, sandbox creation/deletion notifications, buildpack notifications, and other service emails.
Posted Sep 12, 2019 - 16:56 EDT
This incident affected: cloud.gov customer access (Login).