For approximately 22 hours, the cloud.gov platform was unable to send emails for password resets and user invites.
On September 12, as part of a routine task, we pushed data to an approved software service that we use. In this data, we inadvertently included credentials for a service account that we use to send email from cloud.gov to customers (such as password resets and user invites). Within 20 minutes, we recognized the issue and started our security incident response process, which included deleting the data from the service, invalidating the credentials, and reviewing our audit logs to verify that the credentials were not used.
To prioritize security, we chose to first invalidate all the credentials, then later update systems that use them, knowing that this would cause password resets and user invites to be unavailable in the meantime.
On September 13, we updated the email service account credentials, which re-enabled emails for customers.
We also completed auditing the logs for signs of unauthorized use of credentials, and we found no evidence of unauthorized use.
We use an open-source tool (git-seekret) to help prevent secrets from being accidentally exposed. We were able to determine that this tool was in place, scanned the files committed and found no secrets in this case. We’re working to refine the scanning patterns we use to help prevent this type of failure in the future.