In response to the “Brokered S3 credentials written to internal logs” incident, cloud.gov has held an incident postmortem and identified several changes the team will make to the system:
We will implement a filter in our log ingestor for log lines matching well-known access key patterns. We will not ingest log lines that match the patterns, and the system will alert the cloud.gov operations team for investigation.
We will audit the log level of all platform applications to make sure it is appropriate for the production environment.
We will add a ticket to our backlog to route logs from our service brokers to our internal-only log system, instead of a tenant in the customer-facing log system.
Thank you. As always, please feel free to reach out to our support team at support@cloud.gov if you have questions or concerns.
Posted Oct 25, 2023 - 15:45 EDT
Resolved
What happened?
On Friday October 13, a cloud.gov engineer found that the cloud.gov S3 broker, which manages the S3 buckets that customers broker using the platform, was printing AWS secret access keys in clear text to its logs. Specifically, whenever a customer created a service key or created a binding between an S3 service instance and one of their applications, the newly created key was written to the broker log and indexed by our log management system.
How has cloud.gov responded?
We have modified the S3 broker to stop printing these credentials and we have removed all log lines containing the keys from our active log indices. However, the log lines will remain in our long-term log archives due to our retention obligations.
We emailed the managers for all organizations on cloud.gov on October 17 to notify them of the incident. Per our incident response process, we are cross-posting the information to Statuspage now so we can publish a public postmortem.
What is the scope of the issue?
The relevant code and configuration was introduced more than six years ago. Because of the time span, it is likely that all current and past brokered S3 buckets are affected.
Next steps
Because of the factors listed above, although there was no public compromise of your credentials, we recommend rotating your access keys at this time.
You can rotate your keys by un-binding and re-binding your S3 buckets to each bound application, and deleting and re-creating any service keys associated with the service instance. Full instructions have been sent to the managers of all cloud.gov organizations. If you did not receive this email or require additional assistance, please reach out to support@cloud.gov for help.
Is my data at risk?
Cloud.gov does not believe your data is at risk. The logging system is designed to segregate logs from customer tenant access, and we have no indications of unauthorized access. We are continually monitoring our system and will reach out to your team directly if we detect any suspicious activity.
Why did this happen?
Cloud.gov has held an incident postmortem and will post it to Statuspage shortly.
Please feel free to reach out to our support team at support@cloud.gov if you have questions or concerns.