Service broker outage

Incident Report for cloud.gov

Postmortem

Summary

From approximately 12:22 PM to 7:19 PM ET on Thursday, June 11, 2026, Cloud.gov service broker management actions were unavailable. During this window, customers could not create, update, or delete brokered services.

Existing brokered services, including S3 buckets, RDS databases, and other already provisioned services, remained available. Customer applications continued to run, and we did not observe application downtime from this incident.

Impact

During the incident, customers were unable to:

Provision new brokered services
Update existing brokered services
Delete existing brokered services

Existing service instances and customer applications were not interrupted.

Timeline

All times are Eastern Time.

12:22 PM - An automated credential rotation process changed IAM credentials used by Cloud.gov service brokers.
12:57 PM - A Cloud.gov engineer identified that some service broker IAM credentials were no longer working as expected.
1:35 PM - The Cloud.gov began coordinated incident response.
2:34 PM - The Cloud.gov began restoring IAM credentials and redeploying affected service brokers.
7:19 PM - The final service broker was redeployed. All Cloud.gov service brokers were confirmed operational.

Resolution

Cloud.gov created replacement IAM credentials, redeployed the affected service brokers, and tested broker operations. After validation, the team confirmed that customers could again create, update, and delete brokered services.

Follow-up Actions

Cloud.gov will take the following actions to reduce the change of a similar incident and improve recovery:

Add alerting to detect service broker IAM credential failures sooner.
Review IAM credential rotation procedures for service brokers.
Add stronger safeguards for automated credential rotation.
Improve automation for redeploying service brokers after credential changes.
Review incident response runbooks for service broker credential failures.

Thank you for your patience while we resolved this issue. If you have any questions, please contact us at support@cloud.gov.

Posted Jun 15, 2026 - 17:15 EDT

Resolved

Since we deployed our fixes at approximately 5:27 PM ET, the Cloud.gov team has observed no further indications of failures from the services that manage AWS resources. All customer-facing services and service brokers should be fully operational. If you are still experiencing issues, please contact us at support@cloud.gov.

As always, the Cloud.gov team will be conducting a post-mortem analysis of this incident in the coming days. We will share our findings and next steps once our analysis is complete.

Posted Jun 12, 2026 - 09:25 EDT

Monitoring

All services on Cloud.gov that manage AWS resources should now be operational. If you are still experiencing issues, please contact us at support@cloud.gov.

Posted Jun 11, 2026 - 17:27 EDT

Identified

The Cloud.gov team has identified the cause of the outage as inadvertent access key rotation for Cloud.gov applications that manage AWS resources, such as service brokers for RDS databases.

The AWS access keys themselves that customers use to access their resources (e.g. RDS databases, S3 buckets) **are unaffected and are still working**.

The Cloud.gov team is actively working to restore functionality for services that manage AWS resources.

Posted Jun 11, 2026 - 14:48 EDT

Investigating

Service brokers that Cloud.gov customers use to manage AWS resources (e.g. RDS, S3, etc) are currently experiencing an outage. The Cloud.gov team is investigating and will provide further updates as we know more.

Posted Jun 11, 2026 - 13:52 EDT