CDN-Route service partial outage
Incident Report for cloud.gov
Postmortem

The cdn-route service uses Let's Encrypt to provision and renew TLS certificates for custom user domains. Prior to this incident, all Let's Encrypt authorization requests were made under the same Let's Encrypt account, which caused us to reach the rate limit on pending authorizations as usage of the service increased. When attempting to provision new cdn-route instances after this limit was reached, users saw the Error creating new authz :: too many currently pending authorizations error message.

To fix this issue, we updated the cdn-route service to use a separate Let's Encrypt account for each instance, so that requesting a new certificate authorization for one user never impacts another user.

Posted May 23, 2017 - 13:34 EDT

Resolved
This incident has been resolved.
Posted May 18, 2017 - 10:11 EDT
Identified
We have identified the cause of the issue and are working to deploy a fix.
Posted May 17, 2017 - 12:56 EDT
Investigating
The CDN-Route service may fail to provision new instances with an error message that includes "Error creating new authz :: too many currently pending authorizations". We are investigating this issue and will release a fix as soon as possible.
Posted May 16, 2017 - 12:25 EDT