We are resolving this incident as it was never an issue with cloud.gov. We created this issue when a customer reported issues, and there was some question whether some of the TLS certificates we had issued did not have the correct trust chain, but that was not the case.
For context, if you use TLS certificates through our external domain service (https://cloud.gov/docs/services/external-domain-service/), those certs are issued by Let's Encrypt. The certificates serve a trust chain given to us by Lets Encrypt. Clients can use the first cert in the chain to build a full chain up to "DST Root CA X3", which expired 30 September 2021, or the second cert in the chain to build a full chain up to "ISRG Root X1".
If a client (e.g. the web browser on an older system) has "DST Root CA X3" as a trust anchor but not "ISRG Root X1", they will probably get a certificate validation error because "DST Root CA X3" expired earlier today.
If they have BOTH certs in their trust anchors, it's possible they'll get an error, as "DST Root CA X3" is expired, and the client may give up after constructing a bad chain, but most well-behaved clients will continue checking for a valid chain and find it. However, either client configuration is wholly outside cloud.gov's purview.
If you have questions, please consult the resources above, or open a support issue.
Posted Sep 30, 2021 - 15:36 EDT
Monitoring
We have been unable to confirm any SSL/TLS certificate issues for cloud.gov hosted applications. The reports so far seem to only impact clients that have outdated certificate trust stores (for example, for users running older OS/Browser combinations). We will continue to investigate any issues that may be server-side on cloud.gov, as there are some media reports that some certificate chains may have incorrect intermediate certs.
If you are having issues with any cloud.gov-hosted applications, confirm this by:
- testing the site with another device, such as your mobile phone - testing your browser by visiting other LetsEncrypt sites such as https://www.navy.mil or https://letsencrypt.org/ -- if these fail the problem is client side (not cloud.gov) - determining if other users on your network are experiencing similar issues
Posted Sep 30, 2021 - 12:30 EDT
Investigating
We are investigating issues with the SSL/TLS certificates for some cloud.gov partner sites. We updated all customer TLS certificates earlier in 2021 to avoid issues with LetsEncrypt root certificate expiration on 2021-09-30 (https://letsencrypt.org/2020/09/17/new-root-and-intermediates.html).
However, about 5% cloud.gov hosted sites may have an expired intermediate certificate.
We are working to mitigate the issue, and issue new certs as needed. A fuller notice on our findings and any steps to mitigate will be provided as soon as possible.
Posted Sep 30, 2021 - 11:20 EDT
This incident affected: cloud.gov customer applications (Service - Custom Domain Service, External domain service, External domain service - CDN).