What happened?

During routine platform updates that happen a few times a week, hosts for customer applications are rotated out of circulation. When this happens, customer applications are automatically relocated onto other available hosts. This time, our hosts were near their capacity based on the number of apps the platform was already hosting. Taking out a single host out of rotation was enough to put other hosts over their limit as apps were migrated. This prevented new apps from being scheduled. Users began seeing an “Insufficient Resources” error message when pushing applications, and our own tests started failing. Once we identified the problem, we increased the disk available to all hosts. After this change rolled out, the problem was resolved.

What we’re doing about it

• We’re adding alerting so we’ll know when application hosts are nearing capacity for new applications.
• We’re adding alerting for the specific “Insufficient Resources” situation.
• We’re documenting how to increase capacity so anyone on the team can do it.
• We’re going to start formally reviewing our capacity with each new customer added to cloud.gov.
• We’re going to start formally reviewing our utilization on a regular basis so we can make better tuning decisions.