Equinix IBX Site Staff reports that the site is still recovering and the average site temperatures have reduced an additional 2 degrees since the last update to 85° F. IBX Critical Facility Engineers along with onsite mechanical vendors continue working to restore the 5th Chiller. Note that temperatures may vary at different locations across the site. Fourteen portable coolers are online and the final portable cooler install is pending electrical wiring. For customers who powered down equipment, we strongly recommend waiting to restore operations until the average site temperatures reach around 80 °F
Posted Jan 15, 2024 - 22:12 UTC
Update
Equinix IBX Site Staff reports that one of the mechanical vendors arrived on site to assist in troubleshooting and repair. Another mechanical vendor is en route to the IBX. The site is still recovering and the average temperature remains stable at 91° F. Thirteen portable coolers are online and two more continue to be installed. For customers who powered down equipment, we strongly recommend waiting to restore operations until the temperatures reach a lower overall range.
Our whole Chicago workload is still fully online and working properly, only peering and routings were affected. We will keep this incident open as we are still at risk of failures until full cooling capacity at equinix is restored.
Posted Jan 15, 2024 - 16:30 UTC
Update
Equinix IBX Site Staff reports that colocation temperatures continue to decrease slowly with a current average temperature at 85F degrees. Twelve portable coolers are now online and three more are being installed. For customers who powered down equipment, we strongly recommend waiting to restore operations until the temperatures reach a lower overall range.
Posted Jan 15, 2024 - 12:46 UTC
Update
Equinix IBX Site Staff reports that colocation temperatures continue to remain stable with a current average temperature remaining at 95F degrees. Four portable coolers are now online and eleven more are being installed. For customers who powered down equipment, we strongly recommend waiting to restore operations until the temperatures reach a lower overall range.
Posted Jan 15, 2024 - 08:46 UTC
Monitoring
A fix has been implemented and we are monitoring the results.
Posted Jan 15, 2024 - 05:26 UTC
Update
Equinix IBX Site Staff reports that temperatures have stabilized with an average temperature of 120 F degrees. One chiller is now fully operational, and another chiller is online but operating at reduced capacity. Equinix IBX Critical Facility Engineers continue to work on restoring the required 4-out-of-6 chillers to return cooling back to normal operations. Portable chillers are currently enroute with an estimated arrival time of 21:00 Site Local Time. Upon arrival, IBX Critical Facility Engineers and our vendor partner will work diligently to get the portable chillers online as quickly as possible. We will send another update once the portable chillers are operational.
If not already implemented, Equinix site staff highly recommends that customers power down non-essential loads within the data center and/or transfer those operations to backup sites to assist with efforts at lowering the temperature within the facility.
The severe weather in the Chicago area, contributing to the initial failure, continues to hamper efforts at resolution, but our teams are working diligently to resolve the issue as quickly as possible. IBX Critical Facility Engineers continue to supply outside air to decrease the collocation area's temperature, and all fans have been deployed to support cooling the IBX.
Posted Jan 15, 2024 - 03:28 UTC
Identified
The routing issues in Chicago are due overheating issues in Equinix CHI1.
Latest update from Equinix: Equinix engineers are still working with the vendor to rectify the fault Latest update from Zayo: Our facility provider at 350 E Cermak has advised their chillers have stopped due to extreme outdoor temperatures. Our vendor has HVAC repair techs onsite and is working to get the units back in operation at this time. The unit failures is causing high temps on our equipment causing failure and flaps. We are working with them now to try to restore temperatures and then equipment. We will update as progress is made.
Posted Jan 15, 2024 - 02:46 UTC
Investigating
NA - US - Chicago routing issues - workload taken out of production while routing issues are being investigated
Posted Jan 15, 2024 - 02:30 UTC
This incident affected: Gameservers NA (USA - Chicago).