Summary
- An external cable fault interrupted the remaining active utility feeder while the facility was in a reduced-redundancy maintenance configuration.
- PLC logic incorrectly interpreted voltage upstream of an open breaker as evidence of a usable electricity supply.
- Google has outlined control-system, testing, maintenance, and operational changes following a six-hour, eight-minute period of primary disruption.
A configuration error in the controls governing backup generator startup contributed to October’s Google Cloud outage in London, according to the company’s detailed technical investigation.
Google Cloud identified a combination of voltage sensing and programmable logic controller (PLC) configuration problems at the facility serving its europe-west2-b availability zone.
The incident began on 20 October after an external underground cable fault interrupted the electricity supply on the site’s active utility feeder. A second primary feeder had been isolated for corrective maintenance following problems identified during a generator load test on 14 October.
The facility normally uses a duplicated 2N electricity architecture, providing independent power paths for critical loads. During the maintenance period, however, the site was operating in a temporary configuration dependent on one primary utility feed.
When that remaining feed failed, uninterruptible power supply (UPS) batteries initially maintained electricity to the critical equipment. The backup generators were expected to start automatically and assume the load before the battery reserves were exhausted.
They did not receive the required start signal. Once the UPS batteries had been depleted, data hall equipment lost power, affecting servers, storage infrastructure, and networking systems.
Google’s final incident report was published on 4 November. It established the technical sequence behind a disruption previously reported primarily as an external power event.
Control logic failed to identify usable supply
The investigation found that the PLC logic did not correctly combine voltage measurements with the positions of the utility feeder circuit breakers.
Voltage sensing inputs were positioned upstream of the breakers. The isolated feeder therefore continued to show voltage on the incoming side, even though its open breaker prevented that supply from reaching the facility’s internal electrical distribution.
When the active feeder failed, both the primary and redundant PLC controllers detected the voltage remaining on the isolated path. Because the controllers used identical logic, both treated that reading as evidence that a healthy utility supply remained available.
The controllers consequently failed to issue the automatic generator start command. The presence of duplicate controllers did not protect against the shared configuration problem.
Google also reported that the misleading voltage indication inhibited an attempted remote load transfer. Engineers subsequently used the standby utility connection to restore electricity to the facility.
Restoration of the data hall supply was completed at approximately 08:20 BST on 20 October. Computing services did not immediately return, because storage and other infrastructure required further recovery work.
The electrical failure exposed the difference between having redundant equipment installed and ensuring that its controls respond correctly to the actual state of the electrical system.
Generator capacity, duplicated controllers, and UPS batteries could not provide the intended continuous supply once the controls incorrectly interpreted the incoming utility configuration.
Recovery exposed wider regional dependencies
Google’s service incident began affecting customers at approximately 07:00 BST. The primary disruption lasted six hours and eight minutes, with most workloads restored by around 13:08 BST.
The company published the final closure notification later that afternoon, after a smaller number of remaining workloads had recovered. The closure time should not be confused with the period of primary service impact.
Although the physical electricity failure was confined to the infrastructure supporting europe-west2-b, customers using other parts of the London region also experienced problems.
Traffic redirected towards unaffected zones increased demand on available computing systems. Regional control-plane dependencies also caused timeouts and other errors when services attempted to communicate with components in the unavailable zone.
Once electrical power was restored, engineers had to recover storage systems that had experienced unexpected shutdowns. Some workloads remained unavailable while data integrity checks and related restoration activities were completed.
Google delayed parts of the service recovery process until storage systems had reached a suitable condition. Traffic was then reintroduced gradually to avoid additional pressure on the recovering infrastructure.
The company has undertaken corrective work on the PLC configuration and generator testing procedures. It has also outlined further reviews of similar control arrangements at other facilities, including tests covering different utility failure scenarios.
Emergency operating procedures have been updated to include manual response measures during the limited runtime available from UPS batteries. Google also introduced additional change-management controls for facilities with comparable electrical arrangements.
Some remediation activities were completed during October, while others had later November or December target dates in the published report. Those deadlines should not be treated as confirmation that every measure is already complete.
The incident demonstrates how maintenance configuration, electrical sensing, and common control logic can combine to defeat an otherwise redundant power system. The subsequent regional service disruption also shows that a physically localised data centre failure can affect cloud services beyond the immediate facility.
