A series of lightning strikes in Europe have resulted in the permanent loss of important Google data from its cloud processing service. Google basically carries out the computing process at a data center in Belgium which offers virtual machines for processing on pay-per-minute basis.
Media reports state that these lightning strikes also hit a local electricity grid, which resulted in a massive loss of power at the data center. Although automatic auxiliary systems restored power quickly, and the storage systems are designed with battery backup, some recently written data was located on storage systems which were more susceptible to power failure from extended or repeated battery drain.
The affected disks returned I/O errors to their attached GCE instances, and also typically returned errors for management operations such as snapshot creation. Google faced around 5 per cent of disk failure, out of which its engineers could recover only a part of the data.However, Google has admitted that it was the fault of their engineers, apologised and analysed this issue immediately. It then confirmed that SSD Persistent Disks, disk snapshots, and Local SSDs were not affected by the incident.
By August 17, only a very small number of disks remained affected, totalling less than 0.000001 per cent of the space of allocated persistent disks in Europe. In these cases, full recovery is not possible. Google is also working on upgrading its hardware and its Persistent Disk storage is currently running on this hardware. Google engineers have also conducted a series of reviews on electrical distribution systems through computing hardware as well as the software controlling the GCE persistent disk layer.
