Google said today a storage software update was responsible for causing some Gmail users to lose access to their e-mail data and said full functionality would be restored soon.
Some Gmail users–less than 0.02 percent actually affected, according to Google–complained yesterday of suddenly and mysteriously losing e-mails, contacts, and folders. In an update on the situation this evening, Ben Treynor, Google VP of engineering and site reliability czar, apologized for the inconvenience and said the company expects to have the lost data restored soon.
“The good news is that email was never lost and we’ve restored access for many of those affected,” Treynor wrote in a company blog. “Though it may take longer than we originally expected, we’re making good progress and things should be back to normal for everyone soon.”
Treynor explained that, even though Google keeps multiple copies of the data in multiple data centers, sometimes “software bugs can affect several copies of the data.” As a safety measure, the company backs up data on offline tapes, which makes restoring the data more time consuming than transferring requests between data centers, Treynor said.
When Google realized the software update was responsible for introducing the bug, it stopped the deployment and reverted to the old software version, Treynor said.
The issue came to Google’s attention yesterday when users started lighting up the company’s support forums with complaints of lost e-mails. “I have lost ALL on my emails/folders etc. from gmail. Why would this happen? How can I restore everything?” wrote user bkishan wrote in the forum.
Treynor also cautioned that e-mail sent to affect users between 6 p.m. PT yesterday and 2 p.m. today likely went undelivered but that the message sender should have received a notification that the e-mail was undelivered.