A small human error caused Google to mark every site in its search results as “dangerous” earlier today. Even worse, according to some reports users were unable to bypass the warning and continue to the website.
Fortunately, the issue was resolved remarkably fast. Google even posted an explanation of what happened on their blog just hours after the incident. A quick summary:
- Google maintains a list of known malicious sites
- Somebody entered the URL “/” on that list
- The list was pushed to all of Google’s servers between 6:27 and 6:40 a.m.
- Because / is found in any URL, all websites were flagged as potentially dangerous
There are some other details in the post, revealing some interesting information about their infrastructure: they are able to push an update to every server in their global network within just 15 minutes.
Note that this was indeed a human error from one of Google’s employees; an earlier version of their blog indicated the error was in the list supplied by StopBadware.org. The current text no longer implies this. StopBadware.org has their own explanation (Coral link; the main site is down at the moment).
This is an excellent example proving that, no matter how good your procedures and infrastructure are, there will always be human errors. Does anybody remember the day Network Solutions accidentally deleted domain registrations? Or when Dreamhost deleted 100.000 domains from their DNS servers? Or just yesterday, when an admin at Humyo deleted a multi-terabyte database? The good news is that this was noticed and fixed very fast. Now if only they would be just as responsive when dealing with smaller customers…