Lessons from Amazon Web Services' big outage

Last Monday, millions of internet users got a painful answer to a question few even knew to exist. The question was: What do Snapchat, Roblox, Fortnite, Signal, United and Delta airlines and countless other web-based sites and services have in common?

- MICHAEL HILTZIK COLUMNIST

The answer is: They were all brought down by a cascading glitch at a data center in northern Virginia owned and operated by Amazon Web Services, an arm of the giant e-commerce company.

AWS is one of the top three cloud platforms, meaning that it holds its clients' data on its own servers and manages the transfer and transmission of those data within the client companies and between them and end users.

When AWS' northern Virginia data hub went down a few minutes before midnight Sunday Pacific Daylight Time, 141 AWS services went dark, along with client firms reliant on its hub, producing a cascade of outages affecting users around the world. Users of Amazon's own Ring home security devices such as video-enabled doorbells were affected.

Amazon didn't declare that the problem had been fixed until 3:53 p.m. PDT Monday, although some clients were still reporting problems as late as Tuesday.

The damage done to AWS clients and their millions of users is incalculable. As my colleague Queenie Wong reported, web users couldn't access their services or accounts. Customers of some banks, as well as the web brokerage Robinhood, couldn't complete transactions. Delta and United passengers were unable to track reservations, check in online or retrieve their seat assignments; airline employees were forced to resort to manual alternatives, like in prehistoric (i.e., pre-internet) times.

Owners of Eight Sleep mattress covers, which cost thousands of dollars and require an annual fee of $300 or $400, use a web app to adjust temperature and incline, reported being stuck in uncomfortable positions and sweltering under uncontrollable heat. The company's chief executive issued an online apology and said Eight Sleep would roll out a feature allowing owners to connect with their beds via bluetooth if the internet connection failed.