The very human problem with not-quite-self-driving cars

The better an automated system performs, the more complacent and dangerous – we become.

- Sarah O'Connor

As I climbed under the kitchen table with my five-year-old this weekend, she explained that we were in a car, but, "it can drive itself, so we can just relax, OK?" We settled down for a pretend nap on the way to the pretend beach.

I didn't tell her that grown-ups are really struggling to turn this vision into reality. Even Waymo, the company that is furthest ahead, still only has self-driving taxis in a handful of US cities.

In the meantime, carmakers are packing many of their new models with so-called "Level 2" partial automation features instead, which can do a certain amount of driving in some circumstances, but require the human driver to pay attention and take over when necessary. Yet this halfway-house, which relies on humans and machines, is proving troublesome. And it is trouble worth noting, even if you have no interest in cars, because other sectors are also beginning to embrace the concept of automated "co-pilots" to help everyone from coders to doctors.

The big problem is known as "automation complacency". People have been studying the phenomenon for decades in all kinds of partially automated systems, from aviation to manufacturing processes.