How can a few stickers make an AI read a stop sign as a speed limit?

A few stickers can fool an AI because it does not read a sign the way you do. Researchers have altered stop signs so that a self-driving car's vision system read them as a merge or speed-limit sign. In one study, black and white stickers on a real stop sign caused the intended misreading in every lab image tested. The stickers exploit what the network actually learned: patterns of pixels, not the idea of a stop sign.

This is adversarial machine learning, the study of attacks on AI models and of defenses against them. From 2014, Christian Szegedy and others showed that deep neural networks could be fooled by small, calculated changes to an image. One early method, from Ian Goodfellow and colleagues at Google, adds noise too faint to notice, and the model gets the picture wrong. Others fooled networks by changing a single pixel, or 3D-printed a toy turtle that Google's AI called a rifle from every angle.

Simple tricks can work too. The security firm McAfee fooled Tesla's former Mobileye system into driving 50 mph over the speed limit. All it took was a two-inch strip of black tape on a speed-limit sign.

There is an old precedent. In the early 1900s, a horse called Clever Hans seemed to do arithmetic, tapping out answers with his hoof. In 1907, psychologist Oskar Pfungst showed that Hans was really reading tiny, involuntary cues from his trainer. AI researchers now speak of a Clever Hans effect: a model that gets right answers for the wrong reasons, like reading text in a photo instead of looking at the object.

Not everyone sees a big danger on real roads. A small rotation or a change in lighting can destroy a carefully crafted attack. Google Brain's Nick Frosst also notes that tearing a stop sign down is far easier than faking one. Still, Google, Microsoft and IBM now share tools to test how robust their models are.