How Face Recognition Works: The Tech Behind the Scan

DEV Community

How Face Recognition Works

Developer note: Series 3, Episode 3 deep dive for Paragon AI MAG. How does face recognition on your phone work? The mechanism: structured infrared light — a flood illuminator, a dot projector casting 30,000+ points, an infrared camera reading their displacement — builds a depth map that a neural engine turns into an embedding, compared inside the Secure Enclave against the enrolled template, with a 1-in-1,000,000 false-match threshold. We also cover the history (Bledsoe 1964, Eigenfaces 1991, DeepFace 97.35% in 2014, FaceNet's 128-number embedding at 99.63%), the 1:1 verification vs 1:N identification split, the NISTIR 8280 demographic gaps, the Kneron spoof tests, and the practical side: five settings to change tonight and three self-tests. Canonical URL: https://paragonaimag.substack.com/p/how-face-recognition-works-series Watch the video: https://youtu.be/PhatIybN_A4 Every time you unlock your phone, it fires more than 30,000 invisible dots at your face. It does not take a photo of you. It builds a 3D map from those dots and turns it into numbers. That is the secret: literal maths, not a picture. And yet the same core technology once locked an innocent man in jail for 30 hours. This article explains both sides — how the phone version works so well, and why the same idea fails so badly when it is pointed at a crowd. THE MAN THE COMPUTER PICKED On 9 January 2020, police arrived at Robert Williams' home in Detroit and arrested him in front of his wife and his two little daughters, aged two and five. They held him for about 30 hours in a dirty, overcrowded cell. A police computer had matched his driver's licence photo to a blurry surveillance still of a man who had stolen watches from a Shinola shop in 2018. The computer was wrong. Williams was nowhere near the shop, and he barely looked like the suspect. It was the first publicly reported wrongful arrest caused by a face-recognition match in the United States. In June 2024, the city of Detroit settled his case for $300,000 and banned arrests based on a face match alone. So here is the question that carries this whole story: if this technology fails that badly out in the real world, why do we trust it to protect our bank accounts, our private emails, and our most sensitive data? The answer is that the phone and the police computer are doing two different jobs. THE SHORT ANSWER Your phone does not take a photo of you. It projects more than 30,000 invisible infrared dots across your face to build a three-dimensional depth map, then turns that map into a list of numbers — an embedding. The phone compares numbers against numbers, never pictures against pictures. Those numbers live in the Secure Enclave, a separate small computer inside the phone's chip with its own key. The operating system cannot read them, apps cannot read them, and they are never sent to Apple or backed up to iCloud. When you look at the screen, the phone asks the vault one question — does this new maths match the saved maths? — and the vault answers yes or no. THE SEVEN STEPS When an Apple device wakes up and checks your face, it runs seven steps: 1. Flood illuminator. An infrared light bathes your face so the phone can find it — even in total darkness. 2. Dot projector. One laser shines through a special plate and splits into more than 30,000 invisible dots across your face. 3. Infrared camera. The only sensor that can see those dots reads how they bend around the shape of your nose, cheekbones and eye sockets. 4. Depth map. The projector sits at one end of the notch and the camera at the other, so each dot's sideways shift gives its distance. Thirty thousand distances make a 3D face. 5. Neural Engine. A dedicated chip turns the depth map and the infrared photo into a list of numbers. 6. Secure Enclave. The fresh numbers are compared with the enrolled ones inside the vault chip. Only yes or no ever leaves. 7. Threshold. A yes needs the numbers close enough that a random stranger has less than a 1 in 1,000,...

Powered by Readybot.io