MacID: Bringing Face ID to the Mac
I wanted to bring Face ID to the Mac. The resulting product, that I call MacID, works as seamlessly as your OS will allow and does the best possible job it can under the limitations it faces (pun somewhat intended).
MacID doesn't automatically open up your desktop. It needs you to get to the password field and press the return key. Then, if MacID believes that the face it sees matches what it has stored, you're logged into your computer. If not, then the computer treats it the same way as a wrong password, which on macOS is the password bar wiggling side-to-side rather aggressively.
Of course, you can always enter your password normally and still log in; MacID does nothing to interfere with that. Similar to Touch ID, MacID also needs your password after a restart or hibernation before it can start working again.
You can download the desktop app here and read its code here. I'll try and walk you through the core technical details in this note.
Authentication On macOS
At the core of macOS authentication is PAM, pluggable authentication modules. PAM is what Unix systems use to answer “is this person who they say they are?” without every single program having to implement their own answer. sudo, login, ssh and even the macOS lock screen all delegate that question to PAM rather than checking a password themselves.
Each application that consults PAM for authentication is called a consumer. So in my previous example, sudo, login, ssh and the macOS lock screen are all consumers of PAM. Every consumer gets something called a stack, which is a text file located at /etc/pam.d/. Each line in this file contains 4 parts: a type, a control flag, a module path, and arguments for that module.
For example, take this line:
The key components you'll need to understand here are the module path and the control flag. A module is simply a piece of software that PAM calls on to do a particular job. Here pam_krb5.so corresponds to Kerberos, a network authentication protocol.
A control flag can take 3 basic values: required which means the module must send a success response back to PAM; optional which means the module's result doesn't affect the overall result; and sufficient which is the useful one for us here: if that module says yes, the stack short-circuits and nothing below it runs.
One last thing to note about PAM is that the files do act like a true stack wherein the execution order is determined by the line order in the file. So:
auth required pam_A.so auth optional pam_B.so auth required pam_C.so
means that PAM processes them as:
pam_A → pam_B → pam_C
So, in principle, adding a face unlock is as simple as adding a sufficient module ahead of the password check. One difficulty here is that the macOS lock screen's use of PAM is not documented or known to be stable across releases, and nothing tells you when it changes.
Finding The Hook
When looking through files in /etc/pam.d/ I found a couple different stack files of interest including screensaver, screensaver_new and screensaver_la. In order to find the correct file where MacID belongs, I wrote a diagnostic PAM module, that does nothing but log every call it receives, wired it into a bunch of files as auth optional (which always returns PAM_IGNORE and therefore cannot deny anything) to watch what happens.
| stack | used by the lock screen |
|---|---|
screensaver | no |
screensaver_new | yes, with the typed password already collected |
screensaver_la | no (the LocalAuthentication path: Touch ID, Watch) |
sudo | no, this is sudo's own stack, and it runs before the password is typed |
From this, I learnt that the lock screen uses screensaver_new and it calls the stack after the password has already been collected. Most importantly, this stack is called even when you type nothing at all and just press return.
The ability to send in an empty submission is important to MacID. There is no supported way for a PAM module to say “give me a moment, I am looking at a camera”, and no way to draw UI at the lock screen. But pressing return on an empty password field is a perfectly ordinary authentication attempt, and it costs the user exactly one keystroke.
This is therefore the most convenient way to use MacID, by simply pressing return on the password bar. Of course, if you enter the wrong password, and then press return but MacID recognizes your face you will still be logged in. This is the point of having MacID run as sufficient in the stack before your password checker is triggered by PAM. MacID doesn't care what's in the password bar because it never reads it.
So, we've now added the MacID PAM to /etc/pam.d/screensaver_new and it looks like this, a filepath to a sufficient module that triggers before anything else on the system:
+ auth sufficient /opt/macid/lib/pam_macid.soauth optional pam_krb5.so use_first_pass use_kcminit no_auth_ccacheauth optional pam_ntlm.so use_first_passauth required pam_opendirectory.so use_first_pass nullokaccount required pam_opendirectory.so
App Structure
MacID's PAM module, pam_macid.so runs as root inside loginwindow, which is the system process responsible for the graphical login interface. Opening the camera as root inside loginwindow is a terrible idea because camera perms belong to the user and not root, so macOS would refuse the PAM module access to it.
This is why the PAM module itself does no recognition and simply serves as a bridge between macOS and the MacID desktop app, which starts at login (this can be configured), lives in the menu bar and actually contains the recognition service. This is possible because apps are allowed to hold camera perms, which you grant during app setup.
Apart from the desktop app, MacID includes a small helper with higher privileges that needs admin perms to modify the system settings required for lock-screen authentication. This helper exists because of a privilege problem since editing the lock screen config does two things:
- Write our module into
/etc/pam.d/ - Install our PAM module at
/opt/macid/lib/pam_macid.so
Both of these need root perms, but our desktop app itself must never run as root, because a root GUI app is a bad idea for a lot of reasons and the camera permission belongs to your user account. So the root half is a separate binary that only handles writing and deleting these files. You, the user, must still authorize editing and writing to these files through the standard macOS admin password prompt (i.e. MacID would like to do xyz, enter password). After both of these actions are done, MacID never needs root perms.
This system worked great for me until I ran into a problem. When you typically unlock your Mac, macOS also unlocks your Keychain, which holds your saved passwords, tokens and signing keys, using whatever was typed into the password field. These keys and tokens handle things like email signing and git commit signing, and the Keychain is also where apps store sensitive information. For example, Chrome stores a key in the Keychain that it uses to decrypt the saved passwords and cookies in your browser. But currently in MacID, that field is empty after a face unlock, so the Keychain stays locked and everything that reads from it starts failing or prompting. In my case, I was prompted every time I tried to git commit because my GPG key was stored in the Keychain.
Touch ID gets around this with a key the Secure Enclave keeps for the Keychain; however, only Apple can use that key, which is a security feature of the Mac. So MacID asks for your password once, checks that it’s right, and encrypts it with a key that lives inside this Mac's Secure Enclave and never leaves it. After a face unlock, MacID uses it to unlock your Keychain, and for nothing else. If you change your password, MacID notices the saved one no longer works, deletes it, and goes back to asking for your password until you give it the new one.
Recognition
Recognition itself uses w600k_mbf, a pretrained face-recognition model from InsightFace that turns each face into a 512-dimensional embedding. I convert it from the .onnx format into an Apple Core ML model so that it can run locally on your Mac.
Measured through our app's pipeline over 767 image pairs from a famous dataset called LFW it separates the same person from different people with 99.5% accuracy and a 0.0% false acceptance rate. The model is ~3.4M parameters and is ~6.6 MB on your disk. The whole .dmg is 7.5 MB so the model itself is ~88% of what you download.
Since the model handles recognition, my main focus is alignment & normalization. For every image, Apple's Vision API produces 5 landmark regions: two eyes, two corners of the mouth and the nose tip. Our model expects them at fixed positions in a 112x112 crop as input, so each face needs to be warped onto that layout with something called a similarity transform before it is passed through the model. Without this, the embedding is using more of its capacity measuring how your head is tilted rather than whose head it is.
Aside: The Similarity Transform
A similarity transform is the family of 2D transforms that can move (translation: ), rotate (angle: ) and uniformly resize a shape (scale: ) without interfering with anything else. This gives us the 4 unknowns , , and :
is simply a 2-dimensional point in the context of a 1280x720 image (your webcam's resolution), such as (300, 250) for your left eye, which is one of the 5 landmarks. So is where a landmark is in the camera frame, and is where that same landmark should end up in the 112x112 crop that we give to the face recognition model.
Similarity transforms are helpful for this task because they're quite limited. A more flexible transform (e.g. an affine one) could also stretch the face more in one direction than another, or slant it. This could squash a wide face narrower until its landmarks lined up, which erases the proportions that the model is trying to recognize. A similarity transform can only move, turn and resize, so the output image has the same proportions as the input.
Call the five landmarks the camera found (each one an from above), and the five spots the model wants them at (each one an ). Putting one landmark through equation (1) gives two equations, one for and one for , so five landmarks give ten equations but there are only four unknowns. Real landmarks are never placed perfectly, so no single choice of and satisfies all ten.
Instead we take the least-squares answer, which can be written in terms of and instead of and , making all ten equations linear and the answer something we can calculate directly. First, measure each point from the centre of its own set (i.e. and ), and . Then:
The dot products () measure how much the two sets of points line up, and the cross products () measure how far one is turned relative to the other. Those two numbers fill in the whole matrix, so and never need to be worked out separately. The translation then comes from the two centres: , where is the matrix.
This is one of many reasons why face recognition systems will capture your face from several angles because the lock screen will not see you head-on, and it has to work out how much of what it's looking at is noise.
A naive way to calculate noise is to compare the largest distance between two image samples, but this doesn't account for pose since the difference between your head leaning left and your head leaning right is signal about coverage rather than noise. So samples are grouped by the pose they were held at, so that only two samples of the same pose, which differ only by light or micro-expressions, count as noise.
I capture multiple poses during setup, using yaw and pitch to check that each pose has actually been reached before using those frames.
To decide whether a face is yours, MacID measures the distance between the embedding it sees and the one it has stored. The smaller that distance, the more alike the faces, and anything closer than a threshold cutoff counts as you. You choose this threshold value, wherein setting the threshold lower means that it recognizes you less often but is less likely to mistake someone else for you, and vice versa.
Using the same LFW dataset as earlier, I calculate how likely it is to recognize you and to mistake someone else for you, in order to help you make that decision. The blue shaded region in the slider below indicates the range that MacID lets you select from.
In Summary
MacID is as close as I could get to a seamless Face ID experience on the Mac. It does come with some caveats as we've seen—such as still having to interact with the password bar—that can be mostly resolved with some clever engineering, but the one last issue that I can't resolve is physical. The reason Macs don't already have Face ID is that they lack the sensors necessary for it, such as the depth sensor, which can look at a real three-dimensional reconstruction of your face instead of the flat image that your webcam sees. So a good enough photo of you could unlock your Mac.
To account for this, MacID comes with a “require a blink” setting that you can turn on or off, which needs you to blink before it does an unlock. Again, a video might be able to get through, but this is the best possible solution I could think of. Let me know if you have a better idea! Finally, I think it's important to treat using MacID as more of a convenience rather than a replacement for your password.
Thanks so much for reading this note and feel free to reach out if you have any questions!