AR beauty try-on: how it works and who builds it

Point a phone at your face, pick a lipstick, and the shade appears on your own lips. That is AR try-on as the beauty trade uses it. If you have been asked to look into it for a cosmetics range, this is what we know about how it works, where it falls down, who sells it and what a demo should be made to prove.

How makeup try-on works

Two pieces of technology sit behind it. Face tracking comes first. The phone or the browser finds the face in the camera image and works out where the lips, eyelids, cheeks and brows are, several dozen times a second. Phones do this reliably now.

Then the product has to be drawn on top, and this is the hard part. Lipstick has opacity, gloss and a little translucency where the natural lip colour shows through, and it sits differently on a dry lip than on a full one. Foundation is harder again. The whole job is matching a tone to skin, and the camera is already misreading that skin under whatever light the shopper happens to be standing in.

Accuracy therefore varies a lot by product type. Shopify's guide to AR shopping gives difficulty scores by category, with facial complexion products the hardest at 41 percent, followed by self-tanning and body makeup at 38 percent each. The nearer a product gets to reproducing skin itself, the worse the result.

What it does well

Colour choice within a known family. Lipstick, eyeshadow, blush, hair colour, nail polish. The question a shopper has with a red lipstick is usually whether it looks too orange on them, and a try-on answers it in a second.

It also gets used. L'Oréal, which owns ModiFace and runs try-on across many of its brands, reported over 100 million digital try-on sessions in 2023 against 40 million in 2022. That counts sessions and says nothing about sales, but it does show that shoppers pick the feature up on their own.

On sales, the best-known figure comes from L'Oréal's then chief digital officer Lubomira Rochet, who told The Drum that adding AR to a site doubled engagement time and tripled the conversion rate. It is a brand-side statement with no published method, and it compares shoppers who used the feature with those who did not, which always flatters the feature. A business case should not rest on it by itself.

Where it still fails

Foundation and concealer shade matching. The camera white-balances to the room, the screen has its own colour profile, and the shopper is standing under a kitchen bulb. Vendors work around this with reference cards, prompts to find better light and shade data measured in a laboratory. The best of them get close, but the underlying problem has not gone away. If wrong-shade foundation drives your returns, a sample programme will do more than a try-on.

Texture is the other gap. A try-on cannot show whether a mascara clumps, whether a cream feels heavy or whether a shade shifts three hours into wear. Reviews still have to cover that.

Who provides it

Most brands buy from one of a handful of specialists.

Setup is where the money goes. A 400-shade catalogue has to be measured and tuned shade by shade, and that effort grows with the catalogue whatever the app costs per month. Also check whether the try-on runs in the mobile browser or only inside an app or a social platform. A shopper who has to install something first will often give up.

We are not on that list. CharpstAR builds 3D models and web AR for products with a fixed shape, which in a beauty business means the packaging, tools and fixtures covered further down. Face tracking is a separate field with its own vendors.

How to judge a demo

A vendor demo runs on a flattering face in good light, so bring your own tests.

• Try it on at least four skin tones, including the darkest and lightest you sell to. Uneven results across tones are common.
• Test in bad light. Evening, a yellow bulb, a window behind you.
• Photograph the try-on next to the real product on the same face.
• Check load time on a mid-range Android phone over mobile data rather than on the demo iPhone.
• Ask what happens on a desktop browser with no webcam, because a fair share of your traffic is there.

What transfers to the rest of the catalogue

Beauty brands also sell packaging, tools, devices, salon furniture and retail fixtures, and anything with a fixed shape behaves like any other physical product. It is modelled once in 3D, shown from every angle on the page and placed at true scale in a room through the phone's own AR. That is the work we do, and the pipeline is different from face tracking. One model is built and hosted, then dropped into a viewer and reused for renders and AR.

The cost is a fraction of a try-on programme. Our plans run at 10 dollars per product per month on the basic tier and 20 dollars on the tier that adds dimensions, annotations and a colour configurator, and the prices page has the detail. MELIMELI uses that configurator to let shoppers switch fabric on a live sofa model, which is the same idea as a shade switcher applied to an object.

If the fixed-shape part of your range is still sold from photographs, send us one product and we will model it and put it in a viewer for free, yours to keep whether or not you go further: free sample. The solutions page describes what we build.

Sources

• Shopify, AR shopping: category difficulty scores, Rebecca Minkoff and Gunner Kennels results, try-on app pricing.
• Shopify, virtual fitting rooms: providers and pricing.
• PYMNTS, L'Oréal sees 150 percent increase in virtual try-ons.
• The Drum, Conversion rates triple when L'Oréal uses AR tech to showcase products.