How virtual makeup try-on actually works, and why shade accuracy is the hard part

Virtual makeup try-on looks like magic on a product page. You tap a lipstick swatch, the camera opens, and your lips change colour while you move your head. The demo is short and convincing, which is why it sells well internally. The engineering underneath it is not complicated to describe, and knowing how it works tells you exactly where it will let you down.

A note on bias before anything else. We build 3D models, viewers, configurators, web AR and eyewear virtual try-on. We do not build face try-on for makeup. So this is not a pitch for a product we sell. It is an explanation of a category next to ours, written by people who work with the same camera and rendering problems every week.

Step one: finding the face

Everything starts with a face mesh. The phone or browser runs a model that takes each camera frame and returns a set of landmark points on the face, typically a few hundred of them, describing the outline of the lips, the eye openings, the brow line, the cheeks and the jaw. Those points are connected into a low-resolution 3D mesh that sits over the real face and moves with it.

This part is genuinely solved. Face landmark tracking runs at full frame rate on a mid-range phone, survives head rotation and most expressions, and works in the browser without an app. If a try-on feels laggy or the lipstick slides off when you smile, that is an implementation problem rather than a limit of the technology.

Step two: putting colour on the mesh

Once you have the mesh, the product has to be drawn onto it. For lipstick that means masking the lip region and blending a colour into the pixels there. For eyeshadow it means a soft gradient over the lid, following the eye shape. For foundation it means the entire face region, which is the hardest case, because the software has to change skin tone without erasing the skin.

The blend is not a flat fill. A good implementation preserves the luminance of the shopper's own skin and lips, so highlights, the shadow at the corner of the mouth and fine texture stay visible through the colour. When a try-on looks like a sticker, that step has been skipped: a solid colour laid over the region rather than combined with it.

Finish is the second half of the same problem. A matte lipstick, a satin and a high-gloss differ mostly in how they reflect light, so the renderer has to add specular highlights that move correctly as the head turns. Glitter and shimmer need a separate sparkle layer. Get this wrong and every shade in the range looks like the same product in a different colour, which defeats the purpose.

Step three: lighting, which is where it breaks

Here is the part no vendor demo shows you. The camera does not tell the software what light the shopper is standing in. A warm ceiling bulb, a window on an overcast day and office fluorescents produce three different sets of pixels for the same skin, and the phone's own auto white balance and auto exposure are already adjusting all of it before the try-on code sees a frame.

So the renderer has to guess. It estimates the colour temperature and brightness of the scene from the image, then adjusts the product colour to match. When the guess is close, the result is convincing. When it is not, the shade on screen is warmer or cooler than the shade in the tube, and the shopper buys a colour that does not exist.

Why shade accuracy is the hard problem

Everything above compounds into one honest limitation. A makeup try-on has to survive four separate sources of colour error before it can claim to show a true shade:

The last row is the one brands can fix and often do not. If the digital shade came from eyedropping a marketing image rather than from a measured colour value of the physical product, the try-on is inaccurate before the camera is even involved.

This is also why undertone matters more than hue. A shopper is not usually deciding between red and pink. They are deciding whether a particular nude flatters their skin, which is a question about the interaction between two colours, and errors of a few percent in either one change the answer.

What try-on is genuinely good at

None of this makes the technology a waste. It is very good at three things.

It narrows a long range fast. Forty lipsticks is a grid a shopper will abandon. Try-on turns that into a shortlist of three, which is a real improvement in discovery even if the final colour is slightly off.

It answers shape and placement questions accurately. Where a liner sits, how wide a brow reads, whether a blush placement suits a face: these are geometry, and geometry is the part the face mesh handles well.

It gives people a reason to stay. Shopify's merchant data from September 2020 found that interactions with products carrying 3D or AR content converted 94 percent more often than comparable products without. That figure is across categories rather than beauty, so treat it as direction, not a forecast.

The category where the same camera trick is fully solved

Eyewear is the useful contrast, and it is work we do build. A frame is a rigid object with fixed dimensions. The same face mesh that struggles to guarantee a lipstick shade is more than accurate enough to place a frame on the bridge of a nose at the right width and scale. There is no lighting estimation problem to solve because the shopper is judging shape and proportion, not a colour match against their own skin.

The evidence supports the intuition. Fittingbox's 2025 report, cited in Shopify's write-up on AR shopping, found that 54 percent of customers who were not ready to buy glasses online wanted to check that the frames fit properly, and 29 percent of glasses shoppers had already used virtual try-on at least once. That is a defined objection that try-on removes completely.

If you sell beauty, what to take from this

Ask any try-on vendor two questions. Where does your shade data come from, and what do you do about ambient lighting. If the answer to the first is anything other than measured colour values from the physical product, the accuracy conversation is over. If the second answer is vague, expect the try-on to perform well in a bright neutral room and poorly everywhere else.

Then set expectations in your own copy. A line on the page saying that on-screen colour is a guide and can vary with lighting costs you nothing and prevents a specific kind of angry return.

And separate the parts of your catalogue by what the shopper is actually asking. Colour questions go to try-on with caveats. Shape, size and scale questions, which is most of your packaging, tools, devices and any accessory you sell, go to a 3D model, where the answer is exact.

That last part is what we make. Send us one product and we will build the 3D model and put it in a live viewer for free, so you can see the difference on your own page rather than in someone's demo. It is yours to keep either way: free sample. If you want to see the range first, including our eyewear try-on work, the solutions page covers it, and prices are public.

Sources

• Shopify, AR shopping, including Fittingbox 2025 eyewear figures and Rebecca Minkoff results.
• Shopify, merchant data on 3D and AR content, September 2020.