How to A/B test a 3D product page properly

Most 3D rollouts are never tested. The models go live, the numbers move for a dozen reasons, and someone either claims credit on a slide or drops the subject. If you are paying for 3D it is worth knowing whether it works on your site, with your products, at your traffic level.

Decide what you are testing before you touch anything

Three different questions get run under the name of an A/B test.

1. Does a 3D viewer on the page change conversion? Control gets the current page, the variant gets the same page with a 3D viewer. This is the business question.
2. Does interacting with the viewer predict purchase? This is a correlation rather than a test, and it always looks good, because people who engage deeply with a product page were close to buying already.
3. Which 3D placement works better? Above the fold or in a tab, viewer first or image first. Worth running once the first question has a yes.

Almost every impressive 3D statistic you have read is the second kind. Shopify's widely quoted finding that products with 3D or AR content converted 94 percent more often compares product interactions across merchant data rather than coming from a controlled trial, and merchants model their best products first. The Rebecca Minkoff figures, 44 percent more likely to add to cart after viewing a 3D model and 27 percent more likely to order, are also conditioned on interaction. They are useful figures, and your own test will produce something smaller, because it counts everyone who saw the page and not only the people who used the viewer.

Pick the right primary metric

Choose one primary metric. Everything else is secondary and cannot be used to declare a win.

For most furniture and home brands the primary metric is conversion rate on the tested product pages, meaning orders containing a tested product divided by sessions that viewed a tested product page. Revenue per session is a reasonable alternative if basket sizes vary a lot, but it is noisier and needs more traffic.

Metrics that should not be the primary one, and why:

• Time on page. A 3D viewer increases it by itself and says nothing about sales.
• Bounce rate. The same problem in the other direction.
• Add to cart on its own. Fine as a secondary. A model that raises carts without raising orders has moved the problem one step along.
• Anything conditioned on interacting with the viewer. Only the engaged shoppers can be counted in it.

Useful secondaries: add to cart rate, AR open rate on mobile, viewer engagement rate, and return rate on tested products over a longer window.

Work out whether you have enough traffic

Most 3D tests fail here before they start, and it comes down to arithmetic.

A rough rule for a two-sided test at conventional confidence and power: to detect a relative lift of about 10 percent on a 2 percent baseline conversion rate you need tens of thousands of sessions per variant. To detect a 5 percent relative lift you need roughly four times that, because the required sample grows with the inverse square of the effect size. Halve the effect you want to detect and the traffic you need goes up fourfold.

Run the numbers in a sample size calculator with your own baseline conversion rate before you commit. If the answer is six months, there are three options.

• Widen the test. Put more products in both arms so each arm collects sessions faster.
• Accept a bigger detectable effect. Decide up front that you are testing for a large lift, and read a null result as "not large" rather than "no effect".
• Skip the conversion test. Measure engagement and returns instead, and decide on cost and customer feedback. For a small shop that is a reasonable choice.

The one thing not to do is run an underpowered test, see a positive swing and stop there.

Set the duration properly

Two rules apply, and the longer one wins.

At least two full weeks, ideally four. Weekday and weekend shoppers behave differently, and a test that runs Tuesday to Friday measures Tuesday-to-Friday shoppers.

At least one full purchase cycle. Furniture is a considered purchase. People visit, leave, talk it over at home and come back a week later. If the typical time from first visit to order is nine days, a ten-day test credits purchases to a period that did not include the decision.

Fix the end date in advance. Checking daily and stopping when the line looks good is the most common way to produce a false win.

The traps specific to 3D tests

Page speed. If the variant loads a heavy model badly, you are testing page weight. Lazy-load the model, keep it near the roughly 4 megabytes Shopify suggests as a target, and check that the variant's largest contentful paint is not clearly worse than the control's before you start.

Modelling only the good products. Build 3D for your ten best sellers, compare them with the rest of the catalogue, and what you have measured is which products sell best. Randomise at visitor level on the same set of products.

Uneven model quality. One badly proportioned model in the wrong fabric colour drags the whole variant down, and the conclusion becomes that 3D does not work. Check every model against the physical product before launch.

Seasonality and promotions. A campaign that runs mid-test hits both arms if you randomised by visitor. It does not if you rolled 3D out to one country or one device type.

No testing tool. Google Optimize and Optimize 360 stopped being available on 30 September 2023, and Google pointed people to third-party providers. If your last test ran in 2022, check what tool you have before planning anything.

Measure returns separately and slowly

Returns are the other half of the business case and they run on a different clock. You need the full return window plus processing time, so a quarter at least. Track the return rate on tested products and, if you have reason codes, the share coded as size or expectation, because that is the part a correctly scaled 3D model can affect. The published benchmark is modest. Shopify reported Gunner Kennels reducing return rates by 5 percent after adding AR.

What a result lets you do

A clean positive on ten products tells you that 3D works for products like those ten. It says nothing about your accessories, small items or commodity lines. Roll out in the direction the test pointed and test again at the edges.

Our clients have mostly ended up in the same place, by category. Sweef, Contura and MELIMELI use 3D on the large, variation-heavy products where a photograph leaves the main question open.

If you need assets to test with, we build a 3D model of one of your products for free. Send us one product and the model is yours to keep, which is enough for a pilot page. Pricing for a full rollout is on prices, from 10 dollars per product per month on Basic with a 12-month term and a 100-variant minimum. What we build is on solutions.

Sources

• Shopify, 2020 merchant data on 3D and AR, AR shopping and 3D e-commerce.
• Google, Optimize and Optimize 360 sunset.