A/B Testing in UX Design: Lessons from Crystal Clear Memories

What button color to choose? What page layout works better? What typography should you use for your website? No one knows better than your customers do. And A/B testing is the way to reveal it. But when is A/B testing in UX design the right choice for your business? Keep reading to find out when it's worth trying and learn some lessons about the importance of A/B testing from the Crystal Clear Memories case.

And before we go any further, take a look at the designs and try to guess the winner of the A/B tests, and see if you're right at the end of the article. So, is it A, B, or C?

A/B Testing UX/UI: What Is It, and Who Needs It (or When to Skip It)?

Alright, the first things first. So, let’s break it down. Imagine you have to choose between two new ice cream flavors: Laguna Sweet or Mango Madness. If you have no clue about their taste and can only guess what it is like, would you just grab a pack or ask to sample them first? The latter seems to be more logical, doesn't it?

That’s exactly what A/B testing does for your digital content — except instead of flavors, you’re testing landing pages, emails, UI kit changes, and ads to find out what resonates best with your audience to increase your KPIs and ROI. 

To do A/B testing of website design, prepare two or more versions of a single element, then randomly divide your user group to see which version works better. This change could be something small, like a headline or button color, or a major page redesign.

There are several types of A/B testing:

1. Split testing: compare two completely different versions of a webpage to determine which performs better. Perfect for major redesigns.
2. Multivariate testing (A/B/n): Test multiple elements — such as headlines, CTAs, and images — simultaneously to understand how they interact and impact user behavior.
3. Sale funnel testing: evaluate each step of a user’s journey, such as the checkout process or sign-up flow, to identify drop-off points and optimize conversions.

A/B testing uses quantitative tools to monitor user engagement and determine if changes have a positive, negative, or neutral influence on behavior. For a long time, Google Optimize was the top free tool used for A/B testing, before retiring in 2023. Yet, there are enough alternative solutions, including the next popular options like:

• Optimizely: A powerful, enterprise-grade platform offering advanced experimentation capabilities;
• VWO (Visual Website Optimizer): A user-friendly tool great for businesses of all sizes, offering A/B testing, heatmaps, and visitor behavior analysis;
• GA4 (Google Analytics 4): A robust data tracking tool to monitor user interactions and measure A/B test performance.

The duration of an A/B test typically ranges from 1 to 4 weeks, mostly depending on factors such as website traffic and test complexity. Whether you're an e-commerce store, a SaaS platform, or a content-driven site, the advantages of A/B testing are that it allows businesses to continuously improve their digital presence based on real user data, not guesses.

Though they have high effectiveness, A/B tests are not for everyone. If your website has low traffic, achieving statistically significant results can be challenging. Additionally, if the changes you're considering have minimal impact on business goals, A/B testing may not be worth the effort. In such cases, focusing on qualitative research and analytics may be better.

UI/UX A/B Testing in Action: Crystal Clear Memories Success Story

Ready to find out the design winner? Then here we are to tell the story to the end. 

How All That Started?

So, let’s meet Crystal Clear Memories — a company that sells clients high-quality, custom-engraved crystal mementos with the LIVE PREVIEW™ patented feature. 

This is much more than just a feature. It’s a core of the business and a truly unique thing to know about the brand, as it allows customers to see the rendering of their 3D crystal before purchase. It is released as a so-called “wizard,” which is a crystal builder where users can customize the 3D crystal, upload their photo, and check the final result before ordering it. Sounds great, doesn’t it?

Yet, feeling the bigger potential, Crystal Clear Memories approached the Turum-burum agency with a request to increase conversion of the homepage, product pages (PDPs), and product list.

Usability Analysis and Benefits of A/B Testing: Which Of Three Designs Won?

As usual, the work started with a comprehensive UX audit and provided a detailed report with a clear roadmap for further improvements. Besides that, we created detailed mockups with 10 hypotheses for improving the homepage and product details page (PDP) based on analytics data and heatmap insights. 

The next step was to conduct A/B testing. We started with testing two variants of the homepage's first screen on both desktop and mobile. The A/B tests’ goal was to determine the best-performing design to maximize conversions.

The very first hypothesis on the homepage was based on the insight that a significant number of users (20% to 44%) were not progressing beyond the initial step of the product customization wizard. Besides, customers leave the homepage quickly because they didn't understand the unique value proposition and benefits of the company.

That’s why we suggested placing all parts of the wizard (crystal builder) on the first screen so as soon as users land on the page they see it, leaving a banner with the product sample in the background. It is supposed to engage users and increase the conversion rate (CR) to customized product pages by 20%.

The A/B test ran for 19 days, showing the positive changes, suggesting an improvement in overall usage behavior and e-commerce outcomes. Yet, what changed? First and foremost, the bounce rate decreased (-7%), which means fewer users left the site, and, what’s more, they stayed on the site longer (session duration increased by 10%) and viewed more pages per session (+9.7%). Put simply, users found the updated layout more intuitive and engaging. 

It seems to be a great result, yet the primary goal of directing users to the PDP showed insignificant changes. That was not the A/B fail, not at all. It was just indicating that though we chose the right direction, the wizard could be improved further, making the second experiment necessary to reevaluate the hypothesis in the second experiment. 

So, we A/B tested our second hypothesis, where we recommended not only telling but demonstrating “the magic” on the very first screen. This meant that the wizard was to showcase not only the ability of users to upload their photo but immediately to see the final product with it, trying different shapes of 3D crystal right there. This change was expected to increase the microconversion from the wizard to the custom product details page by 25%.

After 16 days of the A/B test runtime, we saw significant changes that made everyone happy. So, more users visited the product page (PDP views +14%) while staying on the site longer (session duration +11%). Another positive change was that fewer people left the homepage (bounce rate -7%). What does it mean? It suggested a higher level of user engagement with the new visual representation. 

So, according to the second and then final A/B test, key metrics and user behavior showed strong, positive changes, so we recommended upgrading the current homepage with the new wizard on the site. 

And what is a result? The happy end, actually. Running A/B tests proved the effectiveness of the hypotheses provided by the Turum-burum agency, so Crystal Clear Memories decided on implementing all the recommendations (homepage, product page, and product list) and saw a significant increase in the website KPIs. Hooray! Satisfied client = glad team.

‍Check it in action here

Final Thoughts: Yes, No, and Why A/B Testing is Important?

Improved user engagement, a lower bounce rate, and a higher conversion rate are the main benefits of A/B testing. No matter what your web project is dedicated to, A/B testing is a low-risk, high-reward strategy that allows businesses to continuously improve their digital presence based on real user data, not guesses. 77% of global companies regularly use A/B testing to improve their digital experiences, maybe it’s because successful A/B tests can lead to an increase of 50% in the average revenue per unique visitor for e-commerce sites. 

So, when launching a new design or feature or validating UX/UI hypotheses before implementing the changes, A/B testing is an effective method to make a decision. Because only your real customers do know the best way your website looks. 

The results of Crystal Clear Memories case show how much design and layout can affect the number of users left the site and a user’s decision to place the order. If you're still hesitating if you need it or not, ask the experienced UX/UI/CRO agency to guide you on the suitable approach for your project.

Oh, by the way, did you bet on the second (B) variant? If yes, then our congrats, as you were just right!