AI Won’t Kill A/B Testing, but Marketers Must Redefine What They Compare
Traditional A/B testing in marketing involves a straightforward comparison: taking two versions of content and evaluating which performs better through a series of predefined options. But as marketers incorporate artificial intelligence into marketing strategies, customer recommendations are more personalized and tailored to individual interests, behaviors, and needs. That leads to the question: How do you A/B test when every customer is receiving a different experience?
AI isn't going to eliminate A/B testing, but it will change what marketers compare. Instead of testing two fixed versions of content, marketers need to analyze the AI strategies that generate personalized customer experiences. With each strategy generating a wide range of messages, marketers must also perform quality checks on top of testing to understand which approach not only drives better business results but is also more accurate and brand consistent
Today, we are entering an era of AI-driven hyper-personalization, where everything from the headline and image to message and timing can be unique to each customer. In testing, the A and B become the prompting strategies, models, or configurations generating these experiences.
For example, personalized recommendations can come from different customer data. Marketers can test different sets of randomly assigned AI instructions where AI might recommend specific products based on a user's browsing activity, while another customer might receive suggestions based on past purchases. Although customers encounter different experiences, the strategies are still identifiable, allowing marketers to test which leads to more sales.
Going beyond product recommendations, businesses that use AI intake assistants to bring in new customers could also test different approaches, such as tracking customer priorities and asking follow-up questions. Each conversation leads to different results for each individual customer, and marketers can test the strategies shaping every interaction.
The focus expands from comparing two finished messages to evaluating a system capable of generating thousands of variations, and this is the shift marketers need to manage.
Avoiding AI slop with quality checks
Personalization at this scale requires an equally scalable way to assess quality. After identifying strategies, the next step isn’t to immediately test them with customers and compare business results. Marketers have to assess the quality of each message, ensuring they are consistent with the brand, compliant with policies, and accurate across product claims and descriptions.
With an abundance of AI slop infiltrating consumers' inboxes and social feeds, repeatable evaluations against defined standards, or evals, are a critical part of the testing toolbox. ;Customers are inundated with AI-generated content that has little to no relevance to their needs, wants, and preferences. Quality-checked messages that are personalized and accurate are what will stand out to a customer.
Before introducing customers to either strategy, marketers should generate variations across a representative set of scenarios and evaluate them. You could ask: Are the recommendations relevant? Are the specialties described accurately? Is the tone appropriate?
If the answer is no, revisit the prompts, adjust accordingly, and repeat the evals until every box is checked and the approach is ready for live testing. For example, an inaccurate output could recommend a product or service that is no longer in stock or offered, leading to customer frustrations and ultimately losing out on a sale or new business. The goal is to address issues before a customer encounters them.
Once evals are finalized, marketers can randomly assign customers to different approaches and measure outcomes, such as whether they go on to purchase a recommended product or remain active for a longer period. The process is continuous from there: generate, evaluate, improve, test and measure. It allows marketers to adapt and iterate quickly without making customers the first line of quality control.
LLM judges can review, but they need a clear rubric from humans
To make this process even more efficient, AI can also review content. This is particularly useful for more nuanced questions, such as whether a message is brand consistent, appropriate for individual situations or too high-pressure. The caveat: humans must always play a role in this process to ensure information relayed to the customer is always accurate and relevant.
One way to tap into AI's capabilities is through an LLM judge a language model that assesses generated content against a rubric and allows marketers to review far more variations than humans could inspect manually. However, the quality of the evaluation is shaped by the quality of the rubric a human develops. To receive a quality assessment, marketers must translate their judgment into clear criteria outlining acceptable and unacceptable responses.
For instance, asking an LLM judge if a message is on brand is vague and can generate an unhelpful evaluation. More specific questions provide the LLM with clearer criteria and are testable: Does this acknowledge the customer's concern? Is it putting too much pressure on the customer? Are there any unsupported promises? Does this provide a clear next step?
The other important aspect in using an LLM judge is ensuring there is a consistent process for addressing problems and ensuring those solutions work. Software engineers have embraced test-driven development, where it's defined how software should behave through tests before following through with writing the code. Marketers can adopt the same discipline to define what quality AI-generated content looks like, fine-tuning to meet standards, and confirming improvements aren't introducing new issues.
Investing in skills for better testing
Responsibility doesn't fall on marketing teams alone to redefine their testing strategies. Leaders are focused on upskilling their teams' content-generation capabilities, but they need to invest in building evaluation skills with the same urgency.
The rules of the marketing game are shifting. Success will increasingly depend on how well marketing teams can design, evaluate, and optimize systems that generate customer experiences and drive results. As it becomes easier to generate more variations, defining clear standards, measuring consistently, and improving systems are the skills that will drive competitive advantage.
Robert Ellison is executive vice president of enterprise at Pearl.