E-Commerce Customer Retention Reports Are Grading the Wrong Customers
Latane Conant argued recently that AI can make the purchase but companies still need to earn the loyalty and that a transaction no longer proves loyalty on its own. I agree, and I would add a point from the e-commerce side. Most online retailers cannot currently tell whether they earned any loyalty at all, because their retention reports give credit to customers who were coming back regardless.
The sequence usually runs like this: A loyalty program or a win-back email series launches. Repeat purchase rate ticks up. The dashboard credits the program, its budget grows, and nobody asks which of those returning customers actually changed their behavior.
Most email and SMS platforms count customers as influenced if they clicked messages, opened email, or in some setups simply had texts delivered within a set window before ordering. Picture someone who has ordered coffee beans every five weeks for two years. In week five he opens a replenishment email, closes it without clicking, and orders at the usual time two days later. The platform books that order as retention revenue.
Discounts make it worse, since many coupons go to customers who would have bought without them. A habitual buyer who receives a 15 percent win-back code will usually use it, because nobody refuses money. The order was going to happen at full price. The report shows a recovered customer while finance quietly absorbs a margin cut on revenue the business already had.
Points programs follow the same curve. Points pile up fastest for customers who buy most often, so redemption concentrates among the people least in need of a reason to return. Rewarding them is a reasonable courtesy, but it belongs in the cost of serving good customers rather than in the retention column.
Why it starves acquisition
A vanity metric would be harmless if it stayed on a dashboard. This one shapes budgets. Channel reviews compare return on spend, and a retention program that claims revenue from habitual buyers will usually look cheaper than paying to find a stranger. So money drifts from acquisition toward whichever program has the most flattering report.
The damage arrives slowly. A store's best repeat customers next year are this year's first-time buyers, and cutting the spend that brings them in thins the repeat base a year or two later. Retention numbers then fall, and the usual response is to put even more into retention. That loop can run for years before anyone checks how many genuinely new customers are arriving each month.
The order history a store already holds is enough to fix this, provided the team will accept a smaller, truer number.
Sort customers by what they did before the program touched them. Anyone with three or more orders at a steady interval before launch is a habitual buyer, so report their repeat rate on its own line and expect it to barely move. If the program works, its lift should show among one-time and lapsed buyers, the group that actually needed a nudge.
Then hold some customers out, the simplest form of causal measurement. Withhold the win-back flow or the loyalty offer from a random slice of eligible customers, roughly one in 10, for a full purchase cycle. The held-out group's repeat rate is what would have happened anyway. Only the gap between that rate and everyone else's belongs to the program.
Marketing teams resist holdouts because they seem to leave revenue on the table. The cost is one purchase cycle on a tenth of the list, and in return the team has a number it can defend in a budget meeting.
Grade what remains on contribution margin. An order recovered with a 20 percent code from someone who would have paid full price is worse than doing nothing, so net the discount, the points liability, and the messaging cost out of every retained order before counting it as a win.
Finally, put new-customer count next to repeat rate on the same page every month. A rising repeat rate alongside a shrinking count of first-time buyers means a store is living off its existing base, and seeing the two together makes that hard to explain away.
When the holdout result comes in well below the reported figure, that rarely justifies cutting the program. The better move is to aim it, spending on the lapsed and one-time buyers where the holdout shows real movement and letting the regulars keep buying at full price.
Freed budget should go back to reaching people who have never bought. Whether that means search, social or Google Ads campaigns built to find new customers, judge it on new-customer count and first-order margin rather than blended revenue, so acquisition is held to the same standard retention now is.
For CRM and marketing leaders, the practical step is one question for whoever owns the retention report. How many of these customers would have come back without us? If nobody can answer it, run a holdout on the next win-back send and compare the two groups after one purchase cycle.