Let’s see. First of all, let’s keep in mind WHO and WHAT we are measuring. The people in the TEST group are customers who, without intervention, will not buy again and their risk score takes into account each customer’s unique repurchasing time. On the contrary, people in the CONTROL group are simply people who have not bought in 60 days. While it’s possible that some of them are churners, it’s just as possible that some of them have much longer repurchasing behavior and that they would have bought at a later time.
For example, assume three people responded to the offer in the test group, then because these people were real churners and the model correctly identified them, we had three incremental sales, sales we would not have had. At the same time, say that five people responded to the offer in the control group, but three of them were just a little “late” and would have bought anyway; then, your real incremental sales were only two, while you gave 45% off to three customers who would have paid full price!!
Now you can see that “number of responses” was NOT a valid KPI to measure the validity and usefulness of a statistical model and that the A/B test, the way it was designed, was useless.
A meaningful A/B test would have controlled for people who would have bought anyway at a later time. And that’s exactly what we did for the client: we re-designed the experiment and tested the methodologies again and this time the results were unquestionable: 88% of customers in the TEST group were truly incremental, compared to only 53% of the control group!
I just described the two most common mistakes; what about the other eight?
Would your organization be able to set up an experiment avoiding all of them? Would it be cost-efficient? Why not leverage Dunn Solutions’ proven expertise in the design and measuring of experiments?
Dunn Solutions’ predictive analytics team would be glad to give you a no-charge evaluation of your data and discuss whether there’s an opportunity to design the correct approach to reach your business goals! Contact us today at firstname.lastname@example.org