You may have heard of A/B Testing and Multivariable Testing (Experimental Design) in terms of website optimization to maximize conversions among other goals. What are these testing regimens? When should you use them?

A/B testing is just that — testing one configured webpage (A) against another (B) and analyzing which is best. Multivariate testing (MVT) is a newer term for experimental design and focuses on the changes of several variables on a given webpage to determine which settings are best. 

A/B testing has its place and can tell you which is better, A or B. BUT, if more than one thing is changed from A to B, you won’t know what led to the improvement. Knowing why is important for further optimization.

The most efficient and effective way to find out which variables and their possible interactions lead to an improvement is to use statistical experimental design. Experimental design has been around for over 90 years and gained momentum during applications in World War II.

To explore these topics, we will use three resources. 

First, Kahlid Saleh, CEO of Invesp, will compare the differences between A/B and Multivariate Testing and the guidelines for using each.

Next, we will get an overview of experimental design from Vijay Sabale, Director at Learn & Apply.

Finally, we will give you a key reference for learning more about the use of experimental design in marketing and website optimization.

On to Khalid Saleh’s presentation: The Difference Between A/B and multivariate testing.

Khalid covers three topics:

  • The difference between AB and Multivariate Testing
  • What type of test should you use?
  • 5 Things to remember when conducting testing on your website

As Khalid suggests, A/B testing won’t allow you to test for interactions between variables. That’s where multivariate testing or experimental design is needed.

You can view Khalid’s video here.

What are interactions and why are they key in optimization? By definition, an interaction occurs when the effect of one variable on a response depends on the level of another variable. They are important in optimization because it is usually the settings associated with interacting variables that lead to an improved response.

To gain more perspective on experimental design, we turn to Vijay Sabale’s presentation: Design of Experiments (DOE). While Vijay’s video does not mention website design, it gives a good introduction to DOE and why DOE is needed to determine interactions between variables.

You can view Vijay’s video here.

To learn more about experimental design in marketing applications such as email campaigns and website design, check out the excellent book Testing 1 - 2 - 3: Experimental Design with Applications in Marketing and Service Operations, 2007, Stanford University Press, Johannes Ledolter and Arthur J. Swersey.

We repeat the information about the book from Amazon below.

“This book is about the power of statistical experiments. In the past, books on experimental design focused almost entirely on manufacturing problems. In contrast, although this book is relevant to manufacturing and includes useful examples in that area, its emphasis is on applications to marketing and service operations. The authors provide a fresh and practical treatment of the key topics in designing and analyzing experiments. Testing in the business world is commonplace, and the usual approach is to change one factor at a time while holding other factors constant. This approach may seem logical and appealing, but, as the authors explain, it is highly inefficient, and may lead to wrong conclusions. The better method, the authors propose, is to test all factors simultaneously. Doing so not only reduces the costs of experimenting but also provides the decision-maker with better information.

Throughout the book, the authors illustrate concepts with practical examples. In addition, the book includes a set of 13 separate real-world cases based on the actual implementation of experimental design methods.”

Here is the link to the book from Amazon.