Calculation For Split Testing

Is there some formula to do what?

What are you actually doing to implement the split?

Cd&

Can you use something like a weighted round-Robin?

You just set up a queue that keeps track of which theme gets next, and it would be filled with 7 occurences for Control, and 3 occurences for TestA.   when the queue to direct the traffic is empty, it is then refilled with the same ratio.

What would random numbers have to do with it?  This is the design pattern.

<?php // RAY_temp_xtreme109.php 2013-10-11
error_reporting(E_ALL);

// THIS USES SESSION TO ILLUSTRATE LOGIC, BUT COULD USE DATA BASE, TOO
session_start();

// THESE NUMBERS WILL GET SENT TO THE 30% TEST SET
$nums = array(3,6,9);

// SET THE COUNTER FROM ZERO TO NINE
if (empty($_SESSION['count'])) $_SESSION['count'] = 0;
$_SESSION['count']++;
$_SESSION['count'] = $_SESSION['count'] % 10;

// CHOOSE THE TEST OR STABLE VERSION
if (in_array($_SESSION['count'], $nums))
{
    echo $_SESSION['count'] . " SENDS THIS REQUEST TO THE TEST CASE";
}
else
{
    echo $_SESSION['count'] . " SENDS THIS REQUEST TO THE STABLE CASE";
}

Select allOpen in new window

Here is the birds eye view of how we anticipate this working.

Admin:
Selects a new theme to test against the control theme.
Admin sets the percentage of traffic to split to each theme (ie: Control 70%, NewTheme 30%) This could any percentage.

Website:
As users entire the site, we need a way to determine what theme should be displayed.

This method should be lightweight (server side) as we average about 1600 people per hour.


**I like the weighted round robin idea, but im trying to conceptualize how to implement it

What granularity do you need?  Are 1% increments OK for the split between new and control?

I would also add that I've done this before, and the round-robin may be difficult to conceptualize because it's not really needed.  If you want to distribute the requests and there is any kind of reasonable arrival rate, just route the requests based on a table of pointers.

I would say 5% increments should more than sufficient between control and new.

90/10
85/15
....
..

Thanks.

So then one way would be to create an array holds 20 items.  each item in the queue represents which you should go to.

if you were at 5/95 for example... then 1 of the 20 items would be set to go to TestA and the rest would go to Control.

Now you can decide if you want to pop an item off the queue and rebuild it each time it get's empty, or could just keep a current pointer that resets to the beginning each time you get to the end.

Below is a sample array that is set to 5% / 95%

Where "next Item" could represent the current pointer concept.


1. TestA
2. Control
3. Control
4. Control
5. Control
6. Control
7. Control
8. Control
9. Control
10. Control <-- Next Item
11. Control
12. Control
13. Control
14. Control
15. Control
16. Control
17. Control
18. Control
19. Control
20. Control

will dive in to this and see how this work with a database..will keep you update. thank you

I modified this to work with a database, and dynamic enough to work with A/B/C+ testing.

Thank you!

Thanks for the points -- it's an interesting question! ~Ray