Calculation For Split Testing

So we are building a custom suite for our website to split test multiple themes.

Currently there is a "control" theme, and a new theme (lets call it "TestA")

We would like to split our traffic 70/30 (control/TestA)

what has not worked:
Random numbers, are not so random through PHP.

Is there some type of formula to do this?

Need some help!
LVL 2
xtreme109Asked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

COBOLdinosaurCommented:
Is there some formula to do what?

What are you actually doing to implement the split?

Cd&
0
Ken ButtersCommented:
Can you use something like a weighted round-Robin?

You just set up a queue that keeps track of which theme gets next, and it would be filled with 7 occurences for Control, and 3 occurences for TestA.   when the queue to direct the traffic is empty, it is then refilled with the same ratio.
0
Ray PaseurCommented:
What would random numbers have to do with it?  This is the design pattern.

<?php // RAY_temp_xtreme109.php 2013-10-11
error_reporting(E_ALL);

// THIS USES SESSION TO ILLUSTRATE LOGIC, BUT COULD USE DATA BASE, TOO
session_start();

// THESE NUMBERS WILL GET SENT TO THE 30% TEST SET
$nums = array(3,6,9);

// SET THE COUNTER FROM ZERO TO NINE
if (empty($_SESSION['count'])) $_SESSION['count'] = 0;
$_SESSION['count']++;
$_SESSION['count'] = $_SESSION['count'] % 10;

// CHOOSE THE TEST OR STABLE VERSION
if (in_array($_SESSION['count'], $nums))
{
    echo $_SESSION['count'] . " SENDS THIS REQUEST TO THE TEST CASE";
}
else
{
    echo $_SESSION['count'] . " SENDS THIS REQUEST TO THE STABLE CASE";
}

Open in new window

0
Big Business Goals? Which KPIs Will Help You

The most successful MSPs rely on metrics – known as key performance indicators (KPIs) – for making informed decisions that help their businesses thrive, rather than just survive. This eBook provides an overview of the most important KPIs used by top MSPs.

xtreme109Author Commented:
Here is the birds eye view of how we anticipate this working.

Admin:
Selects a new theme to test against the control theme.
Admin sets the percentage of traffic to split to each theme (ie: Control 70%, NewTheme 30%) This could any percentage.

Website:
As users entire the site, we need a way to determine what theme should be displayed.

This method should be lightweight (server side) as we average about 1600 people per hour.


**I like the weighted round robin idea, but im trying to conceptualize how to implement it
0
Ray PaseurCommented:
What granularity do you need?  Are 1% increments OK for the split between new and control?
0
Ray PaseurCommented:
I would also add that I've done this before, and the round-robin may be difficult to conceptualize because it's not really needed.  If you want to distribute the requests and there is any kind of reasonable arrival rate, just route the requests based on a table of pointers.
0
xtreme109Author Commented:
I would say 5% increments should more than sufficient between control and new.

90/10
85/15
....
..

Thanks.
0
Ken ButtersCommented:
So then one way would be to create an array holds 20 items.  each item in the queue represents which you should go to.

if you were at 5/95 for example... then 1 of the 20 items would be set to go to TestA and the rest would go to Control.

Now you can decide if you want to pop an item off the queue and rebuild it each time it get's empty, or could just keep a current pointer that resets to the beginning each time you get to the end.

Below is a sample array that is set to 5% / 95%

Where "next Item" could represent the current pointer concept.


1. TestA
2. Control
3. Control
4. Control
5. Control
6. Control
7. Control
8. Control
9. Control
10. Control <-- Next Item
11. Control
12. Control
13. Control
14. Control
15. Control
16. Control
17. Control
18. Control
19. Control
20. Control
0
Ray PaseurCommented:
This seems to work OK.  You might add a header() to redirect to the appropriate routing.

<?php // RAY_temp_xtreme109.php 2013-10-12
error_reporting(E_ALL);

// THIS USES SESSION TO ILLUSTRATE LOGIC, BUT COULD USE DATA BASE, TOO
session_start();

// INITIAL SETTING SENDS ALL REQUESTS TO CONTROL
if (empty($_SESSION['range']))
{
    $_SESSION['range'] = array_fill(0,100,0);
    $_SESSION['kount'] = -1;
}

// IF THIS IS A REQUEST TO RESET THE PERCENT OF TEST CASES
if (!empty($_POST['percent']))
{
    // RESET SESSION TO SEND ALL REQUESTS TO CONTROL
    $_SESSION['range'] = array_fill(0,100,0);
    $_SESSION['kount'] = -1;

    // CHOOSE AN APPROPRIATE PERCENT OF TEST CASES
    $pct = (int)$_POST['percent'];
    if ($pct <  1) $pct =   0;
    if ($pct > 99) $pct = 100;
    if ($pct)
    {
        // PUT SIGNALS INTO THE APPROPRIATE PERCENT OF TEST CASES
        $_SESSION['range'] = array_fill(0, $pct, 1);

        // PAD THE ARRAY WITH ZEROS FOR CONTROL CASES
        $_SESSION['range'] = array_pad($_SESSION['range'], 100, 0);

        // RANDOMIZE THE PATTERN
        shuffle ($_SESSION['range']);
    }
}

// INCREMENT THE COUNTER MODULO 100
$_SESSION['kount']++;
$_SESSION['kount'] = $_SESSION['kount'] % 100;

// GET THE ROUTING: ZERO FOR CONTROL, ONE FOR TESTING
if ($_SESSION['range'][$_SESSION['kount']])
{
    $route = 'Testing';
}
else
{
    $route = 'Control';
}

// SHOW THE CHOICE OF ROUTING (MAYBE USE header("Location") TO REDIRECT
$num = $_SESSION['kount'] + 1;
echo PHP_EOL . "THIS IS NUMBER $num ROUTED TO $route";


// PUT UP THE FORM TO SET/RESET THE PERCENT OF TEST CASES
$form = <<<ENDFORM
<form method="post">
PERCENT FOR TEST?
<input name="percent" />
<input type="submit" />
</form>
ENDFORM;

echo $form;

Open in new window

HTH, ~Ray
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
xtreme109Author Commented:
will dive in to this and see how this work with a database..will keep you update. thank you
0
xtreme109Author Commented:
I modified this to work with a database, and dynamic enough to work with A/B/C+ testing.

Thank you!
0
Ray PaseurCommented:
Thanks for the points -- it's an interesting question! ~Ray
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Algorithms

From novice to tech pro — start learning today.