cotton9
asked on
Statistical Significance - One Variable, 20-200 Groups? (AdWords Split-testing)
Hello,
We're doing some unusual split-testing of ads in AdWords. We manage campaigns for many different local businesses, within a specialty healthcare niche. For a lot of keywords, we use identical ad copy across the different businesses - which isn't an issue, since the businesses are local, and the areas where their ads show don't overlap. \
Within each ad group, a small group of keywords triggers one of two competing ads. The ads are rotated evenly, so each ad is shown 50% of the time.
Because the ads are so specific, and the local targeted areas are so small, it takes a VERY long time to accumulate enough impressions/clicks to determine which of the two ads has a statically higher click through rate in a given area. But across, many different accounts, running essentially the same ads - we should be able to accumulate enough data to declare one of the ads more effective at generating clicks.
I admittedly forgot everything I learned in grad school about stats, but I'm thinking we have 1 variable in question (click thru rate of the individual ad) and multiple groups. Across 20 accounts, 20 groups that saw Ad#1 and 20 groups that saw Ad#2 - and the size of the groups are dramatically different.
What type of test should I use to make this declaration, and what would it look like? Any software I can plug this into?
We're doing some unusual split-testing of ads in AdWords. We manage campaigns for many different local businesses, within a specialty healthcare niche. For a lot of keywords, we use identical ad copy across the different businesses - which isn't an issue, since the businesses are local, and the areas where their ads show don't overlap. \
Within each ad group, a small group of keywords triggers one of two competing ads. The ads are rotated evenly, so each ad is shown 50% of the time.
Because the ads are so specific, and the local targeted areas are so small, it takes a VERY long time to accumulate enough impressions/clicks to determine which of the two ads has a statically higher click through rate in a given area. But across, many different accounts, running essentially the same ads - we should be able to accumulate enough data to declare one of the ads more effective at generating clicks.
I admittedly forgot everything I learned in grad school about stats, but I'm thinking we have 1 variable in question (click thru rate of the individual ad) and multiple groups. Across 20 accounts, 20 groups that saw Ad#1 and 20 groups that saw Ad#2 - and the size of the groups are dramatically different.
What type of test should I use to make this declaration, and what would it look like? Any software I can plug this into?
oz points out some ambiguity in the statement of the problem, but as I read it, the biggest difficulty is determining that the groups have about the same characteristics (same income, education, etc). If they are the same, just add them all together.
ASKER
Thanks guys,
To clarify . . . there are pairs of groups that are the same size, composed of people from the same location. But we're dealing with many different locations.
For example: The question is if Ad#1 is more effective at generating clicks than Ad#2 for a particular search term.
In Chicago:
Ad#1 was seen 1000 times and clicked on 30 times.
Ad #2 was seen 1000 times and clicked on 15 times.
In SLC:
Ad#1 was seen 800 times and clicked on 21 times.
Ad #2 was seen 800 times and clicked on 23 times.
In Baltimore:
Ad#1 was seen 500 times and clicked on 28 times.
Ad #2 was seen 500 times and clicked on 31 times.
Etc. . . . For at least 20 different cities.
The ad copy is essentially the same across all cities.
Obviously, there are many demographic factors that could be considered to make the statistical significance statement more accurate, but that's all we have to work with. Luckily, we're not looking to be published though . . . we just want to be as accurate as possible for our internal uses.
Thanks!
To clarify . . . there are pairs of groups that are the same size, composed of people from the same location. But we're dealing with many different locations.
For example: The question is if Ad#1 is more effective at generating clicks than Ad#2 for a particular search term.
In Chicago:
Ad#1 was seen 1000 times and clicked on 30 times.
Ad #2 was seen 1000 times and clicked on 15 times.
In SLC:
Ad#1 was seen 800 times and clicked on 21 times.
Ad #2 was seen 800 times and clicked on 23 times.
In Baltimore:
Ad#1 was seen 500 times and clicked on 28 times.
Ad #2 was seen 500 times and clicked on 31 times.
Etc. . . . For at least 20 different cities.
The ad copy is essentially the same across all cities.
Obviously, there are many demographic factors that could be considered to make the statistical significance statement more accurate, but that's all we have to work with. Luckily, we're not looking to be published though . . . we just want to be as accurate as possible for our internal uses.
Thanks!
add all the groups together
ie in your example
ad 1 was seen 2300 times and clicked 79 times
ad 2 was seen 2300 times and clicked 69 times
neglecting demographics ad 1 wins but not by much
ie in your example
ad 1 was seen 2300 times and clicked 79 times
ad 2 was seen 2300 times and clicked 69 times
neglecting demographics ad 1 wins but not by much
There really isn't enough information to reliably distinguish demographic factors from add copy factors,
although you can get an estimate of how likely it would be to obtain the distribution you see
under the assumption that the relative effect of add copy is independent of demographic factors, and an estimate for how likely it would be for particular demographic to deviate as much as it does from the overall averages.
although you can get an estimate of how likely it would be to obtain the distribution you see
under the assumption that the relative effect of add copy is independent of demographic factors, and an estimate for how likely it would be for particular demographic to deviate as much as it does from the overall averages.
If you're really wanting to go crazy with the stats, you should look into SAS or MATLAB. If you just want something that you probably already have, then Excel should have what you need. The Analysis Tools Pak (which is optional on install) has a lot of good stats functions. If you don't see it in your tools menu, you can install it from the disk or download it.
ASKER
Thank you all for your comments.
How do I know if the difference is significant? The little knowledge I have of stats tells me that sample/population size needs to be considered. I thought that the number of different locations and their sizes would need to be considered too.
If it's agreed that the different location groups (demographics) are not worth considering and the responses should simply be added up, what type of test should I use to determine statistical significance?
add all the groups together
ie in your example
ad 1 was seen 2300 times and clicked 79 times
ad 2 was seen 2300 times and clicked 69 times
neglecting demographics ad 1 wins but not by much
How do I know if the difference is significant? The little knowledge I have of stats tells me that sample/population size needs to be considered. I thought that the number of different locations and their sizes would need to be considered too.
If it's agreed that the different location groups (demographics) are not worth considering and the responses should simply be added up, what type of test should I use to determine statistical significance?
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
why would the size of the groups be dramatically different for groups that saw Ad#1 and groups that saw Ad#2?