Hi Experts!

This has been bugging me for years so I thought I'd see if I could get a definitive answer here. I have three scenarios. I ask questions about whether respondents like two products using a 10 point scale. There are three samples:

1) 200 people are asked both questions ("Do you like product A" & "Do you like product B")

2) 75 people are asked question one ONLY, 75 people are asked question two ONLY and 50 people are asked both questions

3) 100 people are asked question one ONLY, 100 people are asked question two ONLY

If I want to know if there is a significant difference between the mean of the answer to question one and the mean of the answer to question two, do I use the same formula to stat test these three samples?

Thanks!

Not exactly, but close enough for the moment.

>> If the President's speech is good, doesn't that bias the AFTER results?

What is the question? Are you trying to determine presidential popularity or speech effectiveness?

A good speech may or may not affect the president's popularity. You actually have to run the test to find out.

>> And if I use a really good brewing method for the coffee and a really bad brewing method for the tea, am I not biasing the results?

This would be a horrible way to run a taste test, but I thought you were conducting a poll.

You have to describe one survey/experiment completely, then dig into the details.

>> My question is, understanding there is bias, can't I use a test (or tests) to show that the bias is significant?

You may think or worry that there is a bias, but you can't tell for sure until you run the test.

>> If the rating for liking the coffee is 3.6 and the rating for tea is 4.5, can't I say MY SAMPLE likes tea significantly more than they like coffee?

Even if all you are asking your sample about is coffee vs tea, you actually have to do the test before you say anything about significance.

In an earlier post I said

Here is a better write-up for the T-testand forgot to put in the link.http://www.socialresearchmethods.net/kb/stat_t.php

Even with very large samples, you can have a large difference in means that is not significant. You can't look at just the two means and say anything about significance. You have to actually do the calculations.

==========================

>> If I have a mean of one million coffee drinkers and one million tea drinkers, my

common sensesays that even if two hundred thousand of them drink both, if the rating for liking the coffee is 3.6 and the rating for tea is 4.5 THERE IS A SIGNIFICANT DIFFERENCE between them.This is really where we disagree:

If you have the data, I think it would be much better to do the T-test calculations than to rely on common sense.

Here again, you have collected and dumped the data without specifying the problem carefully.

If you have that much data, you can include all the people that drink both coffee and tea.

Or you can throw out all the people that drink both.

Or you can look at only the people that drink both.

And you should probably do all these three of these tests and more.