Dear experts,

I am writing a computer program (C++) in order to do Monte-Carlo simulations of the variance analysis (ANOVA). Let us assume I have an experiment with one variable at 3 levels. I want to simulate the case that the ANOVA "gets significant" although the effect does not exist in the population.

Thus, I set all 3 mean values and standard deviations to be equal. Then I generate 1000 data sets containing random numbers taken from a population with the above mean and standard deviation.

Now I compute an ANOVA with each data set and count the cases in which the ANOVA "reaches significance" by chance, that is, alpha <= 0.05.

I presumed that 5% of the data sets should reach significance by chance. However I found out that this is not the case. Especially with N > 20 there were far less data sets that reached significance (oftenly not one of 10,000).

Did I do an error in reasoning? I think alpha is defined as the probability that a statistical test reaches significance although there is no effect in the population, that is, all values are taken from the same population.

Any help is very welcome.

Sincerely,

Albert

I have found the reason for the problem. There was an error in my code when standardizing the random matrices to corrct means and standard deviations.

Now, which the error being corrected, the simulated alpha error probability is 0.05 - as it is expected to be.

Thank you and best regards,

Albert