I am writing a computer program (C++) in order to do Monte-Carlo simulations of the variance analysis (ANOVA). Let us assume I have an experiment with one variable at 3 levels. I want to simulate the case that the ANOVA "gets significant" although the effect does not exist in the population.
Thus, I set all 3 mean values and standard deviations to be equal. Then I generate 1000 data sets containing random numbers taken from a population with the above mean and standard deviation.
Now I compute an ANOVA with each data set and count the cases in which the ANOVA "reaches significance" by chance, that is, alpha <= 0.05.
I presumed that 5% of the data sets should reach significance by chance. However I found out that this is not the case. Especially with N > 20 there were far less data sets that reached significance (oftenly not one of 10,000).
Did I do an error in reasoning? I think alpha is defined as the probability that a statistical test reaches significance although there is no effect in the population, that is, all values are taken from the same population.
Any help is very welcome.