Manipulating your statistics to find significance.

Valmont

I've been wondering about the legitimacy of certain statistical procedures employed by researchers. I think my issue is best illustrated by an example:

Researcher X decides to test 50 cases/participants for an experiment and after the analysis finds that there is no significant result for whatever hypotheses they were trying to test. There is however, a trend in the data and researcher X then decides to test more cases until a significant result is achieved.

Surely this is manipulating a weakness of the t-test or ANOVA in that the higher the number of cases tested, the greater the likelihood of achieving a significant difference.

I know that some researchers will keep on testing to turn a trend into a significant result. Is this manipulation of the statistics to achieve one's goal a legitimate form of analysis or is it akin to falsifying one's results? Thanks.

2Scoops

Valmont wrote: »

Researcher X decides to test 50 cases/participants for an experiment and after the analysis finds that there is no significant result for whatever hypotheses they were trying to test. There is however, a trend in the data and researcher X then decides to test more cases until a significant result is achieved.

Surely this is manipulating a weakness of the t-test or ANOVA in that the higher the number of cases tested, the greater the likelihood of achieving a significant difference.

It's true that the larger the sample size, the more likely any difference, however small, will be significant - but that is not necessarily a weakness! A priori power analyses inform the study sample size to overcome random variability. If you only get a 'technically' significant difference by inflating sample size to a ridiculous level, it will be clear that it is not a meaningful difference because the % difference will be tiny. That is why we have many other statistics to fully describe data - effect size, confidence intervals etc.

Valmont wrote: »

I know that some researchers will keep on testing to turn a trend into a significant result. Is this manipulation of the statistics to achieve one's goal a legitimate form of analysis or is it akin to falsifying one's results? Thanks.

If there is a real difference, it will become apparent quickly. I would have no problem with someone topping up the power with a few more data points to achieve what is essentially a mathematical formality to legitimize a real difference. In contrast, if there is no real difference, you will have to chase the imaginary trend with an absurd sample to achieve significance. No one with any understanding of statistics will take seriously a significant mean difference of 1% with a sample size of 1 million! :pac:

Valmont

So if I inflate my sample size to achieve significance the statistical power and effect size will reflect this?

2Scoops

Valmont wrote: »

So if I inflate my sample size to achieve significance the statistical power and effect size will reflect this?

The effect size of a real difference shouldn't change at all, but it will help to determine how meaningful a difference is. Power increases with increasing sample size - the bigger it is, the more you are able to filter out non-systematic variation.

If you have a real difference, you simply won't need a huge sample to achieve statistical significance. And the bigger the effect size, the smaller the sample you need, typically. So, you only really need to inflate the sample to huge numbers if you don't have a real difference, in which case any statistical significance you find will be a type II error, or if the difference is very, very small, which usually means it is trivial and unimportant (with some exceptions, of course).