In this section, we will briefly explain the two statistical approaches you can use to analyze your data.
1. Null Hypothesis Significance Testing (NHST)
This is a statistical framework traditionally used in psychological research and other social sciences. At its core, it has two competing hypotheses about the size of the true (but unknown) values in a population. To decide which hypothesis is actually true, you collect data.
The null hypothesis states that there is no difference between the values you want to compare. This is the hypothesis that is assumed to be true unless there is strong evidence to reject it — it is also the hypothesis that is actually being tested (hence the name: null hypothesis testing).
When there is evidence to suggest that the null hypothesis is false, then the so-called alternative hypothesis is assumed to be true. The alternative hypothesis usually states that there is a statistically significant difference between the values you want to compare (be it the means of two samples you collected or the mean of a sample compared to a population).
After you finish data collection, you’ll need to use a statistical test to see if there is enough evidence to reject the null hypothesis. The statistical test you choose will depend on your study design and type of data you collected, but what ultimately happens is that a so-called test statistic is calculated to compare the results you observed to the results you should have obtained if the null hypothesis was true (This is what the p-value will tell you: the probability that an effect as extreme as the one you found is to be observed in your sample, if the null hypothesis was true).
If the p-value is very small (usually, smaller than .05 or .01), this means that the difference between what the null hypothesis predicted and the data you collected is too large, so the null hypothesis should be rejected.
As you will encounter if you read on, when determining your sample size, there are two types of errors you can make when rejecting or accepting the null hypothesis. Please refer to this section if you want to learn more about these errors.
Sources:
https://opentextbc.ca/researchmethods/chapter/understanding-null-hypothesis-testing/
Pro-Tip: Point and Range Estimations – or: How to make your theory more interesting and useful
2. Bayesian Statistics
The problem with frequentist approaches is that you can only quantify evidence against a hypothesis, but never for it. If you want to test whether two groups are equal, then Bayesian statistics are the most appropriate tool.
Bayesian statistical techniques assess evidence using Bayes Factors, which weigh the evidence for two hypotheses (or models). Bayes Factors are always comparative because they reflect the likelihood of one model being true given the data, compared against the likelihood of another model being true given the data. You can test multiple models, but typically you will test two: a null model, and an effect model. The value of the Bayes Factor reflects whether the data favors one model over the other. If the evidence is equal for both models then the Bayes Factor will imply the data are insensitive (see this table for how to interpret Bayes Factors).
The open-source statistics program JASP provides an easy way to run Bayesian statistical tests.
Bayesian approaches use a combination of two probability distributions (the prior and the posterior) to determine the likelihood of a model being true. Simply put, the prior distribution describes your prior beliefs about a parameter (typically the effect size) before having run the experiment, while the posterior describes your updated beliefs about the parameter, having run the experiment and added the data to the prior.
Sources:
Rouder JN, Speckman PL, Sun D, Morey RD, Iverson G. Bayesian t tests for accepting and rejecting the null hypothesis. Psychonomic Bulletin & Review 2009 Apr;16(2):225–237.
Raftery AE. Bayesian Model Selection in Social Research. Sociological Methodology 1995;25:111–163.
Lee, M. D., & Wagenmakers, E. J.(2014).Bayesian cognitive modeling: A practical course. Cambridge university press.
Navarro, D.(2015).Learning statistics with R,pp. 557-588.
Need further help?
Click here to contact us