When analysing your data, there are two main statistical approaches available: Null Hypothesis Significance Testing (NHST) and Bayesian Statistics. Both approaches help you evaluate evidence in your data, but they answer research questions in different ways.
1. Null Hypothesis Significance Testing (NHST)
Null Hypothesis Significance Testing (NHST) is the traditional statistical framework used in psychology, social sciences, and many other research fields. It helps researchers determine whether the patterns observed in their data are likely to reflect a real effect or simply random variation.
How NHST works
NHST starts with two competing hypotheses:
Null Hypothesis (H₀): Assumes there is no difference, relationship, or effect in the population.
Alternative Hypothesis (H₁): Assumes there is a meaningful difference, relationship, or effect.
The null hypothesis is the default assumption and is the hypothesis that is directly tested. Researchers collect data and use a statistical test to determine whether there is enough evidence to reject it.
Statistical tests and p-values
After collecting data, a statistical test is performed. The choice of test depends on your study design and the type of data you have collected.
Each test produces a test statistic, which compares your observed results to the results that would be expected if the null hypothesis were true. From this, a p-value is calculated.
A p-value represents the probability of obtaining results at least as extreme as those observed if the null hypothesis were true.
Interpreting the results
A small p-value (commonly below 0.05 or 0.01) suggests that the observed data are unlikely under the null hypothesis.
When the p-value falls below the chosen significance threshold, the null hypothesis is typically rejected.
If the p-value is above the threshold, there is insufficient evidence to reject the null hypothesis.
Importantly, a non-significant result does not prove that the null hypothesis is true. It simply indicates that the collected data do not provide strong enough evidence against it.
Potential decision errors
When making decisions based on NHST, two types of errors can occur:
Type I Error: Rejecting the null hypothesis when it is actually true (a false positive).
Type II Error: Failing to reject the null hypothesis when it is actually false (a false negative).
These errors are important considerations when planning a study and determining an appropriate sample size. Please refer to this section if you want to learn more about these errors.
Sources
2. Bayesian Statistics
Bayesian Statistics provides an alternative approach to analysing data. Unlike NHST, which focuses on evidence against a null hypothesis, Bayesian methods directly compare the evidence for competing explanations of the data.
This makes Bayesian methods particularly useful when you want to evaluate evidence both for and against a hypothesis.
Bayes Factors
Bayesian analyses often use Bayes Factors (BF) to compare two statistical models or hypotheses.
Typically, these are:
A null model, representing no effect or difference.
An effect model, representing the presence of an effect or difference.
A Bayes Factor indicates how much more likely the observed data are under one model compared to another (see this table for further information on how to interpret Bayes Factors).
Interpreting Bayes Factors
A Bayes Factor greater than 1 indicates evidence in favour of the effect model.
A Bayes Factor less than 1 indicates evidence in favour of the null model.
A Bayes Factor close to 1 suggests that the data do not strongly support either model and are considered insensitive.
Because Bayes Factors compare models directly, they allow researchers to quantify evidence both for and against the null hypothesis.
Prior and posterior distributions
Bayesian analyses combine two sources of information:
Prior distribution:
The prior distribution represents your beliefs or expectations about a parameter (such as an effect size) before collecting data. Priors can be based on previous research, theoretical expectations, or default assumptions.
Posterior distribution:
The posterior distribution represents your updated beliefs after the observed data have been incorporated into the analysis.
In simple terms:
Prior beliefs + Observed data = Updated beliefs (posterior)
The posterior distribution is then used to make statistical inferences and evaluate the evidence for competing models.
Software for Bayesian analysis
Many Bayesian analyses can be performed using JASP, a free, open-source statistical software package that provides an accessible interface for both Bayesian and traditional statistical methods.
Sources
Rouder JN, Speckman PL, Sun D, Morey RD, Iverson G. Bayesian t tests for accepting and rejecting the null hypothesis. Psychonomic Bulletin & Review 2009 Apr;16(2):225–237.
Raftery AE. Bayesian Model Selection in Social Research. Sociological Methodology 1995;25:111–163.
Lee, M. D., & Wagenmakers, E. J.(2014).Bayesian cognitive modeling: A practical course. Cambridge university press.
Navarro, D.(2015).*Learning statistics with R,*pp. 557-588.