 Navigation

Click to Take the FREE Statistics Crash-Course

Search... 

17 Statistical Hypothesis Tests in Python (Cheat Sheet)

by Jason Brownlee on August 15, 2018 in Statistics

Tweet Share Share

Last Updated on November 28, 2019

Quick-reference guide to the 17 statistical hypothesis tests that you need in

applied machine learning, with sample code in Python.
Although there are hundreds of statistical hypothesis tests that you could use, there is only a small subset that you
may need to use in a machine learning project.

In this post, you will discover a cheat sheet for the most popular statistical hypothesis tests for a machine learning
project with examples using the Python API.

Each statistical test is presented in a consistent way, including: Start Machine Learning
 The name of the test.
What the test is checking.
The key assumptions of the test.
How the test result is interpreted.
Python API for using the test.

Note, when it comes to assumptions such as the expected distribution of data or sample size, the results of a given
test are likely to degrade gracefully rather than become immediately unusable if an assumption is violated.

Generally, data samples need to be representative of the domain and large enough to expose their distribution to
analysis.

In some cases, the data can be corrected to meet the assumptions, such as correcting a nearly normal distribution to
be normal by removing outliers, or using a correction to the degrees Start
of freedom in a statistical
Machine ×
test when samples have
Learning
differing variance, to name two examples.
You can master applied Machine Learning
Finally, there may be multiple tests for a given concern, e.g. normality. We cannot
without math orget crisp
fancy answers to questions with
degrees.
Findatout
statistics; instead, we get probabilistic answers. As such, we can arrive how in this
different free and
answers topractical
the same course.
question by
considering the question in different ways. Hence the need for multiple different tests for some questions we may have
about data. Email Address

Kick-start your project with my new book Statistics for Machine Learning, including step-by-step tutorials and the
START MY EMAIL COURSE
Python source code files for all examples.

Let’s get started.

Update Nov/2018: Added a better overview of the tests covered.

Update Nov/2019: Added complete working examples of each test. Add time series tests.
Start Machine Learning
 Start Machine Learning ×
You can master applied Machine Learning
without math or fancy degrees.
Find out how in this free and practical course.

Email Address

Statistical Hypothesis Tests in Python Cheat Sheet

Photo by davemichuda, some rights START
reserved.
MY EMAIL COURSE

Tutorial Overview
This tutorial is divided into 5 parts; they are:

1. Normality Tests Start Machine Learning

 1. Shapiro-Wilk Test
2. D’Agostino’s K^2 Test
3. Anderson-Darling Test
2. Correlation Tests
1. Pearson’s Correlation Coefficient
2. Spearman’s Rank Correlation
3. Kendall’s Rank Correlation
4. Chi-Squared Test
3. Stationary Tests
1. Augmented Dickey-Fuller
2. Kwiatkowski-Phillips-Schmidt-Shin
4. Parametric Statistical Hypothesis Tests
1. Student’s t-test
Start Machine Learning ×
2. Paired Student’s t-test
3. Analysis of Variance Test (ANOVA) You can master applied Machine Learning
4. Repeated Measures ANOVA Test without math or fancy degrees.
5. Nonparametric Statistical Hypothesis Tests Find out how in this free and practical course.
1. Mann-Whitney U Test
2. Wilcoxon Signed-Rank Test Email Address
3. Kruskal-Wallis H Test
4. Friedman Test
START MY EMAIL COURSE

1. Normality Tests
This section lists statistical tests that you can use to check if your data has a Gaussian distribution.

Shapiro-Wilk Test
Start Machine Learning
Tests whether a data sample has a Gaussian distribution.

Assumptions

Observations in each sample are independent and identically distributed (iid).

Interpretation

H0: the sample has a Gaussian distribution.

H1: the sample does not have a Gaussian distribution.

Python Code

1 # Example of the Shapiro-Wilk Normality Test

2
3
from scipy.stats import shapiro
Start Machine Learning
data = [0.873, 2.817, 0.121, -0.945, -0.055, -1.436, 0.360, -1.478, -1.637, -1.869]
×
4 stat, p = shapiro(data)
5 print('stat=%.3f, p=%.3f' % (stat, p)) You can master applied Machine Learning
6 if p > 0.05: without math or fancy degrees.
7 print('Probably Gaussian')
8 else: Find out how in this free and practical course.
9 print('Probably not Gaussian')

More Information Email Address

A Gentle Introduction to Normality Tests in Python

START MY EMAIL COURSE
scipy.stats.shapiro
Shapiro-Wilk test on Wikipedia

D’Agostino’s K^2 Test

Tests whether a data sample has a Gaussian distribution.
Start Machine Learning
Assumptions

Observations in each sample are independent and identically distributed (iid).

Interpretation

H0: the sample has a Gaussian distribution.

H1: the sample does not have a Gaussian distribution.

Python Code

1 # Example of the D'Agostino's K^2 Normality Test

2 from scipy.stats import normaltest
3 data = [0.873, 2.817, 0.121, -0.945, -0.055, -1.436, 0.360, -1.478, -1.637, -1.869]
4
5
stat, p = normaltest(data)
print('stat=%.3f, p=%.3f' % (stat, p)) Start Machine Learning ×
6 if p > 0.05:
7 print('Probably Gaussian')
8 else: You can master applied Machine Learning
9 print('Probably not Gaussian') without math or fancy degrees.
Find out how in this free and practical course.
More Information

A Gentle Introduction to Normality Tests in Python Email Address

scipy.stats.normaltest
D’Agostino’s K-squared test on Wikipedia START MY EMAIL COURSE

Anderson-Darling Test
Tests whether a data sample has a Gaussian distribution.

Assumptions
Start Machine Learning
 Observations in each sample are independent and identically distributed (iid).

Interpretation

H0: the sample has a Gaussian distribution.

H1: the sample does not have a Gaussian distribution.

Python Code

1 # Example of the Anderson-Darling Normality Test

2 from scipy.stats import anderson
3 data = [0.873, 2.817, 0.121, -0.945, -0.055, -1.436, 0.360, -1.478, -1.637, -1.869]
4 result = anderson(data)
5 print('stat=%.3f' % (result.statistic))
6 for i in range(len(result.critical_values)):
7
8
sl, cv = result.significance_level[i], result.critical_values[i]
if result.statistic < cv: Start Machine Learning ×
9 print('Probably Gaussian at the %.1f%% level' % (sl))
10 else: You can master applied Machine Learning
11 print('Probably not Gaussian at the %.1f%% level' % (sl))
without math or fancy degrees.
More Information Find out how in this free and practical course.

A Gentle Introduction to Normality Tests in Python Email Address

scipy.stats.anderson
Anderson-Darling test on Wikipedia
START MY EMAIL COURSE

2. Correlation Tests
This section lists statistical tests that you can use to check if two samples are related.

Pearson’s Correlation Coefficient

Start Machine Learning
Tests whether two samples have a linear relationship.

Assumptions

Observations in each sample are independent and identically distributed (iid).

Observations in each sample are normally distributed.
Observations in each sample have the same variance.

Interpretation

H0: the two samples are independent.

H1: there is a dependency between the samples.

Python Code
Start Machine Learning ×
1 # Example of the Pearson's Correlation test
2 from scipy.stats import pearsonr You can master applied Machine Learning
3 data1 = [0.873, 2.817, 0.121, -0.945, -0.055, -1.436, 0.360, -1.478,
without math-1.637,
or fancy-1.869]
degrees.
4 data2 = [0.353, 3.517, 0.125, -7.545, -0.555, -1.536, 3.350, -1.578, -3.537, -1.579]
5 stat, p = pearsonr(data1, data2) Find out how in this free and practical course.
6 print('stat=%.3f, p=%.3f' % (stat, p))
7 if p > 0.05:
8 print('Probably independent') Email Address
9 else:
10 print('Probably dependent')
START MY EMAIL COURSE
More Information

How to Calculate Correlation Between Variables in Python

scipy.stats.pearsonr
Pearson’s correlation coefficient on Wikipedia

Start Machine Learning

Spearman’s Rank Correlation
Tests whether two samples have a monotonic relationship.

Assumptions

Observations in each sample are independent and identically distributed (iid).

Observations in each sample can be ranked.

Interpretation

H0: the two samples are independent.

H1: there is a dependency between the samples.

Python Code

1 # Example of the Spearman's Rank Correlation Test Start Machine Learning ×

2 from scipy.stats import spearmanr
3 data1 = [0.873, 2.817, 0.121, -0.945, -0.055, -1.436, 0.360, -1.478, -1.637, -1.869]
You can master applied Machine Learning
4 data2 = [0.353, 3.517, 0.125, -7.545, -0.555, -1.536, 3.350, -1.578, -3.537, -1.579]
5 stat, p = spearmanr(data1, data2) without math or fancy degrees.
6 print('stat=%.3f, p=%.3f' % (stat, p)) Find out how in this free and practical course.
7 if p > 0.05:
8 print('Probably independent')
9 else: Email Address
10 print('Probably dependent')

More Information
START MY EMAIL COURSE

How to Calculate Nonparametric Rank Correlation in Python

scipy.stats.spearmanr
Spearman’s rank correlation coefficient on Wikipedia

Kendall’s Rank Correlation

Start Machine Learning
Tests whether two samples have a monotonic relationship.

Assumptions

Observations in each sample are independent and identically distributed (iid).

Observations in each sample can be ranked.

Interpretation

H0: the two samples are independent.

H1: there is a dependency between the samples.

Python Code

1 # Example of the Kendall's Rank Correlation Test Start Machine Learning ×

2 from scipy.stats import kendalltau
3 data1 = [0.873, 2.817, 0.121, -0.945, -0.055, -1.436, 0.360, -1.478, -1.637, -1.869]
You can master applied Machine Learning
4 data2 = [0.353, 3.517, 0.125, -7.545, -0.555, -1.536, 3.350, -1.578, -3.537, -1.579]
5 stat, p = kendalltau(data1, data2) without math or fancy degrees.
6 print('stat=%.3f, p=%.3f' % (stat, p)) Find out how in this free and practical course.
7 if p > 0.05:
8 print('Probably independent')
9 else: Email Address
10 print('Probably dependent')

More Information
START MY EMAIL COURSE

How to Calculate Nonparametric Rank Correlation in Python

scipy.stats.kendalltau
Kendall rank correlation coefficient on Wikipedia

Chi-Squared Test
Start Machine Learning
Tests whether two categorical variables are related or independent.

Assumptions

Observations used in the calculation of the contingency table are independent.

25 or more examples in each cell of the contingency table.

Interpretation

H0: the two samples are independent.

H1: there is a dependency between the samples.

Python Code

1 # Example of the Chi-Squared Test Start Machine Learning ×

2 from scipy.stats import chi2_contingency
3 table = [[10, 20, 30],[6,  9,  17]]
You can master applied Machine Learning
4 stat, p, dof, expected = chi2_contingency(table)
5 print('stat=%.3f, p=%.3f' % (stat, p)) without math or fancy degrees.
6 if p > 0.05: Find out how in this free and practical course.
7 print('Probably independent')
8 else:
9 print('Probably dependent') Email Address
More Information
START MY EMAIL COURSE
A Gentle Introduction to the Chi-Squared Test for Machine Learning
scipy.stats.chi2_contingency
Chi-Squared test on Wikipedia

3. Stationary Tests
Start Machine Learning
This section lists statistical tests that you can use to check if a time series is stationary or not.

Augmented Dickey-Fuller Unit Root Test

Tests whether a time series has a unit root, e.g. has a trend or more generally is autoregressive.

Assumptions

Observations in are temporally ordered.

Interpretation

H0: a unit root is present (series is non-stationary).

H1: a unit root is not present (series is stationary).
Start Machine Learning ×
Python Code
You can master applied Machine Learning
1 # Example of the Augmented Dickey-Fuller unit root test without math or fancy degrees.
2 from statsmodels.tsa.stattools import adfuller Find out how in this free and practical course.
3 data = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
4 stat, p, lags, obs, crit, t = adfuller(data)
5 print('stat=%.3f, p=%.3f' % (stat, p))
Email Address
6 if p > 0.05:
7 print('Probably not Stationary')
8 else:
9 print('Probably Stationary') START MY EMAIL COURSE

More Information

How to Check if Time Series Data is Stationary with Python

statsmodels.tsa.stattools.adfuller API.
Augmented Dickey–Fuller test, Wikipedia.
Start Machine Learning
Kwiatkowski-Phillips-Schmidt-Shin
Tests whether a time series is trend stationary or not.

Assumptions

Observations in are temporally ordered.

Interpretation

H0: the time series is not trend-stationary.

H1: the time series is trend-stationary.

Python Code
Start Machine Learning ×
1 # Example of the Kwiatkowski-Phillips-Schmidt-Shin test
2 from statsmodels.tsa.stattools import kpss You can master applied Machine Learning
3 data = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
4 stat, p, lags, crit = kpss(data) without math or fancy degrees.
5 print('stat=%.3f, p=%.3f' % (stat, p)) Find out how in this free and practical course.
6 if p > 0.05:
7 print('Probably not Stationary')
8 else: Email Address
9 print('Probably Stationary')

More Information
START MY EMAIL COURSE

statsmodels.tsa.stattools.kpss API.
KPSS test, Wikipedia.

4. Parametric Statistical Hypothesis Tests

Start Machine Learning
This section lists statistical tests that you can use to compare data samples.
Student’s t-test
Tests whether the means of two independent samples are significantly different.

Assumptions

Observations in each sample are independent and identically distributed (iid).

Observations in each sample are normally distributed.
Observations in each sample have the same variance.

Interpretation

H0: the means of the samples are equal.

H1: the means of the samples are unequal.
Start Machine Learning ×
Python Code
You can master applied Machine Learning
1 # Example of the Student's t-test without math or fancy degrees.
2 from scipy.stats import ttest_ind Find out how in this free and practical course.
3 data1 = [0.873, 2.817, 0.121, -0.945, -0.055, -1.436, 0.360, -1.478, -1.637, -1.869]
4 data2 = [1.142, -0.432, -0.938, -0.729, -0.846, -0.157, 0.500, 1.183, -1.075, -0.169]
5 stat, p = ttest_ind(data1, data2) Email Address
6 print('stat=%.3f, p=%.3f' % (stat, p))
7 if p > 0.05:
8 print('Probably the same distribution')
9 else: START MY EMAIL COURSE
10 print('Probably different distributions')

More Information

How to Calculate Parametric Statistical Hypothesis Tests in Python

scipy.stats.ttest_ind
Student’s t-test on Wikipedia Start Machine Learning
Paired Student’s t-test
Tests whether the means of two paired samples are significantly different.

Assumptions

Observations in each sample are independent and identically distributed (iid).

Observations in each sample are normally distributed.
Observations in each sample have the same variance.
Observations across each sample are paired.

Interpretation

H0: the means of the samples are equal.

Start Machine Learning ×
H1: the means of the samples are unequal.
You can master applied Machine Learning
Python Code
without math or fancy degrees.

1 # Example of the Paired Student's t-test

Find out how in this free and practical course.
2 from scipy.stats import ttest_rel
3 data1 = [0.873, 2.817, 0.121, -0.945, -0.055, -1.436, 0.360, -1.478, -1.637, -1.869]
4 Email
data2 = [1.142, -0.432, -0.938, -0.729, -0.846, -0.157, 0.500, Address
1.183, -1.075, -0.169]
5 stat, p = ttest_rel(data1, data2)
6 print('stat=%.3f, p=%.3f' % (stat, p))
7 if p > 0.05: START MY EMAIL COURSE
8 print('Probably the same distribution')
9 else:
10 print('Probably different distributions')

More Information

How to Calculate Parametric Statistical Hypothesis Tests in Python

scipy.stats.ttest_rel Start Machine Learning
 Student’s t-test on Wikipedia

Analysis of Variance Test (ANOVA)

Tests whether the means of two or more independent samples are significantly different.

Assumptions

Observations in each sample are independent and identically distributed (iid).

Observations in each sample are normally distributed.
Observations in each sample have the same variance.

Interpretation

H0: the means of the samples are equal. Start Machine Learning ×
H1: one or more of the means of the samples are unequal.
You can master applied Machine Learning
Python Code without math or fancy degrees.
Find out how in this free and practical course.
1 # Example of the Analysis of Variance Test
2 from scipy.stats import f_oneway
3 data1 = [0.873, 2.817, 0.121, -0.945, -0.055, -1.436, 0.360,Email Address
-1.478, -1.637, -1.869]
4 data2 = [1.142, -0.432, -0.938, -0.729, -0.846, -0.157, 0.500, 1.183, -1.075, -0.169]
5 data3 = [-0.208, 0.696, 0.928, -1.148, -0.213, 0.229, 0.137, 0.269, -0.870, -1.204]
6 stat, p = f_oneway(data1, data2, data3) START MY EMAIL COURSE
7 print('stat=%.3f, p=%.3f' % (stat, p))
8 if p > 0.05:
9 print('Probably the same distribution')
10 else:
11 print('Probably different distributions')

More Information
Start Machine Learning
 How to Calculate Parametric Statistical Hypothesis Tests in Python
scipy.stats.f_oneway
Analysis of variance on Wikipedia

Repeated Measures ANOVA Test

Tests whether the means of two or more paired samples are significantly different.

Assumptions

Observations in each sample are independent and identically distributed (iid).

Observations in each sample are normally distributed.
Observations in each sample have the same variance.
Observations across each sample are paired.
Start Machine Learning ×
Interpretation
You can master applied Machine Learning
without math or fancy degrees.
H0: the means of the samples are equal.
Find out how in this free and practical course.
H1: one or more of the means of the samples are unequal.

Python Code Email Address

Currently not supported in Python.

START MY EMAIL COURSE

More Information

How to Calculate Parametric Statistical Hypothesis Tests in Python

Analysis of variance on Wikipedia

Start Machine Learning

5. Nonparametric Statistical Hypothesis Tests
Mann-Whitney U Test
Tests whether the distributions of two independent samples are equal or not.

Assumptions

Observations in each sample are independent and identically distributed (iid).

Observations in each sample can be ranked.

Interpretation

H0: the distributions of both samples are equal.

Start Machine Learning ×
H1: the distributions of both samples are not equal.
You can master applied Machine Learning
Python Code without math or fancy degrees.
Find out how in this free and practical course.
1 # Example of the Mann-Whitney U Test
2 from scipy.stats import mannwhitneyu
3 data1 = [0.873, 2.817, 0.121, -0.945, -0.055, -1.436, 0.360, -1.478, -1.637, -1.869]
Email Address
4 data2 = [1.142, -0.432, -0.938, -0.729, -0.846, -0.157, 0.500, 1.183, -1.075, -0.169]
5 stat, p = mannwhitneyu(data1, data2)
6 print('stat=%.3f, p=%.3f' % (stat, p))
7 if p > 0.05: START MY EMAIL COURSE
8 print('Probably the same distribution')
9 else:
10 print('Probably different distributions')

More Information

How to Calculate Nonparametric Statistical Hypothesis Tests in Python

Start Machine Learning
scipy.stats.mannwhitneyu
 Mann-Whitney U test on Wikipedia

Wilcoxon Signed-Rank Test

Tests whether the distributions of two paired samples are equal or not.

Assumptions

Observations in each sample are independent and identically distributed (iid).

Observations in each sample can be ranked.
Observations across each sample are paired.

Interpretation

H0: the distributions of both samples are equal. Start Machine Learning ×
H1: the distributions of both samples are not equal.
You can master applied Machine Learning
Python Code without math or fancy degrees.
Find out how in this free and practical course.
1 # Example of the Wilcoxon Signed-Rank Test
2 from scipy.stats import wilcoxon
3 data1 = [0.873, 2.817, 0.121, -0.945, -0.055, -1.436, 0.360,Email Address
-1.478, -1.637, -1.869]
4 data2 = [1.142, -0.432, -0.938, -0.729, -0.846, -0.157, 0.500, 1.183, -1.075, -0.169]
5 stat, p = wilcoxon(data1, data2)
6 print('stat=%.3f, p=%.3f' % (stat, p)) START MY EMAIL COURSE
7 if p > 0.05:
8 print('Probably the same distribution')
9 else:
10 print('Probably different distributions')

More Information

How to Calculate Nonparametric Statistical Hypothesis Tests in Python

Start Machine Learning
 scipy.stats.wilcoxon
Wilcoxon signed-rank test on Wikipedia

Kruskal-Wallis H Test
Tests whether the distributions of two or more independent samples are equal or not.

Assumptions

Observations in each sample are independent and identically distributed (iid).

Observations in each sample can be ranked.

Interpretation

H0: the distributions of all samples are equal. Start Machine Learning ×
H1: the distributions of one or more samples are not equal.
You can master applied Machine Learning
Python Code without math or fancy degrees.
Find out how in this free and practical course.
1 # Example of the Kruskal-Wallis H Test
2 from scipy.stats import kruskal
3 data1 = [0.873, 2.817, 0.121, -0.945, -0.055, -1.436, 0.360,Email Address
-1.478, -1.637, -1.869]
4 data2 = [1.142, -0.432, -0.938, -0.729, -0.846, -0.157, 0.500, 1.183, -1.075, -0.169]
5 stat, p = kruskal(data1, data2)
6 print('stat=%.3f, p=%.3f' % (stat, p)) START MY EMAIL COURSE
7 if p > 0.05:
8 print('Probably the same distribution')
9 else:
10 print('Probably different distributions')

More Information

How to Calculate Nonparametric Statistical Hypothesis Tests in Python

Start Machine Learning
 scipy.stats.kruskal
Kruskal-Wallis one-way analysis of variance on Wikipedia

Friedman Test
Tests whether the distributions of two or more paired samples are equal or not.

Assumptions

Observations in each sample are independent and identically distributed (iid).

Observations in each sample can be ranked.
Observations across each sample are paired.

Interpretation
Start Machine Learning ×
H0: the distributions of all samples are equal.
You can master applied Machine Learning
H1: the distributions of one or more samples are not equal.
without math or fancy degrees.

Python Code Find out how in this free and practical course.

1 # Example of the Friedman Test Email Address

2 from scipy.stats import friedmanchisquare
3 data1 = [0.873, 2.817, 0.121, -0.945, -0.055, -1.436, 0.360, -1.478, -1.637, -1.869]
4 data2 = [1.142, -0.432, -0.938, -0.729, -0.846, -0.157, 0.500, 1.183, -1.075, -0.169]
5 data3 = [-0.208, 0.696, 0.928, -1.148, -0.213, 0.229, 0.137, START MY-0.870,
0.269, EMAIL COURSE
-1.204]
6 stat, p = friedmanchisquare(data1, data2, data3)
7 print('stat=%.3f, p=%.3f' % (stat, p))
8 if p > 0.05:
9 print('Probably the same distribution')
10 else:
11 print('Probably different distributions')

More Information Start Machine Learning

 How to Calculate Nonparametric Statistical Hypothesis Tests in Python
scipy.stats.friedmanchisquare
Friedman test on Wikipedia

Further Reading
This section provides more resources on the topic if you are looking to go deeper.

A Gentle Introduction to Normality Tests in Python

How to Use Correlation to Understand the Relationship Between Variables
How to Use Parametric Statistical Significance Tests in Python
A Gentle Introduction to Statistical Hypothesis Tests

Summary Start Machine Learning ×

In this tutorial, you discovered the key statistical hypothesis tests thatYou
youcan master applied Machine Learning
may need to use in a machine learning
without math or fancy degrees.
project.
Find out how in this free and practical course.

Specifically, you learned:

Email Address
The types of tests to use in different circumstances, such as normality checking, relationships between variables,
and differences between samples.
START MY EMAIL COURSE
The key assumptions for each test and how to interpret the test result.
How to implement the test using the Python API.

Do you have any questions?

Ask your questions in the comments below and I will do my best to answer.

Start Machine Learning

Did I miss an important statistical test or key assumption for one of the listed tests?
Let me know in the comments below.

Get a Handle on Statistics for Machine Learning!

Develop a working understanding of statistics
...by writing lines of code in python

Discover how in my new Ebook:

Statistical Methods for Machine Learning

It provides self-study tutorials on topics like:

Start Machine Learning ×
Hypothesis Tests, Correlation, Nonparametric Stats, Resampling, and much more...
You can master applied Machine Learning
Discover how to Transform Data into Knowledge
without math or fancy degrees.
Skip theFind out how in
Academics. thisResults.
Just free and practical course.

SEE WHAT'S INSIDE

Email Address

START MY EMAIL COURSE

Tweet Share Share

About Jason Brownlee

Start Machine Learning
 Jason Brownlee, PhD is a machine learning specialist who teaches developers how to get results with modern
machine learning methods via hands-on tutorials.
View all posts by Jason Brownlee →

 How to Reduce Variance in a Final Machine Learning Model A Gentle Introduction to SARIMA for Time Series Forecasting in Python 

61 Responses to 17 Statistical Hypothesis Tests in Python (Cheat Sheet)

Start Machine Learning ×
REPLY 
Jonathan dunne August 17, 2018 at 7:17 am # You can master applied Machine Learning
without math or fancy degrees.
hi, the list looks good. a few omissions. fishers exact test and Bernards test (potentially more power than a
Find out how in this free and practical course.
fishers exact test)

one note on the anderson darling test. the use of p values to determineEmail
GoF has been discouraged in some fields .
Address

START MY EMAIL COURSE

REPLY 
Jason Brownlee August 17, 2018 at 7:43 am #

Excellent note, thanks Jonathan.

Indeed, I think it was a journal of psychology that has adopted “estimation statistics” instead of hypothesis tests in
reporting results.
Start Machine Learning
 REPLY 
Hitesh August 17, 2018 at 3:19 pm #

Very Very Good and Useful Article

REPLY 
Jason Brownlee August 18, 2018 at 5:32 am #

Thanks, I’m happy to hear that.

Barrie August 17, 2018 at 9:38 pm # Start Machine Learning ×

REPLY 

Hi, thanks for this nice overview. You can master applied Machine Learning
without math or fancy degrees.
Some of these tests, like friedmanchisquare, expect that the quantity of events is the group to remain the same over
Find out how in this free and practical course.
time. But in practice this is not allways the case.

Lets say there are 4 observations on a group of 100 people, but the sizeEmail
of theAddress
response from this group changes over
time with n1=100, n2=95, n3=98, n4=60 respondants.
n4 is smaller because some external factor like bad weather.
What would be your advice on how to tackle this different ‘respondants’ START MY EMAIL
sizes over time? COURSE

REPLY 
Jason Brownlee August 18, 2018 at 5:36 am #

Good question. Start Machine Learning

 Perhaps check the literature for corrections to the degrees of freedom for this situation?

REPLY 
Fredrik August 21, 2018 at 5:44 am #

Shouldn’t it say that Pearson correlation measures the linear relationship between variables? I would say that
monotonic suggests, a not necessarily linear, “increasing” or “decreasing” relationship.

REPLY 
Jason Brownlee August 21, 2018 at 6:23 am #

Right, Pearson is a linear relationship, nonparametric methods like Spearmans are monotonic relationships.
Start Machine Learning ×
Thanks, fixed.
You can master applied Machine Learning
without math or fancy degrees.
Find out how in this free and practical course.
REPLY 
Fredrik August 23, 2018 at 8:59 pm #
Email Address
No problem. Thank you for a great blog! It has introduced me to so many interesting and useful topics.

START MY EMAIL COURSE

REPLY 
Jason Brownlee August 24, 2018 at 6:07 am #

Happy to hear that!

Start Machine Learning

 REPLY 
Anthony The Koala August 22, 2018 at 2:47 am #

Two points/questions on testing for normality of data:

(1) In the Shapiro/Wilk, D’Agostino and Anderson/Darling tests, do you use all three to be sure that your data is likely to
be normally distributed? Or put it another way, what if only one or two of the three test indicate that the data may be
gaussian?

(2) What about using graphical means such as a histogram of the data – is it symmetrical? What about normal plots
https://www.itl.nist.gov/div898/handbook/eda/section3/normprpl.htm if the line is straight, then with the statistical tests
described in (1), you can assess that the data may well come from a gaussian distribution.

Thank you,
Anthony of Sydney

Start Machine Learning ×

You can master applied Machine Learning
REPLY 
Jason Brownlee August 22, 2018 at 6:15 am # without math or fancy degrees.
Find out how in this free and practical course.
More on what normality tests to use here (graphical and otherwise):
https://machinelearningmastery.com/a-gentle-introduction-to-normality-tests-in-python/
Email Address

START MY EMAIL COURSE

REPLY 
SEYE April 25, 2020 at 8:42 pm #

This is quite helpful, thanks Jason.

Jason Brownlee April 26, 2020 at 6:10 am # Start Machine Learning REPLY 
 You’re welcome.

REPLY 
Tej Yadav August 26, 2018 at 4:07 pm #

Wow.. this is what I was looking for. Ready made thing for ready reference.

Thanks for sharing Jason.

REPLY 
Jason Brownlee August 27, 2018 at 6:10 am #

I’m happy it helps! Start Machine Learning ×

You can master applied Machine Learning
without math or fancy degrees.
REPLY 
Nithin November 7, 2018 at 11:23 pm # Find out how in this free and practical course.

Thanks a lot, Jason! You’re the best. I’ve been scouring the internet for a piece on practical implementation of
Email Address
Inferential statistics in Machine Learning for some time now!
Lots of articles with the same theory stuff going over and over again but none like this.
START MY EMAIL COURSE

REPLY 
Jason Brownlee November 8, 2018 at 6:08 am #

Thanks, I’m glad it helped.

Start Machine Learning

 REPLY 
Nithin November 8, 2018 at 11:12 pm #

Hi Jason, Statsmodels is another module that has got lots to offer but very little info on how to go about
it on the web. The documentation is not as comprehensive either compared to scipy. Have you written anything
on Statsmodels ? A similar article would be of great help.

REPLY 
Jason Brownlee November 9, 2018 at 5:22 am #

Yes, I have many tutorials showing how to use statsmodels for time series:
https://machinelearningmastery.com/start-here/#timeseries

and statsmodels for general statistics:

Start Machine Learning ×
https://machinelearningmastery.com/start-here/#statistical_methods
You can master applied Machine Learning
without math or fancy degrees.
Find out how in this free and practical course.
REPLY 
Thomas March 29, 2019 at 10:02 pm #
Email Address
Hey Jason, thank you for your awesome blog. Gave me some good introductions into unfamiliar topics!

If your seeking for completeness on easy appliable hypothesis tests like those, I suggest to add the Kolmogorov-Smirnov
START MY EMAIL COURSE
test which is not that different from the Shapiro-Wilk.

– https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.ks_2samp.html
– https://www.researchgate.net/post/Whats_the_difference_between_Kolmogorov-Smirnov_test_and_Shapiro-Wilk_test

Start Machine Learning

 REPLY 
Jason Brownlee March 30, 2019 at 6:27 am #

Thanks for the suggestion Thomas.

REPLY 
Paresh April 16, 2019 at 5:17 pm #

Which methods fits for classification or regression data sets? Which statistical tests are good for Semi-
supervised/ un-supervised data sets?

Jason Brownlee April 17, 2019 at 6:55 am # Start Machine Learning ×

REPLY 

This post will help: You can master applied Machine Learning
without math or fancy degrees.
https://machinelearningmastery.com/statistical-significance-tests-for-comparing-machine-learning-algorithms/
Find out how in this free and practical course.

Email Address
REPLY 
Luc May 1, 2019 at 10:01 pm #

Hello, START MY EMAIL COURSE

Thank you very much for your blog !

I’m wondering how to check that “observations in each sample have the same variance” … Is there a test to check that ?

Start Machine Learning

 REPLY 
Jason Brownlee May 2, 2019 at 8:03 am #

Great question.

You can calculate the mean and standard deviation for each interval.

You can also plot the series and visually look for increasing variance.

REPLY 
João Antônio Martins June 2, 2019 at 4:39 am #

Is there a test similar to the friedman test? which has the same characteristics “whether the distributions of two
or more paired samples are equal or not”.
Start Machine Learning ×
You can master applied Machine Learning
without math or fancy degrees. REPLY 
Jason Brownlee June 2, 2019 at 6:45 am #
Find out how in this free and practical course.

Yes, the paired student’s t-test.

Email Address

START MY EMAIL COURSE REPLY 

MIAO June 27, 2019 at 3:37 pm #

HI, Jason, Thank you for your nice blog. I have one question. I have two samples with different size (one is 102,
the other is 2482), as well as the variances are different, which statistical hypothesis method is appropriate? Thank you.

Start Machine Learning

 REPLY 
Jason Brownlee June 28, 2019 at 5:57 am #

That is a very big difference.

The test depends on the nature of the question you’re trying to answer.

REPLY 
Adrian Olszewski February 27, 2020 at 11:32 pm #

Practically ALL assumptions and ALL interpretations are wrong in this cheatsheet. I cannot recommend this,
as if a student repeats that on a stat exam or on an interview led by a statistician, one’s likely to fail it. I am
messaged regularly by young aspiring data scientists who experienced problems after repeating texts from the
Start Machine Learning
internet, that’s why I ask you to not exposing learners to such situations. ×
1. Assumptions of the paired t-test are totally wrong, or copy-pasted. The interpretation is wrong too.
You can master applied Machine Learning
2. Anova is not a test, but OK, let’s pretend I didn’t see it. The interpretation isn’t correct. If you follow that, you may
without math or fancy degrees.
be really surprised doing the post-hoc
Find out how in this free and practical course.
3. interpretation of the RM-ANOVA is wrong
4. Mann-Whitney is described imprecisely.
Email Address
5. Paired Wilcoxon has wrong interpretation.
6. Normality tests – all is wrong. What “each sample” – in normality test? and it doesn’t tell if it’s Gaussian! It says of
the data is approximated by the normal distribution acceptably well at this sample
START size.
MY EMAIL In a minute I can give you
COURSE
examples drawn from log-normal or Weibull reported as “Gaussian” .

It’s worth noting there are over 270 tests, 50 in constant, everyday use, varying across industries and areas of
specialization. Type “100 statistical tests PDF” into Google or find the handbook of parametric and non-parametric
methods by Sheskin (also available in PDF), to get some rough idea about them. The more you know, the less you
are limited. Each of those tests has its weaknesses and strengthens you should know before the use. Always pay
attention to the null hypothesis and the assumptions. Jason Brownlee Start Machine Learning
 REPLY 
Jason Brownlee February 28, 2020 at 6:09 am #

Thanks for your feedback Adrian.

REPLY 
Mr.T March 1, 2020 at 9:08 am #

You sir, are patronizing.

I am an early stage learner of all of this, and Jason’s posts have been incredibly helpful in helping me construct a
semantic tree of all the knowledge pieces. Without a lot of his posts, my knowledge pieces would be scattered.
Start
I am not certain about the accuracy as you have pointed out, but Machine
your lack Learning
of constructiveness in your comment
×
is concerning. You do not provide what you believe is the correct interpretation.
You can master applied Machine Learning
I truly hate to see a comment like this. Keep up the good workwithout
Jason! math or fancy degrees.
Find out how in this free and practical course.

Email Address
REPLY 
Jason Brownlee March 2, 2020 at 6:10 am #

Thanks for your support! START MY EMAIL COURSE

REPLY 
MIAO June 28, 2019 at 5:50 pm #

Start Machine Learning

Thank you. Jason. The problem I process is that: I have results of two groups, 102 features for patient group and 2482
features for healthy group, and I would like to take a significant test for the features of two groups to test if the feature is
appropriate for differentiate the two groups. I am not sure which method is right for this case. Could you give me some
suggestions? Thank you.

REPLY 
Jason Brownlee June 29, 2019 at 6:37 am #

Sounds like you want a classification (discrimination) model, not a statistical test?

MIAO July 1, 2019 at 10:52 am #

Start Machine Learning ×
REPLY 

Yeah, I think you are right. I will use SVM to classify the features. Thank you.
You can master applied Machine Learning
without math or fancy degrees.
Find out how in this free and practical course.
REPLY 
Veetee August 6, 2019 at 1:04 am #
Email Address
Hi Jason, thanks for the very useful post. Is there a variant of Friedman’s test for only two sets of
measurements? I have an experiment in which two conditions were tested on the same people. I expect a semi-constant
START MY EMAIL COURSE
change between the two conditions, such that the ranks within blocks are expected to stay very similar.

REPLY 
Jason Brownlee August 6, 2019 at 6:40 am #

Yes: Wilcoxon Signed-Rank Test Start Machine Learning

 REPLY 
wishy September 6, 2019 at 10:09 pm #

Dear Sir,

I have one question if we take subset of the huge data,and according to the Central limit theorem the ‘samples averages
follow normal distribution’.So in that case is it should we consider Nonparametric Statistical Hypothesis Tests or
parametric Statistical Hypothesis Tests

REPLY 
Jason Brownlee September 7, 2019 at 5:29 am #

I don’t follow your question sorry, please you can restate Start
it? Machine Learning ×
Generally nonparametric stats use ranking instead of gaussians. You can master applied Machine Learning
without math or fancy degrees.
Find out how in this free and practical course.

REPLY 
gopal jamnal September 28, 2019 at 10:43 pm # Email Address

What is A-B testing, and how it can be useful in machine learning. Is it different then hypotheisis testing?
START MY EMAIL COURSE

REPLY 
Jason Brownlee September 29, 2019 at 6:12 am #

More on a/b testing:

https://en.wikipedia.org/wiki/A/B_testing
Start Machine Learning
 It is not related to machine learning.

Instead, in machine learning, we will evaluate the performance of different machine learning algorithms, and
compare the samples of performance estimates to see if the difference in performance between algorithms is
significant or not.

Does that help?

More here:
https://machinelearningmastery.com/statistical-significance-tests-for-comparing-machine-learning-algorithms/

REPLY 
Peiran November 14, 2019 at 8:57 am #

Start
You can’t imagine how happy I am to find a cheat sheet like this! Machine
Thank Learning
you for the links too. ×
You can master applied Machine Learning
without math or fancy degrees.
Jason Brownlee November 14, 2019 at 1:43 pm # Find out how in this free and practical course. REPLY 

Thanks, I’m happy it helps! Email Address

START MY EMAIL COURSE

REPLY 
Chris Winsor December 3, 2019 at 2:23 pm #

Hi Jason –

Thank you for helping to bring the theory of statistics to everyday application !

Start Machine Learning

I’m wishing you had included an example of a t-test for equivalence. This is slightly different from the standard t-test and
there are many applications – for example – demonstrating version 2.0 of the ml algorithm matches version 1.0. That is
actually super important for customers that don’t want to re-validate their instruments, or manufacturers that would need
to answer why/if those versions perform the same as one-another.

I observe a library at
http://www.statsmodels.org/0.9.0/generated/statsmodels.stats.weightstats.ttost_paired.html#statsmodels.stats.weightstat
s.ttost_paired
but it doesn’t explain how to establish reasonable low and high limits.

Anyway thank you for the examples !

Jason Brownlee December 4, 2019 at 5:28 am # Start Machine Learning ×

REPLY 

Great suggestion, thanks Chris! You can master applied Machine Learning
without math or fancy degrees.
Find out how in this free and practical course.

REPLY 
makis January 29, 2020 at 4:58 am #
Email Address
Hi Jason,

Great article. START MY EMAIL COURSE

If I want to compare the Gender across 2 groups, is chi-square test a good choice?
I want to test for signiicant differences similarly to a t-test for a numerical variable.

Start Machine Learning REPLY 

Jason Brownlee January 29, 2020 at 6:48 am #
 It depends on the data, perhaps explore whether it is appropriate with a prototype?

REPLY 
jessie June 29, 2020 at 7:10 pm #

Hi Jason,
I wanna use Nonparametric Statistical Hypothesis Tests to analysis ordinal data(good, fair, bed) or categorical data,
would i encode them to numerical data and follow the above steps? Would u give some suggestion?
Thanks.

Jason Brownlee June 30, 2020 at 6:21 am #

Start Machine Learning ×
REPLY 

Good question. No, I don’t think that would be correct.

You can master applied Machine Learning
Perhaps seek out a test specific for this type of data? without math or fancy degrees.
Find out how in this free and practical course.

Email Address
REPLY 
Jonathan August 23, 2020 at 8:43 am #

Repeated measures ANOVA can be performed in Python usingSTART

the Pingouin library
MY EMAIL https://pingouin-
COURSE
stats.org/generated/pingouin.rm_anova.html

REPLY 
Jason Brownlee August 23, 2020 at 1:16 pm #
Start Machine Learning
Thanks for sharing.
 REPLY 
Kenny August 31, 2020 at 7:17 pm #

Hi Jason,
Thanks for the very informative Article. It looks great to see all Hypothesis tests in one article.
1) Would you be able to help saying when to use Parametric Statistical Hypothesis Tests and when to use Non-
Parametric Statistical Hypothesis Tests,please?
Knowing what to use in given situations could be a lot helpful.
2) For doing A/B Testing with varying distributions in the 2 experiments under conditions of multiple features involved,
would you recommend Parametric Statistical Hypothesis Tests or Non-Parametric Statistical Hypothesis Tests?
( I have tried Parametric Statistical Hypothesis Tests but it was getting hard to meet the statistical significance, as there
are multiple features involved)
Start Machine Learning ×
You can master applied Machine Learning
without math or fancy degrees.
REPLY 
Jason Brownlee September 1, 2020 at 6:28 am # Find out how in this free and practical course.

Use a parametric test when your data is Gaussian and well behaved, use a non-parametric test otherwise.
Email Address
I don’t know about significance test for A/B testing off hand sorry. The sample distribution is discrete I would expect.
Perhaps a chi squared test would be appropriate? I’m shooting from the hip.
START MY EMAIL COURSE

REPLY 
MARCILIO DE OLIVEIRA MEIRA September 4, 2020 at 1:08 pm #

Hi Jason, make any sense using an statistical hypothesis tests for image classification, with machine learning?
What method is more suitable for a problem of image classification to determine if a image
Start Machine belong to a class A or class
Learning
B?

REPLY 
Jason Brownlee September 4, 2020 at 1:38 pm #

Not in this case, a machine learning model would perform this prediction for you.

REPLY 
Kenny September 21, 2020 at 4:59 pm #

Hi Jason,
Thanks for the article .Its quite informative.
Start Machine Learning ×
Say if the data for some reasons has a non-monotonic relationship between the variables, would Hypothesis testing be
of much help? You can master applied Machine Learning
Doesn’t it make sense to first check the prior belief by actually verifyingwithout
if the relationship is monotonous
math or fancy degrees. or not, before
doing any specific Hypothesis tests to get further statistical insights? Find out how in this free and practical course.

Email Address

REPLY 
Jason Brownlee September 22, 2020 at 6:43 am #
START MY EMAIL COURSE
It depends on the question you want to answer.

REPLY 
Hugo November 11, 2020 at 2:58 am #
Start Machine Learning
Hi Jason,
 Congratulations on the work you are doing with such subjects. It really helps me every time I need to get quick and
pŕecise content in this field.

I do have a question, though. About the stats.f_oneway module (ANOVA), I’m trying to run it with samples that have
different sizes, and that is returning an error “ValueError: arrays must all be same length”.

I tried to find the solution for this in the community, but I failed in finding it. Could you please help me out with this?
Should I input np.nan values to “fill” the empty spaces in the samples so they all match the same length?

Thanks in advance!

Best regards from Brazil.

Jason Brownlee November 11, 2020 at 6:52 am #

Start Machine Learning ×
REPLY 

Thanks! You can master applied Machine Learning

Perhaps a different test is more appropriate? without math or fancy degrees.
Find valid)?
Perhaps you can duplicate some samples (might make the result less out how in this free and practical course.

Perhaps you can find an alternate implementation?

Perhaps you can develop your own implementation from a textbook?
Email Address

I hope that gives you some ideas.

START MY EMAIL COURSE

Leave a Reply

Start Machine Learning

 Name (required)

Email (will not be published) (required)

Start Machine Learning ×

Website
You can master applied Machine Learning
without math or fancy degrees.
Find out how in this free and practical course.
SUBMIT COMMENT
Email Address

Welcome!
START MY EMAIL COURSE
I'm Jason Brownlee PhD
and I help developers get results with machine learning.
Read more

Never miss a tutorial: Start Machine Learning

Picked for you:

Statistics for Machine Learning (7-Day Mini-Course)

A Gentle Introduction to k-fold Cross-Validation

Start Machine Learning

How to Calculate Bootstrap Confidence Intervals For Machine Learning Results in Python ×
You can master applied Machine Learning
without math or fancy degrees.
A Gentle Introduction to Normality Tests in Python Find out how in this free and practical course.

Email Address

How to Calculate Correlation Between Variables in Python

START MY EMAIL COURSE

Loving the Tutorials?

The Statistics for Machine Learning EBook is

where you'll find the Really Good
Startstuff.
Machine Learning
 >> SEE WHAT'S INSIDE

© 2020 Machine Learning Mastery Pty. Ltd. All Rights Reserved.

Address: PO Box 206, Vermont Victoria 3133, Australia. | ACN: 626 223 336.
LinkedIn | Twitter | Facebook | Newsletter | RSS

Privacy | Disclaimer | Terms | Contact | Sitemap | Search

Start Machine Learning ×

You can master applied Machine Learning
without math or fancy degrees.
Find out how in this free and practical course.

Email Address

START MY EMAIL COURSE

Start Machine Learning