Calculate the Chi Square Statistic
|Degrees of Freedom:|
|Chi Square Statistic:|
What is a Chi-Square Statistic?
A Chi-Square (χ2) statistic is a measurement of distance between expected and actual counts for categorical data.
In A/B/n testing terms, your visitors are categorized in two ways:
Conversion Status: Converted vs. Not Converted Recipe Applied: Control vs. Variant 1 vs. Variant 2, etc.
We use the Chi-Square Testing Calculator to determine if the recipe applied influences conversion status by measuring the distance of the actual counts from those that we would expect if they were not related. Specifically, it tests whether each recipe affects conversion differently than the each of the others.
My Test is Significant. Now What?
A significant Chi-Square test tells you that a statistically significant difference exists, but not which difference or differences caused significance. The next step after identifying a significant Chi-Square test is to test the difference between all pairs of treatments (e.g. Variant 1 vs. Control) using our Difference of Proportions Calculator.
Even if you can see visually what is likely to be a significant difference, it is important to run all tests in order to make sure no statistically significant differences are missed.
What does my confidence level mean to me in a business sense?
If you are 95% confident that at least one recipe is different than another, there is a one in 20 chance that the patterns are all the same. If, after doing your pair-wise tests, you roll out the statistically best recipe, there is only a 1 in 20 chance that you will not see that lift.