Here’s an excellent layman’s explanation of what it means when analysts or scientists say that something is statistically significant. This article offers detailed descriptions of the three components of significance: mean difference, variance and sample size.
Source: Harvard Business Review, February 16, 2016.
When you run an experiment or analyze data, you want to know if your findings are “significant.” But business relevance (i.e., practical significance) isn’t always the same thing as confidence that a result isn’t due purely to chance.
INSIGHTS: The main component of statistical significance is sample size, so when your analyses are being conducted on 100,000 customers or 1 million invoices, all differences will be statistically significant. They key to managing and interpreting complex analytics is determining when changing processes based on results can have real impact. Sometimes big differences, such as a 50 percent increase in sales on a SKU, are less impactful than a small difference, like a .005 percent conversion increase on thousands of orders a day. Understanding the downstream impact will determine which opportunities should be pursued.