Statistical significance in a survey tells you whether a difference you see in the results is likely to be real or could plausibly be down to random chance. A result is called statistically significant when the difference is large enough, relative to your sample size, that random sampling alone is an unlikely explanation.

What statistical significance means

When you survey a sample rather than everyone, every number comes with some wobble. If Segment A scores 60% and Segment B scores 55%, the 5-point gap might reflect a true difference or it might just be sampling noise. Statistical significance is the standard for deciding which. It does not tell you the difference is large or important, only that it is probably not an accident of who happened to answer.

P-values and confidence in plain language

  • A p-value is the probability of seeing a difference at least as big as the one you observed if there were really no difference at all. A small p-value means "this gap would be surprising if nothing were going on."
  • The common threshold is p < 0.05, which corresponds to 95% confidence. It means there is less than a 5% chance the difference is purely random.
  • Confidence intervals express the same idea as a range: "the true value is 55% ± 4%." When two groups' intervals barely overlap or do not overlap, the difference is more likely real.

So "significant at 95% confidence" and "p < 0.05" describe the same bar from two angles.

Worked example

You compare conversion intent between two regions. Region A: 62% of 400 respondents. Region B: 55% of 380 respondents. The 7-point gap, tested against those sample sizes, comes back with p ≈ 0.04 — below 0.05 — so it is statistically significant at 95% confidence. Now imagine the same 7-point gap but with only 40 respondents per region: the same difference would likely return p > 0.20 and would not be significant, because small samples are noisy. The lesson: the same gap can be real or noise depending entirely on sample size.

Sample-size caveats

  • Small samples hide real effects and inflate fluke ones. A subgroup of 15 people can swing 10 points from a couple of answers. Treat very small groups as directional only.
  • Significant is not the same as important. With a huge sample, a trivial 1-point difference can be "significant" yet meaningless in practice.
  • The more comparisons you make, the more false positives you get. Slice your data into many segments and some will look significant by chance alone, so be skeptical of a single standout in a sea of comparisons.

How to read segment differences responsibly in SegmentOS

SegmentOS is built to keep you honest about this. The notable differences panel only surfaces meaningful gaps (it uses thresholds such as roughly 10 percentage points for choice questions) and applies low-sample guards: it labels a segment as an early signal when it has fewer than ~20 responses and tells you when a segment is simply too thin to analyze. When you explore the Segment & Cross-Tabulate view, check each segment's displayed size before trusting a difference, and make sure your sample size is large enough for the comparisons you want to make. When in doubt, collect more responses before drawing conclusions.

Frequently asked questions

What does "statistically significant" actually mean?

It means the difference you observed is unlikely to be explained by random sampling chance alone, given your sample size. It does not say the difference is large or practically important — only that it is probably not an accident.

What is a good p-value for surveys?

The conventional cutoff is p < 0.05, equivalent to 95% confidence. Some studies use a stricter 0.01 (99% confidence) or a looser 0.10 (90% confidence) depending on the stakes; pick a threshold before you analyze, not after.

Can a difference be significant but not meaningful?

Yes. With a very large sample, even a tiny difference can be statistically significant while being too small to act on. Always weigh significance together with the size of the difference and its business relevance.

Why do small segments give unreliable results?

Smaller samples carry more random variation, so a few individual answers can produce big swings. Differences within small segments are often not significant, which is why SegmentOS flags low-sample segments as early signals and you should treat groups under about 30 responses as directional.