Understanding margin of error

What is margin of error?

Margin of error is information that’s provided alongside the results of a piece of research, such as a poll, survey or a scientific study. You’ll recognize it because it’s expressed with a plus and minus sign together, e.g. +-1%

Researchers use margin of error to provide additional information that helps you interpret their results and to understand how the study was carried out. The margin of error figure tells you that the true result may differ from the percentage figure provided, and how much more or less than the stated percentage the reality might be.

Margin of error provides a clearer understanding of what a survey’s estimate of a population characteristic means. A margin of error of plus or minus 2 percentage points means that if we ask this question using a simple random sample 100 times, 95 of those times it would come out at the estimated value plus or minus 2 points. The larger your random sample (the more respondents you interview), the smaller your margin of error will be, says Elizabeth Dean, XM Scientist.

When is margin of error used?

Margin of error is used when you have a random or probability sample. That means the sample has been selected at random from your population as a whole and every population member has a known, non-zero probability of being included.

It’s not appropriate if the sample has been selected in a non-random way, for example when you use an opt-in research panel.

“A research panel sample is typically a quota sample, where participants are selected because they have particular characteristics. Additionally, the respondents volunteer for the panel in return for benefits, so they are not randomly selected from the population at large,” explains Elizabeth.

So although margin of error is a popularly known term, it has a specific application in survey research and it won’t always be relevant to your market research data.

Here are a couple of scenarios where margin of error would apply.

• A sports team has a complete list of everyone who has purchased tickets to their games in the past year. If they randomly select a sample of that population for a survey, they can calculate the margin of error on the percent of people who reported being a fan of the team.
• An organization has a complete list of employees. They poll a simple random sample of these employees on whether they preferred an additional day of leave or a small bonus payment. They can report the margin of error on the percentage preferring each option.

Other kinds of error

Margin of error accounts for the level of confidence you have in your results, and the amount of sampling error you expect based on the size of the sample. But there are other kinds of survey  error that may influence your results too. These include coverage error, where your sampling frame doesn’t cover the population you are interested in, non-response error, which happens when certain respondents don’t take part in your survey, and measurement error, which can arise from problems with the questionnaire.

To learn about other sources of error, check out our guide to random and non-random sampling error.

How do you calculate margin of error?

Margin of error is calculated using a formula:

Z * √((p * (1 – p)) / n)

Where

Z* is the Z*-value for your selected confidence level, which you’ll look up in a table of Z scores (this one is from sjsu.edu)

p is the sample proportion

n is the sample size

The sample proportion is the number within the sample which has the characteristic you’re interested in. It’s a decimal number representing a percentage, so while you’re doing the calculation it’s expressed in hundredths. For a 5% sample proportion, it would be 0.05.

The most commonly-used confidence level is 95%, so we’ll use that for an example calculation. The Z*-value for a 95% confidence level is 1.96.

We’ll set our sample size at 1000.

Here’s how the calculation works.

1. Subtract p from 1. If p is 0.05, then 1-p = 0.95.
2. Multiply 1-p by p. So that’s 0.05 x 0.95 – which gives you 0.0475.
3. Divide the result (0.0475) by the sample size n. So 0.0475 divided by 1000 = 0.0000475.
4. Now we need the square root of that value, which is 0.0068920. This is the standard error.
5. Finally, we multiply that number by the Z*-value for our confidence interval, which is 1.96. So 0.0068920 x 1.96 = 0.0134395. That’s a margin of error of just over 1%.

Let’s try it with a real-world example.

Imagine you are a business surveying your current customers. You’ve run a study with a randomly selected sample of 1,000 people from your CRM list. The results tell you that of these 1,000 customers, 52% (520 people) are happy with their latest purchase, but 48% (480 people) are not – yikes. You want to add a margin of error to these results when you report them to your shareholders.

We’ll assume you want a 95% level of confidence, so the z*-value you’re working with is once again 1.96.

The number of customers who are happy with their latest purchase was 520, so that’s the number you’ll use to work out the sample proportion. 520 (p) / 1,000 (n) = 0.52

1. 1-p is 0.48
2. 0.52 (p) x 0.48 (1-p) = 0.2496
3. 0.2496 / 1,000 = 0.0002496
4. The square root of 0.0002496 = 0.0157987
5. 0.0157987 x 1.96 (the z*-value) = 0.0309654, or in other words, 3.1% (when you round it up).

You can now report with 95% confidence that 52% of your customers were happy with their latest purchase, + or – 3.1%

There are a couple of conditions for using this formula. They are:

1. n x p must equal 10 or more
2. n x (1-p) must equal 10 or more

Usually survey research involves quite high numbers of people in a sample, so unless you have a very small sample size, or the sample proportion within your sample is very small, there won’t be a problem. If you’re getting numbers below 10 for either of these checks, you may need to increase your sample size.