APPENDIX 2: GUIDE TO STATISTICAL RELIABILITY
APPENDIX 2: GUIDE TO STATISTICAL RELIABILITY
1 As noted previously in the main report, the sample for the online survey was self-selecting and the response rate was low. These are both factors which make it likely that there is a high non-response bias. It is impossible to know the impact of this bias.
2. Because the possibility for bias exists, the figures obtained in the survey are estimates rather than the ‘true’ values that would have been obtained if every visitor to the website has completed the survey. However, the variation between the sample estimates and the ‘true’ values can be predicted based on the size of the samples on which the results are based and the number of times that a particular answer is given.
3. The confidence with which researchers generally wish to make estimates is 95% - that is, they wish to be sure that there is only a 5% chance that the estimate has been obtained by chance. Strictly speaking, statistical theory cannot be applied to the online survey due to the self-selecting nature of the sample. This sample is a non-random sample as it is dependent upon the respondent having replied to the survey invitation on the website. However, as a very rough guide, the table below illustrates the predicted ranges for different sample sizes and percentage results at the ‘95% confidence interval’, for a randomly selected sample.
Table A1: Predicted ranges for different sample sizes at the 95% confidence interval, assuming a simple random survey
Size of sample on which survey result is based |
Approximate sampling tolerances applicable to percentages at or near these levels |
||
|---|---|---|---|
10% or 90% |
30% or 70% |
50% |
|
100 |
6 |
9 |
10 |
200 |
4 |
6 |
7 |
300 |
3 |
5 |
6 |
496 (the sample for the online survey) |
3 |
4 |
4 |
1000 |
2 |
3 |
3 |
Source: Ipsos MORI
4. For example, on a question where 50% of the people in a sample of 496 respond with a particular answer, the chances are 95 in 100 that this result would not vary by more than four percentage points above and four percentage points below the value that would be achieved from a complete coverage of the entire population using the same procedures. However, while it is true to conclude that the ’actual‘ result (95 times out of 100) lies anywhere between 46% and 54%, it is more likely to be closer to the centre of this band’ (i.e. at 50%).
5. Statistical reliability should also be considered when comparing results from different parts of the sample, e.g. frequency of visit to the website. Differences in results between groups have only been commented on where they are of a level to suggest that there is a real difference in opinions or experiences. A difference, in other words, must be of at least a certain size to be considered statistically significant. Table A2 is a very rough guide to the sampling tolerances applicable to this study, for a randomly selected sample.
Table A2: Sampling tolerances
Size of samples compared |
Differences required for significance at or near percentage levels |
||
|---|---|---|---|
10% or 90% |
30% or 70% |
50% |
|
100 and 100 |
8 |
13 |
14 |
200 and 200 |
6 |
9 |
10 |
93 and 231 (frequent and infrequent visitors to the website) |
7 |
11 |
12 |
163 and 324 (first-time visitors to the site and those who had used the site before) |
6 |
9 |
9 |
496 and 97 (all respondents and those working in the transport sector |
5 |
8 |
9 |
Source: Ipsos MORI