CHAPTER 2: METHODOLOGY

2.1 This chapter provides details of the methodologies used to conduct the review.

2.2 The review comprised the following methodologies:

an online survey of 496 visitors to the website
10 usability interviews with members of the general public eligible for concessionary travel or affected by Transport Scotland projects, comprising 5 interviews with people who had previously used the site and 5 interviews with those who had not.
10 usability interviews with transport professionals, MSP (Members of the Scottish Parliament) researchers and representatives from transport special interest groups
10 depth interviews with key stakeholders; MSPs, journalists and senior employees from Scottish Government, Local Authorities and Transport Scotland
8 usability interviews with members of the general public with a disability
an accessibility audit of a sample of 20 pages from the website

Online survey

2.3 An online survey was placed on the Transport Scotland website from 8 July to 26 August 2008 to gather visitors’ views of the website. The survey invitation appeared to all visitors to the Transport Scotland homepage during the survey period. It informed them of the purpose of the survey and invited them to participate during their visit to the website.

Questionnaire design

2.4 The online questionnaire was designed by Ipsos MORI in close consultation with Transport Scotland and the Scottish Government. The survey content was piloted among members of staff at Transport Scotland who were familiar with the website.

2.5 The survey contained 23 questions covering:

use of the website
finding information on the website
satisfaction with the website design, content and usability
improvements to the website
respondent demographics

2.6 The full questionnaire can be found in Appendix One.

2.7 The majority of the survey questions took the form of ‘closed’ questions, in which respondents were invited to select their answer from a list of possible responses. Respondents were, however, given the chance to suggest improvements to the website in their own words.

2.8 Respondents were also asked whether they would be willing to participate in follow-up research to explore their views towards the site in more detail. Respondents meeting the eligibility criteria (described in paragraph 2.19 to 2.20) were then invited to participate in the usability testing.

Response to the survey

2.9 During the fieldwork period, there were 34,898 unique visitors² to the Transport Scotland website. Of these, 20,064 visited the homepage during their visit. The survey invitation was displayed the first time a visitor visited the homepage. A total of 496 website visitors participated in the online survey on the Transport Scotland website. Based on the number of visitors to the website during the survey period, this gives response rate of approximately 2.5%³.

2.10 In order to validate responses to the survey, measures were put in place to ensure that respondents could not complete the survey more than once. When the survey appeared, a cookie was placed on the respondent’s computer to prevent the survey from appearing again. This also served to ensure that frequent users of the site were not frustrated by the survey invitation popping up every time they used the site.

Analysis of the online survey data

2.11 Prior to the full analysis of the findings, data tables were produced using software that takes the raw data from the online survey and reorganises it into data tables that illustrate responses to each question overall and by pre-determined subgroups. The data tables were manually checked against the raw data file to ensure that they had been compiled accurately. Responses to each question were analysed against three key variables, namely:

sector of occupation (transport, public, other sector, retired, student, unemployed)
frequency of website use (first-time user, frequent user, infrequent user⁴)
information looked for during the website visit (news, road, rail, project, report and concessionary travel)

2.12 These analyses enabled us to identify any variation in the views and experiences of different sub-groups of users. It should be remembered at all times that the results in this report are only based on a sample of visitors to the website and not all visitors, so they are subject to sampling tolerances, meaning that not all differences will be statistically significant. A guide to statistical reliability is provided in Appendix Two. However, this assumes that the sampling was entirely random, which is not the case with this survey and so must be used as a rough guide only.

Usability testing

2.13 In order to explore views of the website in more detail, 20 one-on-one usability interviews were conducted between 6 August and 9 September 2008.

Profile of participants

2.14 In order to ensure that the overall profile of participants reflects the broad range of users of the Transport Scotland website, participants were recruited to meet a set of criteria agreed between Scottish Government, Transport Scotland and Ipsos MORI. The profile of participants recruited is shown in Table 2.1 below. The sample included a mix of respondents in terms of age, sex and frequency of use of the site.

Table 2.1: Usability testing participant profile

Stakeholder Group	Number of interviews
People covered by concessionary travel schemes	5
People affected by Transport Scotland’s work or projects	5
Researchers working for MSPs	2
Rail industry and road interest groups representatives	4
Transport / engineering consultants	4

2.15 The majority of those who took part in the usability interviews had used the website and had interesting thoughts on how to develop the site to best meet their needs. However, it is from non-users that the most insightful usability observations can often be found; an experienced user of a website may well be able to locate information quickly and easily, but that is not to say that the website is easy to use, simply that they have learnt by previous experience how to overcome usability barriers that exist. For this reason, the groups of people covered by concessionary travel schemes and those affected by Transport Scotland’s work or projects comprised both users of the website and non-users of the website. All participants in the usability tests had experience of using the internet.

Recruitment

2.16 Participants were recruited through a variety of methods. The methods used for each group are detailed below.

People covered by concessionary travel schemes

2.17 For the usability testing element of the project, people were recruited who were eligible for concessionary travel⁵ on account of their age (either older or younger people), rather than on account of a disability. Those with a disability were included under a separate part of the review (see section 2.41 – 2.45).

2.18 Non-users of the Transport Scotland website were recruited using face-to-face (on-street) recruitment in Glasgow and Edinburgh. A question in the recruitment questionnaire was used to determine whether or not they had previously visited the Transport Scotland website (along with other websites so that they would not know for whom the research was being conducted).

2.19 Current users of the website were recruited both face-to-face, in the same way described above for non-users, and by telephone among participants who had completed the online survey and indicated a willingness to be approached to participate in a further stage of the research.

People affected by Transport Scotland’s work or projects

2.20 Recruitment was conducted using face-to-face recruitment and by telephone, in the same way described above for people eligible for concessionary travel.

2.21 As part of the recruitment screener, people were asked whether they were affected by the Forth Replacement Crossing (Edinburgh participants only) and the Glasgow Airport Rail Link (Glasgow participants only).

2.22 Those eligible for concessionary travel and those affected by Transport Scotland’s work or projects who participated in a usability interview were compensated for their time and travel costs with a cash payment of £30.

Other Stakeholders

2.23 Recruitment of the other stakeholders (researchers working for MSPs, rail industry and road interest groups representatives and transport/engineering consultants) was conducted by telephone. Suitable individuals were selected using the following methods:

using existing contacts within transport and engineering consultancies
contacting MSP researchers working for MSPs who are on the Transport, Infrastructure and Climate Change Committee
liaison with Transport Scotland and the Scottish Government to identify individuals who would be particularly suitable to participate
internet searches to find other suitable journalists and MSPs

Structure of usability interviews

2.24 Interviews with people covered by concessionary travel schemes and people affected by Transport Scotland’s work or projects were conducted in viewing studios in Glasgow and Edinburgh⁶, giving the chance for members of staff at Transport Scotland and their website designers to observe the interviews. This is extremely useful as it allows those involved in the design of the website to see first-hand the issues that people have when looking for information on the website. These interviews were video recorded, with the permission of the participant. Participants were informed during recruitment that interviews would be observed and video recorded. Participant personal data was stored securely and not made available to employees of Transport Scotland or the Scottish Government, nor anyone else. Participants were informed of the purpose of the research, how their data would be used and that any personal information they supplied would remain confidential.

2.25 In order to make it more convenient for them to take part, interviews with participants from the other stakeholder groups took place at their offices or at the Ipsos MORI Edinburgh office. Therefore, it was not possible for the Transport Scotland team to observe these.

2.26 Each interview lasted between 45 minutes and one hour and involved a participant sitting at a computer with the moderator next to them. The participant was in control of the computer and was guided through the interview by the moderator. The website was explored using both free browsing and completion of a set of tasks, with the participant asked to vocalise their thoughts as they went through the website.

2.27 A discussion guide was designed by Ipsos MORI, in conjunction with Transport Scotland and the Scottish Government. This covered:

general use of the internet
initial exploration of the website
completion of a number of tasks (up to eight from a list of 14)
detailed exploration of views of the main pages of the website covering content, navigation, design, language and terminology
review of the Highways Agency, Scottish Government and Department for Transport websites and comparisons with the Transport Scotland website
interest in new ways of presenting content on the website
suggestions for improvement

2.28 The full discussion guide, including the list of tasks, can be found in Appendix Three. A review of how successful users were in completing the tasks can be found in Appendix Four.

Depth interviews with key stakeholders

2.29 In addition to the usability interviews, 10 depth interviews were conducted with other key stakeholders between 28 July and 12 November 2008. These interviews focused on the role that the website plays in information dissemination and whether the design and content reflects the aims and the status of Transport Scotland.

Profile of participants

2.30 Table 2.2 shows the profile of the respondents who participated in the stakeholder depth interviews.

Table 2.2: Key stakeholder participant profile

Stakeholder Group	Number of interviews
Journalists reporting on Scottish transport issues	2
Senior employees in Transport Scotland	2
Senior employees in the Scottish Government	2
Senior employees in local authorities	2
MSPs who are members of the Transport, Infrastructure and Climate Change committee	2

2.31 All participants were recruited by telephone. Suitable journalists and MSPs were found using internet searches. Transport Scotland provided guidance about the most appropriate individuals to interview within Transport Scotland, the Scottish Government and local authorities.

Structure of the depth interview

2.32 The interviews lasted between 40 minutes and one hour and were conducted in the respondent’s office or at the office of Ipsos MORI in Edinburgh.

2.33 A discussion guide was designed by Ipsos MORI, in conjunction with Transport Scotland and the Scottish Government. The guide covered stakeholders’ use of the Transport Scotland website, their perceptions of the design and navigation of the site, whether the content met their information needs and any improvements that they would like to see made to the website. A copy of the discussion guide can be found in Appendix Five. The guide was designed to be flexible in order to cover the issues that were most relevant to the particular interviewee. The interviewer took detailed notes throughout all of the interviews, which were used for analysis.

Analysis of the usability and depth interviews

2.34 Analysis of the results of the usability and depth interviews involved identifying common themes that cross the different pages of the website (such as feedback on overall design and content issues), as well as on a page by page basis, based upon notes made by the interviewer, observers and a video review of key parts of interviews. Throughout the report, we will comment on recommendations that will improve the experience of visitors to the Transport Scotland website. A full list of recommendations, listed by section, in priority order, is provided in Chapter Eleven of this report. Quotes from the usability and depth interviews are used extensively throughout this report to illustrate key issues that are reported.

Accessibility testing

2.35 In order to establish whether the current Transport Scotland website meets AA accessibility standards⁷ and to identify any problems disabled people experience using the website, Ipsos MORI worked with AbilityNet⁸, a national charity that helps disabled adults and children to use computers and the internet.

2.36 There were two parts to the accessibility testing: an audit of the website to ensure that it is compliant to Web Content Accessibility Guidelines (WCAG) checkpoint AA⁹ and moderated usability testing sessions of the Transport Scotland website among eight disabled users.

Accessibility audit

2.37 Experienced AbilityNet auditors tested the accessibility of the Transport Scotland website between 24 and 28 July 2008. In order to cover a range of page types across different sections of the website, twenty pages were selected by Ipsos MORI and Transport Scotland for testing. These pages represented the different ways in which content is displayed upon the website and largely focused on the more complex pages, including those with interactive tools, tables and graphics. Auditing the more complex pages of a website ensures that all accessibility issues are identified as some might not be observed if text-only pages are selected.

2.38 The audit involved testing each of the twenty pages selected for the audit against the 46 WCAG Priority One and Priority Two checkpoints. Whilst some of these checks could be automated, the majority of checkpoints required manual evaluation. As an example, while the presence of ALT tags¹⁰ could be checked for automatically, their appropriateness required manual checking in every instance.

2.39 The main tool used to assist in the evaluation process was the web accessibility toolbar¹¹ which gives access to a range of features to facilitate accessibility testing. These features include the colour contrast checker, quick links to the WC3 HTML Validation page and shortcuts to disable images, Javascript and CSS within the browser (which replicates how disabled users accessing with assistive technologies will view the page). In addition, direct access to the underlying page code was required to check other issues and was accessed via the ‘View Source’ menu option in Internet Explorer.

2.40 The findings of the audit are presented in a separate document, which can be found in Appendix Six.

Disabled user testing

2.41 Moderated disabled user testing was undertaken by AbilityNet, in London, on 5 and 6 August 2008, with eight people with a disability.

Profile of participants

2.42 All eight participants were members of AbilityNet’s panel of disabled website users. None had used the Transport Scotland website previously. Table 2.3 below shows the participants’ disabilities and the access methods or assistive technologies used by each.

Table 2.3: Profile of disabled participants

Disability	Access method/assistive technology
Severe vision impaired	Screen reader
Medium vision impaired	Magnification software
Mild vision impaired	Large text/high contrast
Severe motor difficulties	Voice recognition
Motor difficulties and cognitive impairment	No special access methodologies
Mild motor difficulties	Fine mouse control difficulties
Medium dyslexic	No special access methodologies
Learning disability	No special access methodologies

Structure of the tests

2.43 The disabled user testing was undertaken in broadly the same manner as the usability testing described above, though each session lasted slightly longer at between one hour and 90 minutes. The disabled user testing comprised solely of the moderator asking participants to complete from the same range of tasks as used in the usability sessions. On average, users completed 7 of the 14 tasks. As with the general public and transport professionals’ usability testing, the participant sat at the computer with a moderator beside them. However, the moderator remained passive throughout the session, with the user in control of the computer at all times. The user was asked to comment on their actions throughout the testing session.

Analysis of the disabled user test results

2.44 Disabled users, while identifying usability issues relating to their disability, also identified standard usability issues. Where an issue was identified both by users with and without a disability, no distinction has been drawn out in this report. Where an issue was identified only by one or more disabled users, specific reference to this fact has been made.

2.45 As well as a summary of reaction to the website’s homepage, design, layout and navigation, each of the tasks was analysed at an individual participant level, since a task that is easy for a user with one impairment might be much harder for someone with a different impairment. The task analysis focused on identifying the positive and negative experiences the user had in completing the task, the success rate across all users in completing it and recommendations for the design of the site to minimise barriers and improve accessibility. Findings from the disabled user tests have been incorporated into this report.

Presentation and interpretation of the findings

Online survey findings

2.46 A pop-up survey on the Transport Scotland website was adopted as it is the most effective method by which to reach a broad range of current users of the website. However, in all quantitative surveys there exists the potential for bias because of differences in profile, attitude or behaviour among those who respond and those who do not. With this particular survey, where there was little control over who responded to the survey and a low response rate, the likelihood of non-response bias is high. For example, visitors with a keen interest in the website, visitors with a lot of time to spare or those with a particular grievance about the website are perhaps more likely to take part than other users. Therefore, the results should be seen as indicative, rather than statistically representative, of the whole population of visitors to the site.

2.47 Where percentages reported on do not sum to 100%, this may be due to computer rounding, the exclusion of ‘don’t know’ categories or because multiple answers were possible.

Qualitative research findings

2.48 The usability interviews, depth interviews and the accessibility interviews used qualitative methods. Qualitative research is often compared and contrasted with quantitative research. Qualitative research is concerned with understanding opinions and experiences ("why?", "how?" etc) rather than measurement ("how many?", "how often?" etc). The qualitative component of the review explored attitudes to the website in much more depth then the quantitative research. The aim of qualitative research is not to generalise to the wider population in terms of the prevalence of attitudes (e.g. ‘20% of Transport Scotland website users do not like the colours used on the site), but to identify and explore issues and themes relating to the subject being researched, in depth.

A note on reporting usability research

2.49 Usability research has more in common with ethnographic interviewing than traditional qualitative research, in that much of the learning comes from observation, rather than verbal feedback. Observation of task completion, coupled with verbal feedback and supplemented by the experience of the usability moderator are what form the basis for recommendations for website improvement.

A note on the audience definitions used in this report

2.50 Throughout this report, we have used the following terms to define the different audiences contacted during the review. These are summarised here.

‘Users’ / ‘Stakeholders’: General terms to describe all Transport Scotland website users.

‘Respondents to the online survey’ / ‘Respondents’: The 496 visitors to the Transport Scotland website (or a sub-set of the 496) who completed the online survey.

‘Those who participated in the usability sessions’: The 20 participants who took part in the usability tests, comprising members of the general public, those working in transport consultancies / engineering firms and researchers working for MSPs.

‘Members of the general public’: 10 members of the general public who participated in the usability tests.

‘Transport professionals’: Those working in transport consultancies, engineering firms and local authority transport departments.

‘Disabled users’ / ‘Users with an impairment’: The eight disabled users who participated in the usability testing

‘Key stakeholders’: Senior personnel at Transport Scotland, the Scottish Government, MSPs and journalists