APPENDIX 4: TASK REVIEW

APPENDIX 4: TASK REVIEW

1. For the usability testing among members of the general public, disabled users and some stakeholder groups, fourteen tasks were developed for users to complete by using the website. The tasks varied in complexity, but all involved locating content or using functions on the website. Around half of the general public users were given time to freely browse the website before undertaking the tasks, while the remainder were asked to complete them immediately, from the homepage. Those from other stakeholder groups completed the tasks immediately. Users were presented with the tasks randomly, rather than in order from one through to fourteen and not all users were presented with all tasks. Some of the tasks were purposefully left rather vague, in order that users were not provided with too much guidance about where they could expect to find certain content on the website.

2. A short summary of users’ experiences while completing the tasks are presented in this appendix.

Task One – Find the environmental statement for the A90 Balmedie to Tipperty trunk road project and its completion date

3. Users struggled to complete this task. Many users expected to find this information under the ‘Road’ link in the primary navigation menu, which they came to before the ‘Project’ link, when reading from left to right. Users then tended to scan the secondary navigation menu for a link to project information and usually clicked ‘Motorway and Trunk Road Programme’, which was considered to be the most relevant link listed. None of the users opened the programme document as they did not think they had reached the correct place to find what they were looking for.

4. A minority of users did find the ‘Projects’ link, either immediately, or after first trying the ‘Road’ link. All were eventually able to locate and operate the menu which provided access to all road projects. The listing by road number was considered to be appropriate by all users, although a number expected the Motorway projects to be listed before the Trunk Roads. Some expected to move onwards by simply selecting ‘A90 Balmedie to Tipperty Dualling Project’ from the list and did not expect to have to click the ‘Go’ button. Eventually, all did this, however.

5. Users scanned the project information without reading much of the detail. As a consequence, some people missed the link to the environmental statement and the link to the programme document, which contained the completion cost information. Eventually all found these links.

6. A small number of users had to resort to using the website search tool to locate the project. Most searched on "A90 Balmedie to Tipperty" and the search returned the project’s summary page at the top of the listings. One user, however, typed "Balmede (sic) to Tipperty" and received the search error page. As the user had not recognised the spelling mistake, she was unable to progress without moderator assistance.

Task Two – Find a map of the M74 project, find out the cost of the project and look for details on the archaeological dig taking place

7. Once users found the ‘Projects’ page, all clicked on the image of the M74 to access more specific details. Users tended to skim the content on the page for the information they were looking for. At this point, a number noticed the tertiary menu which linked to the M74 Dig information, although the frame width meant that users had to scroll across to the right-hand side to see this. Users expected the information on the summary page to provide the project cost information and were surprised that it was not listed here.

8. Most users did find the secondary navigation link to ‘The Project’, which provided the cost information and noticed the link within that menu to the map.

Task Three – Find the announcement by Ministers in the second half of 2007 announcing the preferred option to replace the Forth Road Bridge

9. This task was successfully completed by most users. All but two used the ‘News’ link from the primary navigation menu and searched the archive by using the dropdown menu to select ‘2007’ and ‘July’. Users simply searched each month between July and December, before locating the correct news release.

10. Two users tried to access this information using the search tool, but were unable to find the relevant news article in this way.

Task Four – Find out what stations will be located on the Glasgow Airport Rail link route and find the timetable for the project

11. A number of users initially selected the ‘Rail’ link, rather than ‘Projects’ to find this information. However, once users had reached the ‘Projects’ landing page, all clicked the image to proceed to the project’s summary page. Some found the route information by scanning through the text on this page, while others selected ‘The Route’ link in the secondary navigation menu and used the map to identify stations on the route.

12. Accessing information about the project timetable caused more problems. Eventually most did locate the information on the project’s pages, but spent considerable time doing so.

Task Five – Find the details of the national concessionary travel scheme as it applies to disabled travellers

13. This task was completed successfully by all users. All users clicked on the ‘Concessionary Travel’ link from the primary navigation menu and then the ‘Who Qualifies?’ link in the secondary menu. This led to the required information.

Task Six – Find information on the review of transport projects currently being undertaken by Transport Scotland

14. This task was completed easily by users. After understanding what they were being asked to look for, all users expected to find this information on the ‘Projects’ page and navigated successfully to this using the primary navigation menu. While some users expected the review to be located at the top of the page – providing an overview – all successfully found it at the bottom of the page.

15. Given the recommendation that the project link be removed from the primary navigation bar (see Recommendation section), it is important to consider where best to place the ‘Strategic Transport Project Review’ information to enable users to access it. We would recommend that it be placed in the ‘Reports’ section, although links to it could also be placed from the ‘Road’ and ‘Rail’ project listings pages.

Task Seven – Find the latest Scottish Transport Appraisal Guidance and information relating to the evaluation of an option as part of the post-appraisal

16. This task proved difficult for all members of the general public and some transport professionals, as they had no understanding of what the document was they were looking for. Some searched ‘Scottish Transport Appraisal Guidance’ in the search box and the search returned the relevant link at the top of the page of results. All those using the search successfully clicked through to the summary page. Some then clicked on ‘The Guidance’ link in the body text to access the main document and searched the contents page for the term ‘post-appraisal’.

17. Among those transport professionals who were aware of the document, most located it by clicking into the ‘Reports’ section of the website and clicking the Scot-TAG link in the secondary navigation menu. All of these users clicked the ‘Scottish Transport Appraisal Guidance’ link into the document and successfully navigated to the evaluation of an option in the post-appraisal chapter.

18. All users were prompted to discuss where this document should be located. All agreed that the reports section was the most appropriate, although many expected the reports to be listed on the main page, rather than via links in the secondary navigation menu.

Task Eight – Find volume of traffic information on the M8 for a location of your choosing

19. Overall, users experienced considerable difficulty when completing this task. All users were able to navigate to the application that provides traffic information by clicking on the ‘Road’ link on the primary navigation menu and then on the ‘Traffic Count’ link on the secondary navigation menu. However, not all users noticed the ‘map application’ link placed within the text on the webpage which leads to the required tool.

20. Most users instead clicked on the map on the page expecting to be able to find traffic information. When asked why they had clicked on this, users explained that because it was placed prominently on the webpage their attention was drawn to it. Some users also noted that the map already provided information on the average daily flow of traffic and thus expected to be able to find more detailed information on the M8 specifically. Users were confused when the new window opened with a map which provides the same information as the link they had just clicked on. Placing the link to the application in a more prominent position would make it more visible to users.

21. The average daily flow map presented additional problems for users with visual impairments. The use of colour to distinguish the different traffic flow information means those with colour blindness could not make out differences. Distinguishing between the flow rates using lines of different widths as well as different colours would improve this.

"This is assuming that everyone can distinguish colour."
Vision impaired user, using magnification software

22. Subsequently, users tended to skim read the text on the webpage and some missed the ‘map application’ link placed at the bottom of the text. These users then resorted to using the website’s search engine and typed in phrases which mostly included the words "M8" and "traffic information". No user managed to successfully find the map application using the search engine. Eventually all users were able to locate the application by re-reading the text on the webpage. One user noted that it is necessary to scroll down the webpage in order to see the link and felt that was inappropriate as it is the most important piece of information on the whole page.

23. Once users found the link, further problems were identified. One transport professional was unable to load the map as she was using Internet Explorer 7. This was particularly frustrating as she had required similar information in the past and had to install Mozilla Firefox, an alternative website browser, specifically to access this application.

24. Another user also noted that the physical information on the map such as mountains, glens and lochs made the map look unnecessarily complicated. Removing these details would simplify the map.

"All the background clutter [on the map] is hard to make out. You could do without that."
Member of the general public, previous user of the website

25. In terms of the usability of the map application itself, many users felt that it was not particularly obvious what the red dots (which represent data points) scattered around the map show and therefore many failed to click on these. This problem was compounded when some users tried to click on these dots and nothing happened. The red dots themselves will need to be more sensitive and allow for clicking on their edges, rather than in the centre of them.

"It is not that intuitive. I guessed that it would be under ‘Traffic Count’. When you get to the map, the search takes some playing with."
Transport / Engineering Consultant

26. A few users tried using the search engine on the application and typed in "M8". No results were displayed, which caused frustration. Thus all users looked for the M8 manually. Users were able to click on the map and move to the area where the M8 is, although some users initially clicked on the zoom in and out buttons. When they had found the M8, users clicked on the red dots. One user was pleased that he was able to export this information to an Excel document. Tagging the map application appropriately will enable those who use the search engine to find it. We would suggest using the phrases such as "traffic information", "traffic flow information" and "traffic count".

27. Clear instructions should also be provided on how to operate the tool, particularly covering how to navigate around the map, zoom in or out and operate the search facility. They should also state what the red dots represent and that they are clickable.

Task Nine – Find information about the planned improvements to the Edinburgh to Glasgow railway line

28. This task proved very difficult for users, even if they knew or had learnt the structure of the website. All users assumed this information would be included in the ‘Project’ information, not being aware of the distinction between on-going projects and consultations, which are published under the ‘Reports’ link.

"The logical place would be under projects. Improvements would be project-based."
Member of the general public, previous user of the website

29. All users navigated to the ‘Projects’ section of the website and from there onto the list of rail projects. Once the information was not located, most users were unsure about how to proceed. Instead, most users continued to look in the ‘Rail’ section and searched without success. Only a small number of stakeholders who had used the website before looked at the ‘Reports’ section.

"This sounds like it is something planned, but not yet underway. I am not sure whether many people would understand that."
Member of the general public, previous user of the website

30. Even for these users, few were sure whether the document would be archived under ‘Consultation Papers and Responses’ or ‘Publications and Guidance’. Users resorted to clicking on one or the other, and searching to find it. Users who reached the report section located the information, with all navigating to the rail reports.

Task Ten – Find Transport Scotland’s Freedom of Information Publication Scheme

31. Most users completed this task easily, locating the ‘Freedom of Information’ link in the organisation menu. However, some users, particularly with those with a visual impairment, found the grey font on white background to be difficult to make out. Once a user had clicked the link, all were able to proceed to the ‘Publication Scheme’ by clicking the link at the top of the list.

32. A minority of users spent a considerable amount of time looking for the ‘Freedom of Information’ link, as their attention was drawn to the content of the website below the ‘Transport Scotland’ banner heading.

Task Eleven – Find what jobs are published on the website

33. Users expected to find this information via a menu at the top or the bottom of the website, along with other organisational information.

34. As a result, all users were able to find the ‘Careers’ link. Many immediately noticed the ‘Current Vacancies’ link in the secondary navigation menu and were able to access the available jobs. However, others did not notice this menu, because the links are aligned with the image at the top of the page, not with the content.

"The graduate training scheme stands out, but the vacancies are quite hidden. It should have a click here for vacancies to make it more prominent."
Severe vision impaired user, using screen reading software

35. Part way through the usability testing, the two vacancies that had been displayed were removed. Users struggled to identify this was the case and expected some explanation that no vacancies were currently being advertised.

"It is not immediately clear that there are no vacancies. There is nothing to tell you that."
Rail industry interest group representative

Task Twelve – Find statistics showing First ScotRail’s most recent performance on the Edinburgh to Glasgow railway line

36. This task proved very difficult to complete successfully. Most users expected to find this information in the ‘Rail’ section of the website and navigated to it by using the primary menu. Once there, users were unsure which link from the secondary menu would lead them to the required information.

37. Some users clicked on the ‘Rail Franchise’ link as they were aware that First Scotrail hold the rail franchise agreement and subsequently clicked on the ‘First Scotrail’ link that appears. They were surprised that the information they were asked to look for was not under this section of the website.

38. Users also clicked on the ‘Improving Railways’ link on the secondary menu from the ‘Rail’ landing page. When asked why they did so, these users stated that this was the most relevant link as monitoring performance is an aspect of improving the railways. From there, users skim read the page and clicked on the subsequent link provided. Again, users were surprised that the information was not there.

39. Similar to other tasks, a number of users resorted to utilising the search engine on the website. Each user searched on a combination of the words the task was put into with "statistics", "Scotrail" and "performance" being the most popular ones. The search engine failed to provide users with an appropriate link.

40. Eventually, users clicked on ‘Service Quality Incentive Regime’, either as a last resort or because they were prompted by the moderator. All general public users and several transport professionals failed to understand what the term meant. Using a link such as ‘Rail franchise performance statistics’ will provide better guidance to users of what to expect.

41. Users skim read the page and usually clicked on one of the two maps or selected the ‘your line of route’ link at the end of the text. Those who clicked on the map were unable to proceed and had to return. A minority then clicked the ‘Results for your line’ link in the secondary navigation menu.

"I was not sure what I was looking for. The term service quality incentive regime meant nothing. Something like train performance statistics would be better."
Transport / Engineering Consultant

42. When faced with the ‘Station’ and ‘Line’ drop down boxes, some users went directly to the latter one expecting to be able to select the Edinburgh to Glasgow line. As there was no content in this box, these users were confused and a few assumed that the application was faulty. All eventually selected either ‘Edinburgh Waverley’ or ‘Glasgow Queen St HL’ from the drop-down list in the ‘Station’ box. Further confusion was created as the page refreshed itself without displaying the required information or without an action having been taken.

43. A few users managed to select the correct train route from the ‘Line’ box though the rest had to be prompted by the moderator. Expectations were again not met as the page refreshed itself and a new ‘Train Class’ box appeared. No users knew the class of the Edinburgh to Glasgow train but since there is only one option, all could proceed. However, no users clicked the ‘View Report’ button for either the station or the train line – all did so for the train class.

"That was really hard. Doing the class, the route, the line. I assumed you did one or the other, route or station, and class came up"
Member of the general public, non-user of the website

44. The information again failed to meet expectations. Users were expecting some charts and graphs charting service performance, but this information could only be accessed by clicking a further link in the first paragraph of text. This link did not convey its importance.

45. The way users are expected to search for a particular line is not intuitive and it requires too many steps. Users should be asked to input all the criteria they want to search on immediately (such as station, line or train class) and then proceed with the search. Once the search has been made, links to the performance charts and statistics need to be made clearer, such as ‘Performance statistics for Glasgow Queen Street High Level Station’ or ‘Performance statistics for Glasgow to Edinburgh via Falkirk line’.

Task Thirteen – Find information on the winter maintenance service carried out on Scotland’s trunk roads

46. Generally, users completed this task easily. All users clicked into the ‘Roads’ section of the website. Most then skimmed the main text on the page, before looking through the list of links in the secondary navigation menu.

47. While most identified the correct link here and clicked through to the relevant information, a small number demonstrated that they were not really reading the text at all and missed the link itself. These users selected a range of other links which they considered might be appropriate, including ‘Climate Change & the Roads’ and ‘Motorway and Trunk Road Programme’.

Task Fourteen – Subscribe to the Forth Replacement Crossing newsletter

48. All users who were asked to complete this task were able to do so without experiencing any major problems. These users navigated to the ‘Project’ landing page and clicked on the ‘Forth Replacement Crossing’ link. Once on the webpage for this project, all users noticed the link provided at the right of the webpage though some initially looked on menu on the left. Users were able to fill the subscription form out though some noted that they were asked for too much personal information²².