Road Accident Data Collection Form Design Research Project
4 Consultation
Aims
The purpose of this phase was to engage with those who interact most with the STATS19 form and data (the people who record, enter, process and use the data) in order to gain a better understanding of challenges and/or barriers to accurate data collection that users experience.
The specific aims of the consultation were:
1. To undertake qualitative interviews with stakeholders;
2. To analyse the interview data in order to identify key themes and trends in responses;
3. To identify any differences or conflicts between users' perceptions and use of the form.
These aims were achieved by a series of qualitative interviews with different stakeholder groups in Scotland, as well as engagement with those involved in the project from Transport Scotland.
Issues known to Transport Scotland
Transport Scotland provided documents with known areas of concern in the collection of the STATS19 data. These issues are recorded and reported in order that they can be reviewed periodically in line with the quinquennial reviews carried out by the Standing Committee on Road Accident Statistics (SCRAS). Table A-1 in Appendix A highlights the results of one such report including detailed information on common issues with data recording on the STATS19 form. Data inaccuracies were flagged relating to contributory factors and to casualty, vehicle and accident records. This information was compiled by Transport Scotland based on their own experiences with STATS19 data and feedback from other stakeholders.
Method
Sample
Telephone interviews were undertaken with eighteen stakeholders from Police Scotland, Local Authorities, and statisticians from Transport Scotland. All eight of the legacy Scottish Police force areas, believed to be using different STATS19 recording systems were represented in the sample:
1. Central Scotland Police
2. Dumfries and Galloway Constabulary
3. Fife Constabulary
4. Grampian Police
5. Lothian and Borders Police
6. Northern Constabulary
7. Strathclyde Police
8. Tayside Police
For some of the areas, more than one type of STATS19 form user was interviewed, although for Lothian and Borders, only Local Authority representatives were interviewed. This was mostly due to participant availability to take part in the interviews.
The final sample was comprised of 18 interviewees across the three stakeholder groups. Table 1 provides a breakdown of the sample and the areas represented.
| Area | Current division code | Stakeholder group | Number of interviewees |
|---|---|---|---|
| Dumfries and Galloway | V | Police | 1 |
| Grampian | A, B | Police | 3 |
| Northern | N | Police | 1 |
| Central | C | Police | 2 |
| Strathclyde | U | Police/ Local Authority | 2 (1 each group) |
| Fife | P | Police/ Local Authority | 2 (1 each group) |
| Lothian and Borders | E, J | Local Authority | 3 |
| Tayside | D | Police | 1 |
| Transport Scotland | n/a | Statisticians | 3 |
| TOTAL | 18 |
Typically, the level of experience with STATS19 data of those interviewed (excluding the Transport Scotland representatives) was high. The average level of experience reported was approximately 15 years. The participant with the least experience had worked with this data for 5 years (Police); while the two participants with the most experience reported having worked with STATS19 for 25 years (one from the Police and the other from a Local Authority).
One participant (from a Local Authority) reported having "very limited" experience with STATS19 data, as their role only involved using high level data mostly for educational interventions and other aspects of evidence-based practice.
Recruitment
Recruitment was facilitated in part by contacts provided by Transport Scotland, who served as gatekeepers into the organisations of interest. TRL staff then established contact with these organisations, either to book an interview with a particular named individual or to obtain details for other potential interviewees.
Interviews were arranged at a time convenient to the interviewee and participants did not receive an incentive for taking part in the research.
Format of telephone interviews
A semi-structured interview format was used in order to ensure consistency in the data collection process.
Two topic guides were developed that covered issues relating to the different stakeholder groups involved in the interviews. One version was created specifically for Police STATS19 users as these were expected to have different experiences with the form and be aware of different issues with STATS19 data collection compared to those from Local Authorities and Transport Scotland (who had another version of the topic guide).
The duration of interviews was between 30 and 40 minutes depending on the amount of experience of using STATS19 data and the information that participants were willing to share.
Although most interviews were carried out on a one-on-one basis (one researcher and one participant), one interview (carried out with the Transport Scotland statistics team) was completed with three interviewees at the same time. All interviews were undertaken by one of two experienced qualitative researchers from TRL.
Outputs
During the interviews, the researcher took detailed notes of the participant's responses. Each interview was summarised according to the following key factors:
- data completion process and related challenges
- suggested new variables
- misunderstood variables
- variables that are often coded or entered incorrectly
- clarification requests
- use of STATS19 data and STATS20 manual
- general remarks on STATS19 use, improvements, or understanding
Analysis
Qualitative thematic content analysis was applied to notes made in the interviews and is reported in the following sections. Thematic content analysis is a technique that can be defined as the "systematic, objective, and quantitative analysis of message (or theme) characteristics" (Neuendorf, 2002). The steps involved in this analysis included:
- preparing the data for analysis – this involved reading all of the interview notes to ensure that they were accurate representations of what was said in the interviews.
- closer examination of the text – the text was reviewed line by line to facilitate micro-analysis of the data
- initial identification of themes – this occurred in two stages. Firstly, researchers identified topics of interest individually which were sorted into 'themes' – i.e. quotes and sections of the interviews relating to similar topics. Secondly, the two researchers took part in a short workshop where possible emerging themes were discussed and justified
- re-examination of the text for relevant examples of each theme – each set of notes was re-examined for information relating to the themes identified in the above exercise
- construction of the final structure of each theme – the name, definition and supporting data were re-examined for the final construction of each theme using all of the data relating to it
- reporting of themes – each theme was described and illustrated by use of quotes from the original text (where possible) to help communicate participants' meaning
Findings
Current STATS19 data collection practices
Before discussing the results of the interviews, it is important to contextualise these results by providing insight into the types of forms and processes used by the legacy Police areas interviewed.
The process for collecting data reported by most Police interviewees involved the attending Police Officer recording accident details in a notebook at the scene. Upon returning to the Police station, the same attending officer entered the information into a STATS19 'form' (which may be the illustrative DfT/Transport Scotland form or local versions). In some cases this was a fully electronic format (including drop-down boxes) while in others it was a Microsoft Word template (with blank fields). One participant suggested that the reporting officer does not always fill in the form, sometimes it is the enquiry officer on duty who undertakes this task based on the attending officer's notes.
Another participant reported a practice where attending officers telephone a 'voice bank' who then input the data. However, even though attending officers did not complete the form personally, in this case, notes were still made and kept by the officer. The use of a 'voice bank', however, did not seem to be widespread among legacy Police Force areas.
The final method reported by interviewees involved the use of a Personal Digital Assistant (PDA) used to collect data at the scene, which then synchronises and uploads data automatically onto a computer. This did not appear to be common practice across other legacy forces.
Themes
A number of themes emerged from the interviews undertaken. These have been divided into major and minor themes. Major themes are those that emerged consistently across interviews, and minor themes are topics that were not repeated frequently, but that warrant discussion. These are presented below.
Major themes
Experience governs the way in which data is captured and entered
One prominent finding from interviews, particularly those carried out with Police, was the role played by 'experience'. Police Officers do not necessarily memorise the particular details of the STATS19 form, instead they rely on their experience to know and understand the information that needs to be collected at the scene of the accident.
"We have a knowledge of the information that's required…" – (Police)
There is some understanding among Police Officers of the limitations of the manual process of note-taking, for example identifying the precise location and compass points of the road traffic collision (RTC).
"You're not going to know that [compass points] just off the top of your head." (Police)
Nonetheless, respondents in this group believed that the method they currently employ is the most efficient way of collecting the necessary data.
"There isn't any other way of recording [RTC data]" (Police)
In fact, when asked why this method was selected, responses generally indicated that it was standard practice, "what's always been done".
"If we attend a road traffic collision, we'd always have notes… It's just something everyone does…" (Police)
Experience was not only valued by stakeholders representing the Police; some Local Authorities also described using their experience to work with the STATS19 data. For example, one participant mentioned using a combination of available data and their own experience to correct errors relating to location. Another participant believed his organisation had the systems and knowledge available to check missing or incorrect data; they will only resort to seeking clarifications from the Police if the data is for a serious or fatal accident. No further information was provided as to who (administrative staff or attending Police Officers) provide the clarifications.
Accurate logging of location is a major challenge
Although, generally, Police interviewees did not believe there were major issues with obtaining accurate RTC information, several participants mentioned some issues related to specific details such as location (geo coding, and the compass points – for example, was the accident-involved vehicle travelling North to South). This type of information may not be written down at the scene of the accident; instead the attending officers described having to conduct some research when they get back to their station in order to complete this part of the form. However, officers seemed to believe that the experience of having attended the scene is enough to be able to complete this information accurately.
The issue regarding location was also raised during interviews with Local Authorities and Transport Scotland representatives as well, particularly as this data was described as being more prone to errors when submitted. The issue of location inaccuracies seemed to be particularly problematic for Local Authorities as this data was viewed as important in reaching the organisations goals in road safety.
"Grid references are critical…" (Local Authority)
Some participants provided information about the number of incorrect forms to quantify the size of the problem, particularly referring to clarifications relating to location.
"We may send back 2 or 3 cases [out of ~35] for clarifications, in a month."
(Local Authority)
"In most cases it's incorrect [grid reference] and I have to give it a new grid reference based on my experience, the accident description and the location description." (Local Authority)
The scale of the problem, however, was not the same for all users of the data. Although not all participants from Local Authorities mentioned specific organisational issues arising from location inaccuracies, given that most reported they use the data to identify hotspots or areas for concern, it may be implied that accurate location information is important. The difference may be in the systems or expertise they have available to correct any inaccuracies in the data.
"[Our] system's got road numbers and different classifications of roads so we can actually make the data set a wee bit better…" (Local Authority)
Some of the checks done on the data reported by Local Authorities included superimposing data on a GIS network, checking against other data sources (own records of road type and number), and checking grid references.
Improvements to the location data collection process
Some suggestions were made for improvements to the collection of location data. A number of participants believed that technology was the solution to this problem. For example, one Local Authority participant believed that making this process more automated would "take out human error".
Another participant recommended that GPS data were collected at the scene of the accident. This would remove the opportunity to make mistakes in logging the location. The participant added that with the electronic format,
"You can't skip any information and can't get it wrong" (Police)
Other suggestions related to more detailed training of those who collect the data and improving understanding of the importance of this data for other users. Suggestions were provided by both Police and Local Authorities.
"They [Police] don't understand the importance of data they are collecting…" (Local Authority)
"Emphasise to Police what the data is used for…" (Local Authority)
"They [new officers] don't quite understand what the form is trying to achieve in the end… and for me that's something that as a force now, or as a company, that we should address…" (Police)
One participant also suggested that better training of people collecting data may result in more accurate data collection, particularly in terms of knowledge of details such as compass points and direction of travel
Other missing or incorrect data
Although the key issue seems to relate to the accuracy of the location of the RTC, several other areas prone to errors were also discussed by participants; particularly Local Authority representatives. Issues were quite varied, and in many instances were not viewed as being particularly problematic.
Accurate and complete information about the age of casualty or driver was considered important by statisticians. Participants commented that details on ages are important in identifying child casualties and accurately monitoring trends as this is part of Transport Scotland's Road Safety Framework to 2020 (Transport Scotland, 2009).
Other issues mentioned included: the number of casualties, Local Authority reference number, and descriptions of vehicles. Statisticians interviewed also mentioned issues with dates, severity of accidents, and missing records (such as casualty and vehicle).
On the other hand, when consulting those who complete the STATS19 form, issues with data collection or particular variables were not generally raised. When asked specifically to comment on variables that may be difficult to record accurately, officers tended to report that the process was "fairly straight forward". One of the few issues mentioned related to recording specifics on makes or types of vehicles (for example, motorcycle engines). Officers may not have pre-existing knowledge in this area and must wait until they return to the station to carry out some research.
It is important to note that none of the problems reported in this section were believed to be particularly prevalent or challenging to the organisations' operations.
Contributory factors generally not viewed as difficult to code
In general, interviewees did not find that the contributory factors (CFs) on the existing STATS19 forms were particularly problematic. Representatives from both the Police and Local Authorities found the collection and interpretation of these to be generally straight forward, however a few specific issues were raised.
For example, one participant from Police Scotland reported that there are "No hard and fast rules" for assigning contributory factors. They also mentioned that some officers only apply one CF. However, it is worth noting that STATS20 does include guidelines for assigning the contributory factors.
Another Officer shared similar views and reported that CFs are not robust and he believed that perhaps a way forward would be to remove these options and simply have officers write down what they believe the CF to be. Another participant believed that recording certain CFs, such as speeding, may be tricky, particularly if there were no witnesses. A further officer mentioned 'careless driving' as a CF that is difficult to record accurately under certain situations.
A participant mentioned that a positive aspect of the current form was the option on the form to assign a 'confidence' level to the CFs. This was viewed as an improvement to the accuracy of the information (such as speeding).
A participant from a Local Authority was concerned that the wording for 'pedestrian failed to look' and 'driver failed to look' was the same in the form and that this could lead to inaccuracies in the data collection process .
One Police Officer mentioned that the term 'participant' may cause confusion because it may not be clear which person (involved in the RTC) they are referring to.
A lack of confidence in the data by some Local Authorities was perceived. This may be because of their perception of the data quality for variables such as accident locations. As mentioned previously the findings showed that, in general, monthly RTC reports sent to the Local Authorities interviewed may have one or more errors, according to participants.
In terms of the contributory factors specifically, more than one Local Authority participant explained that they do not use this data because they do not trust the accuracy of it. One participant mentioned that this was because he did not believe it was the attending officers who completed the STATS19 form, and hence assumed that the person who had completed it may not have the knowledge or experience necessary to make an accurate allocation of CF.
"I don't know the experience of the person inputting the data" (Local Authority)
However, as described above, most of the participants representing legacy Police forces reported data was inputted onto the illustrative STATS19 form (or an equivalent form) by the attending officer.
The concerns raised by participants from Local Authorities may arise from a misunderstanding of the data collection process. Perhaps a better mutual understanding of how the data is collected and inputted, why this process is important and the importance of accurate information is required. An open discussion between all users of the data may facilitate each stakeholder group's understanding of the importance of the data collection process.
Improvements for CFs
Of all those interviewed, only one participant (from a Local Authority) mentioned a suggested variable they would like to see collected; this related to recording whether or not a child passenger was restrained. The participant believed that this variable could help inform campaigns aimed at parents, particularly providing an evidence-base for such campaigns. Although we are aware that this does not fit in with the aims of the contributory factor data collected on the STATS19 form, and that there is an option to complete restraint status for any vehicle occupant casualty, the participant believed that this could be an added benefit to having the data.
In terms of other changes relating to CFs, a number of improvements were mentioned, though these mostly related to general ease of use of the form and accurate data collection.
For example, one Local Authority participant commented that a sound qualitative account of what happened (i.e. a plain word description of the collision) was as important as the quantitative data collection "a good accident story and a good location…". He believed the story (narrative) regarding the circumstances of the accident came directly from the attending officer.
However, other participants viewed the data collected as necessary; one interviewee from Police Scotland acknowledged that the data collection process is a "necessary evil", but felt that the amount of information required is too detailed. No suggestions were provided as to what variables could be edited or deleted.
"If I could take away form filling from the Police Officers, I would" (Police)
Minor themes
Aims of the form
There was a divide between the perceived aims of completing the STATS19 form. For Police, participants seemed to understand form completion as part of their job. When asked regarding their motivations for completing the form, participants tended to relate it back to the requirements of their roles.
"Because I have to - that's the bottom line, it's procedure" (Police)
Some knowledge on how data is used by Local Authorities was also expressed by Police who generally understood that it is used for road safety, for example "identifying hot spots" was cited by most interviewees.
Conversely, Local Authorities reported some very clear aims for the data collected; to have data that will help them in achieving the organisation's goals in road safety, particularly in identifying hotspots or any casualty trends that they need to be aware of. Other uses included a range of activities from road safety engineering, producing reports, and carrying out investigations.
Suggestions for improvement
As mentioned previously, generally, the form was not believed to be particularly problematic. Form users believed that experience is key in understanding what data is required and how to fill in the form. Hence, the main improvements brought up during interviews related to the convenience and effort required from Police to fill in the necessary data.
Form modifications
Participants from the Police expressed a desire for any revised form to be as user-friendly as possible. While many fill in a word version of the form which requires they simply "work through it", others described fully electronic systems that make the process easier (e.g. no issues with handwriting, can carry out necessary checks on site). For example, one legacy Police force area reported using a PDA to collect data on site, which can then be uploaded directly onto a word document. This was believed to be a step forward in ensuring collection of more accurate data.
There was a desire expressed by one participant from Police Scotland that the form was made so that "you can't put wrong information anywhere".
The addition of drop-down menus was particularly favoured by some of the interviewees who believed that they would help in situations where officers are unsure of how to complete a question or where there was too much room for interpretation. Two officers commented:
"Drop-downs would be useful" (Police)
"A drop-down system may alleviate some of these problems" (Police)
In fact, one participant interviewed represented a Police division that had recently (2013) changed to a new electronic form which mostly employs drop-down menus. This was viewed as a significant improvement from the previous (manual, paper) form.
The order of the categories presented was also viewed as a potential area for improvement. One participant suggested that the most frequently recorded categories should be nearer the top of the list.
Finally, although not an additional variable, there was a strong feeling by one of the interviewees that when assigning a road class and road type, there should be an option provided for "Motorway". Although the option for motorway already exists on the current STATS19 data (i.e. motorways are classified as dual carriageways in road type, and then further specified in the road name), according to the participant this is a source of confusion which in turn may lead to inaccuracies in the data collected.
Similarly, another participant commented that there wasn't an option for ambulances or camper-vans in the vehicle type. This was also seen as a welcome addition.
Although some changes to the form were suggested, one participant considered the implications of redesigning a form and believed that this would have significant cost implications for their organisation given some in-house limitations.
"Our computer system is not maintained in house… we have to go to an external consultant so it's quite pricey to get things changed." (Local Authority)
Procedural modifications
There was a desire across all stakeholder groups for some sort of consistent form/system to be used across the whole country. Some of the recommendations for improvement included having a central database where data was collected and available for viewing by all users. This could be of potential benefit, particularly as one participant reported differences between databases[3].
"The Police database doesn't necessarily match the national database." – (Local Authority)
A further recommendation by a Local Authority representative was for any revised form to re-introduce a sketch or diagram of the accident. The participant reported they used to receive this information but no longer do. This information was deemed to be "of great value", particularly given the lack of confidence in the recording of location.
The use of technology was also viewed as something that could improve data collection. One participant was part of a two-year pilot where officers are provided with PDAs to input the data at the scene for the RTC. The participant reported a large improvement from the previous system, as this allows officers access to sources such as the Police National Computer (PNC) directly on site. It was also viewed as a positive step toward increasing accuracy as they suggested that this prevented Officers from skipping or entering incorrect information.
Interestingly, consultation with Transport Scotland statisticians revealed that although both former Strathclyde and Northern divisions have updated to what is considered a more "user friendly" PDA form format, they are not among the divisions with the fewest errors encountered. However, it is not clear if this is related to the relatively short time these systems have been in place, to the methods used to identify errors in the data collection or to some other factors.
Knowledge and use of STATS20
Views were mixed regarding the familiarity with and ease of use of the STATS20 guidance. Most participants reported having some knowledge of it (at least most of them had heard of it), and its uses seemed to relate to conducting checks (e.g. definitions) or referring to particular data when unsure (e.g. vehicle subdivisions).
Some Local Authority participants found it very useful, and could quantify their use of it, such as "once a week". Some Police Officers interviewed also had positive views on it.
"Just about every officer refers to it when completing the form." (Police)
"It's written in such a way that I can understand" (Police)
However, other users from Police Scotland found it to be a "long read" and believed it could be improved by making it more condensed.
Only two participants reported not knowing of STATS20; one was a Police representative and the other represented a Local Authority.
There did not seem to be a significant difference in use of STATS20 between Local Authorities and Police.
Summary
Generally the types of form used, the processes followed and user perceptions were varied. It became clear throughout the interviews that no consistent approach is used, despite widespread agreement that such consistency is desirable.
Some of the issues associated with identifying location data accurately may relate to a lack of information available to Police Officers of compass points or direction of travel at the scene. This may be further hindered if any information relevant to the accident location has to be researched when officers get back to their station, particularly as Police may have to be on site for several hours at a time in some cases.
A further issue concerning how data is input relates to whether the attending officer or someone on their behalf completed the form. While Local Authorities believed that this was a concern, interviews with Police Officers suggested that on most occasions it is the attending officer who completes the form, although typically not at the scene of a collision.
Although no major areas of improvement to the data collection procedure were identified, a number of minor issues were brought up. These mostly related to user friendliness and adding information or options that can make data more accurate – for example, adding in options for types of vehicles that may be difficult to record accurately (ambulances or camper vans), further expanding the option list of road types (i.e. adding an option for "motorway"), and even providing forms with drop-down menu options that remove some of the subjectivity of the data collection process. A further suggested improvement relating to how data was collected was to investigate more sophisticated technology options as a potential avenue to help improve the accuracy of the data collection process. This was particularly viewed as useful for recording location data.
This said, overall, the form was considered to be "straight forward" and the data collected was viewed as valuable. None of the participants identified specific redundant or less useful variables, even when they suggested shortening the form to make it more user-friendly.
Finally, although this was not a widely held opinion among participants interviewed, one participant believed that changes to the form would have significant cost implications for his organisation (cost of implementation was not raised by others). This is something that should also be balanced out when considering overall improvements to the STATS19 data collection process.