Road Accident Data Collection Form Design Research Project

6 Vignette study

Methodology

The aim of the vignette study was for stakeholders to provide feedback on the revised form, both in terms of ease of use of the form and the accuracy of data recorded, based on using the form for a set of fictional collisions.

Vignettes were developed for four collisions. The ten Police participants from the consultation were sent the vignettes, together with the revised form and a brief description of the vignettes study.

In order to compare the reliability of the revised form with existing methods, each participant was asked to complete STATS19 data for half the cases using their existing method, if possible, and half using the revised form. For each participant we stated which of the four cases should be completed using which method, aiming to achieve five responses for each case using the revised form, and five using their existing methods.

Police divisions which input directly onto a database/online system were not able to use their current method as there was no way of telling the system that the collision is a dummy collision and not a real collision to be included in the final STATS19 data set. These officers were only asked to complete the revised forms.

The STATS19 data completed using the current processes and the revised form were compared for each of the vignettes to assess the reliability of the data using the different processes.

Any feedback that the participants supplied relating to the revised form was also reviewed.

Content

The vignettes consisted of fictional collision reports containing a text description of a collision and included a map, or photo. Any photos used were from a set made available from the On-The-Spot (OTS) project, a project for the Department for Transport which involved TRL expert investigators attending the scene of collisions. These photos have been sanitised so that personal data and the true locations of accidents cannot be determined from the photos.

The vignettes were designed to include those collision types that were reported to cause confusion or result in inconsistencies from the literature review and consultation:

  • All of the cases required coding of the location of the accident based on a map and description of the location.
  • Vehicles involved included a van and a motorcycle.
  • The severities of participants in the collision were described in terms of their injury so that the reporting officer needed to determine the casualty severity.
  • The date of births or ages of the participants were provided so that the driver/rider/casualty age could be completed.

Some data that a Police Officer would normally have access to, such as the Vehicle Registration Number or the home postcode of the driver or casualty were not included.

The vignettes included:

  • Case A: Collision at a roundabout to assess the recording of junction type, junction locations, vehicle manoeuvres and compass points
  • Case B: Pedestrian impaired by alcohol crossing road masked to assess the consistency between the pedestrian movement and CF and the impaired by alcohol CF for driver/rider and pedestrian
  • Case C: Collision at private drive to assess coding of 2nd road class
  • Case D: Collision on a motorway to assess coding of road type

The vignettes were reviewed and tested using the revised form before being sent to the participants. The full vignettes are provided in Appendix C.

Results

A total of eleven revised forms and nine existing forms were completed by seven officers based on the provided vignettes. The result of having a smaller sample than expected is due to complications that arose throughout the data collection process that included not being able to use existing systems for 'dummy' cases, Police resourcing issues as well as challenges concerning compatibility of the form using existing versions of Microsoft Excel. The breakdown of the number of cases reported using each form can be found in Table 2.

Table 2: Vignette study sample size
Vignette Number of existing forms completed Number of revised forms completed Total
Case A 1 3 4
Case B 2 4 6
Case C 4 1 5
Case D 2 3 5
Total 9 11 20

Previous stages of this project identified key areas of the existing forms that required improvements. These fields included: weather, time, road type, first road class, junction detail, junction control, second road class, grid reference, vehicle type, manoeuvre, casualty class, casualty severity, contributing factors and inconsistencies between different fields (e.g. helmet worn for a pedestrian casualty).

Table 3 gives a summary of the data provided for these variables for existing forms and the draft revised form. Recommendations for revisions to the draft revised form to improve the completeness and accuracy of the data are also included. See Appendix E for detailed descriptions of revisions to the form.

Table 3: Discrepancies between revised and existing forms based on the Vignette Study
Section of the Form Field Existing forms Draft revised form

Accident

Weather

Two forms contained less detail than required (i.e. 'rain' instead of 'rain with high winds')

One form had no field in which to record the weather

All data contained the same amount of detail as a result of the dropdown menus

Three forms listed the weather as 'unknown'. This option has subsequently been removed

Time

Mixture of hhmm and hh:mm formats

All results in hh:mm format

Road type

One form had no field in which to record the road type

All information entered accurately

1st Road class

Inconsistencies with how unclassified roads are listed ('U' and 'Unclassified')

Inconsistencies with how M, A and B roads are listed ('B' and 'B9119')

All information entered accurately and consistently

Junction detail

Three forms had no field in which to record junction detail

Result is not being able to determine if subsequent blank junction control and 2nd road class fields were because the accident did not take place at a junction or if the officer forgot to fill them in

Four forms left junction detail blank. A reminder has subsequently been added to remind the officer that this is a required field

Junction detail has subsequently moved from 'junction accidents only' section to main section of form

Junction control

No inconsistent data (i.e. no information entered if junction detail was listed as 'Not at or within…')

No inconsistent data (i.e. no information entered if junction detail was listed as 'Not at or within…')

2nd Road class

Inconsistencies with how M, A and B roads are listed ('B' and 'B7078')

All information entered accurately

Grid reference

No inconsistencies with the ranges listed

Three forms were left blank.

No inconsistencies with the ranges listed

Three forms were left blank. A reminder has subsequently been added to remind the officer that this is a required field

Vehicle

Vehicle type

One form only prompted the officer to record the make of the vehicle

Two forms included only general vehicle information ('Lorry' and 'Motorcycle')

All information entered accurately

Vehicle manoeuvre

Three forms had no field in which to record the vehicle manoeuvre

All information entered accurately

Vehicle reference (Casualty Form)

Inconsistencies with how data is entered (e.g. '1', 'vehicle 1' and 'Astra')

One pedestrian casualty not linked to a vehicle. A reminder was subsequently added

Casualty

Casualty class

All information entered accurately

All information entered accurately

Casualty Severity

All information entered accurately

All information entered accurately

Conflicting information

One form listed an individual as a driver and then as a passenger on the 'CasForm'

One form provided conflicting driver sex data on the 'VehForm' and 'CasForm'

One form listed 'Seatbelt in use: unknown' for a pedestrian casualty

One form listed 'Seatbelt in use: unknown' for a pedestrian casualty. Warning was subsequently added to prevent this

No other conflicting information

Contributing Factors

Conflicting information

One form provided conflicting information (vehicle reference for a casualty CF)

All information entered accurately

It is important to note that further testing is required to validate these results before the form is adopted or rolled out widely. However, despite the small number of responses, the data that were provided using the revised form were both more accurate/detailed and more consistent as compared with the data provided using the existing forms. Errors that were present in the revised forms have been accounted for using warnings and validation procedures as described below.

Feedback from use of revised form

As mentioned at the beginning of this section, feedback from the participants relating to the revised form was also reviewed. Of the seven participants who returned the completed forms, six provided general feedback regarding their experience.

Several of the Police Officers believed that the new form was easy to use and an improvement on the current form used by their legacy Police area:

"I certainly found the new forms easier to complete" (Police)

"The form seems to be fairly user-friendly and certainly an improvement on what we currently use" (Police)

"The new form is light years better than the old paper form which is much more cumbersome" (Police)

One Police Officer believed some aspects of the form were a backward step, particularly the element of inputting the grid reference points on the form.

"This will lead to mistakes being made and vehicles ending up in a totally wrong location due to operator error" (Police)

However, this may be due to the participant having access to a more sophisticated system for collecting this type of data in their current STATS19 system; from the consultation exercise described on page 15, it was clear that not all regions have access to this type of system. The participant described the current method used by his legacy Police area which he believed to help reduce the likelihood of errors. Evaluation of such systems to establish whether they actually improve the accuracy of location data would determine whether there would be value in Police Scotland rolling out this sort of system more widely.

"We have a system here where you use a Google earth type of map, you click on the exact locus, the computer calculates the grid reference and then a compass comes up on the screen where you show the direction of travel." (Police)

Although the form was generally well received, a number of suggestions were made for further improvement. This included the addition of free-text boxes for elements such as details of injuries sustained (other than severity), damage caused to road-side furniture (such as fencing or barriers), or a brief summary of the collision.

Some respondents also identified that there were no spaces available to enter the full details (including name, address and post code) or drivers, witnesses or casualties.

A number of participants also commented regarding the lack of a space available to collect insurance details.

However, some of these additional details, although need to be recorded, are not part of the STATS19 data requirements. While the capture of additional details beyond the STATS19 data variables were out of the scope for this project, future revisions of the form could include space for such information (acknowledging that any additional data capture requirements would increase the time taken to record this and would increase complexity of the dataset collected).

As mentioned previously, some participants did not provide any detailed feedback regarding their experience with the form; however, the redesigned form seemed to be well received among many of the respondents. The next section highlights how the feedback obtained through this (and other parts of this study) were collated to further enhance the revised form.

Summary and revisions to form

Four vignettes were developed, refined and tested with Police Officers who had participated in the consultation. Each vignette described a fictional collision report – providing the Police offices with a text description and a map or photo. They were designed to include collision types known to cause confusion or result in inconsistencies. Although the sample size achieved was small, qualitative feedback from users indicated that the revised form was well-received, and in addition, the vignette data collated from the draft revised form yielded more accurate and more consistent results than the forms that they were used to completing.

The feedback and analysis from the vignette study, along with further information from the literature review and the project team was collated and is summarised in Appendix E, together with the response for each comment. In many cases further enhancements were incorporated into the revised form, but in some cases the suggestions were not implemented, with reasons given in the table.

Key amendments included:

  • further guidance included for breath test, vehicle movement and driver/casualty ages
  • vehicle type refined to two levels so that users select from a shorter list, with a further drop down menu giving further details for motorcycles, goods vehicles and other vehicles
  • further validation to remind users to complete key data and to check casualty sex, car passenger, bus/coach passenger, seat belt and cycle helmet based on the casualty class
  • compatibility checks for earlier versions of Excel