Methodology
This process evaluation was designed to assess the implementation and perceived impact of the Phase 2 Safe System Capacity and Capability Study. A mixed-methods approach was adopted, combining qualitative and quantitative data to provide a comprehensive view of programme delivery, stakeholder experiences, and emerging outcomes.
The evaluation was guided by the overarching aim of understanding how the training programme and related outputs were delivered and received, and whether they supported the intended capacity-building goals. The evaluation focused on both process-level insights (how activities were delivered) and user-level outcomes (how the activities were experienced and applied).
Scope of the evaluation
It is important to note that not all programme components were included in the scope of this process evaluation. Specifically, the Safe System Cultural Maturity Survey, which was delivered directly by Agilysis, falls outside of the remit of this report. Additionally, the Safe System Manual was still in development at the time of data collection. As such, its role and anticipated value are touched on only briefly, based on interview insights from Transport Scotland and Agilysis.
This process evaluation therefore focuses primarily on the training elements of the programme, exploring how the in-person and online sessions were delivered and received, and examining perceived outcomes from the perspectives of the delivery partner (Agilysis), the client (Transport Scotland), and training participants.
Data collection methods
Data for the evaluation were collected through three primary sources: stakeholder interviews, participant surveys, and document review.
A series of semi-structured interviews were conducted with key individuals involved in programme planning, delivery, or participation (See Annex A). These included representatives from the client organisation, Transport Scotland, and the delivery partner, Agilysis. In addition, one stakeholder who had taken part in the training also participated in an interview, providing a direct participant perspective. The interviews explored experiences of programme implementation, participant engagement, and training impact; perceptions of the Safe System Manual; coordination between partners; reflections on programme outcomes; and suggestions for future development.
Two online surveys were distributed to participants who attended the Safe System Foundation (In-person) and Safe System Principles (Online) training sessions (See Annexes B and C). A pre-training survey was used to gather baseline data on participants’ existing knowledge of the Safe System, their understanding of key concepts, and their motivations and expectations for attending the training. The follow-up post-training survey collected feedback on participants’ experiences of the training, perceived knowledge gains, confidence in applying the content, and intended changes to work practices. Participants were also asked about potential barriers to implementation and their need for further support. Both surveys were designed for ease of completion and were distributed by email to attendees of the in-person and online sessions.
In addition, a range of project documents, including training plans, milestone reports, and communications, were reviewed to provide background context and support understanding of how the programme was delivered.
Data analysis techniques
The evaluation used a mixed-methods approach to analyse qualitative and quantitative data from surveys and stakeholder interviews, ensuring a comprehensive understanding of programme delivery, participant experience, and early outcomes.
Qualitative analysis
Stakeholder interviews were reviewed and summarised thematically using a structured approach informed by the evaluation objectives. Interview transcripts were collated and key excerpts were organised under broad topic areas aligned with the evaluation framework. Thematic summaries were then developed iteratively to identify common insights, differences in perspective, and overarching patterns. The final thematic structure reflected six core themes, which underpin the findings presented in Section 4.1:
- Training programme delivery and implementation
- Training participant experience and perceived impact
- Safe System Manual development
- Project coordination and communication
- Reflections on programme outcomes
- Suggestions for future programmes
A single interview with a training participant was also included in this analysis to provide additional context from the learner perspective and to illustrate how the training was experienced first-hand.
Qualitative data from open-ended survey responses were reviewed separately and are integrated throughout Section 4.2 (Survey findings). While these responses were not thematically coded, a structured review was undertaken to identify illustrative comments that contextualised and expanded upon the quantitative findings. These insights provide additional depth into participants’ reflections, perceived training value, and suggestions for improvement.
Quantitative analysis
Survey responses were analysed using descriptive and inferential statistics. Descriptive statistics (means, standard errors, and frequency distributions) were used to summarise participant characteristics, baseline knowledge, and training outcomes. Independent-samples t-tests were used to compare pre- and post-training scores on key outcome measures, including overall knowledge ratings and composite knowledge scales.
Effect sizes (Cohen’s d) were calculated to assess the magnitude of any observed changes, and Cronbach’s alpha (α) was used to assess internal consistency across multi-item scales. Due to the anonymous nature of the surveys, responses could not be matched across time points, and results were analysed as independent samples.
Results are presented in Section 4.2, supported by visual summaries (tables and figures) to highlight key patterns and areas of change in participants’ knowledge, attitudes, and intended practice.
Limitations
Several limitations should be considered when interpreting the findings of this evaluation. Although survey response rates were sufficient to identify trends and themes, they represent only a portion of the more than 500 individuals who participated in the training. This introduces the potential for self-selection bias, as those who responded may have been more engaged or positively predisposed to the training.
All data collected were self-reported, which may be subject to recall bias or socially desirable responses, particularly in questions relating to intended behaviour change or perceptions of impact. While both pre- and post-training surveys were conducted, responses could not be matched at the individual level due to anonymity. As such, the data were analysed as independent samples. It is possible that some individuals completed both surveys, introducing a risk of non-independence; however, this could not be verified. This limitation affects the strength of inferences about individual-level change over time.
The qualitative component of the evaluation included interviews with the commissioning client and delivery partner, but only one stakeholder who had directly attended the training sessions. This limited the depth and breadth of first-hand participant perspectives captured through interviews, though the open-ended survey responses provided rich supplementary insight into participant experiences.
Finally, the evaluation was conducted within a relatively short time frame following the delivery of programme components. This meant that longer-term outcomes, such as sustained changes in practice, cultural shifts, or policy alignment, could not yet be assessed.