Methodology

This paper is based on analysis undertaken by ScotRail and Transport Scotland, drawing together rail specific analysis with an initial assessment of the multi-model impacts. This is an interim report and the issues it raises will be explored fully in the final evaluation of the pilot.

ScotRail data and survey analysis

Initial ScotRail data analysis covers the first 11 weeks of the trial, from Monday 2nd October up until Saturday 16th December 2023. It includes an assessment of the impact on ScotRail services in terms of passenger journeys and associated revenue, passenger loadings and impact on capacity as well as emerging outcomes from market research with customers. 

ScotRail have access to a customer panel of over 6000 people, which provides information on existing rail customers. The panel was supplemented via initial customer research that was pushed out via the ScotRail on-train Wi-Fi platform and 66 stations during the initial weeks of the trial to obtain immediate views on awareness and perceptions of the trail and impacts on passengers.

Transport Scotland survey analysis

This section details the methods used by Transport Scotland to understand the impact of the pilot and specifically to understand whether more people now consider rail as an affordable, viable travel choice; whether more people choose to travel by rail more often; and whether in doing so are switching from private car to rail. The pilot is also testing whether removing peak fares pilot is value for money (VfM). The specific methodology used to assess VfM is discussed below.

The focus for the Transport Scotland data collection is to understand the wider impact of the pilot on the transport network alongside existing rail customers. The aim of the TS data collection was to compliment ScotRail’s work by filling key data gaps and provide a sample that is more representative of the population, as well as a wider range of rail customers less likely to access Wi-Fi.

The scoping exercise identified the following data gaps, which would be filled through additional survey work with the public.

• Travel behaviour of non-rail passengers, occasional rail passengers, and any barriers to rail travel.
• Rail passengers not able to access Wi-Fi at ScotRail Stations
• Different demographic groups and protected characteristics, e.g. gender, age disability, and different socioeconomic groups
• Mode shift - people reporting intention to shift/actual shift from other modes to rail and the reasons
• Where possible, mode shift from observed data (rail, car traffic, bus). However, this is challenging to quantify during the 6 month pilot as behaviours take time to embed.

The final scope of TS survey work concluded that an online questionnaire survey was needed to capture rail users and non-rail users behaviour during the pilot period, the former complementing any work undertaken by ScotRail.

The table in Appendix A details the survey data outputs required to assess the impact of the Pilot, and whether this was being collected by TS or ScotRail. 

Target Groups

The online survey was designed to target four distinct population groups to understand the impact of the pilot. The groups are listed below with respective sample size. In total 1,476 responses were received.

• User Group 1 – Rail users, who do not change their behaviours: 445 responses (30%)
• User Group 2 – Rail users, who do change their behaviours; 444 responses (30%)
• User Group 3 – Non-Rail users, who do change their behaviours; 162 responses (11%)
• User Group 4 - Non-Rail users, who do not change their behaviours; 425 responses (29%)

As well as considering the different groups using or not using rail services, the sample was designed to capture a representation across certain protected characteristics, in line with the equalities outcomes around accessible and affordable transport detailed in the National Transport Strategy. This included splits by age, disability, gender and income – as an indicator of socioeconomic status. In addition, participants were asked which local authority they lived in and their employment status.

Demographic Characteristics

A quarter of responses were provided by those between the ages of 51 to 65. Those under the age of 30 provided just under a quarter of responses, whilst the three age bands between 31 and 50 provided just over half (51%) of responses.

58% of respondents indicated that they do not suffer from any long-term illness; 12% indicated suffering from mental health issues; and a further 10% indicated suffering from a physical illness or health condition that affects their ability to travel.

Overall, there was almost an even split between male and female respondents, with 48% of responses submitted by male participants, 51% by female participants and 1% by non-binary participants.

Geographically, most responses were submitted by residents of the City of Glasgow (16%) Local Authority area, with a further 8% from residents of the City of Edinburgh. Fife residents provided a further 6% of responses, while the remaining responses were submitted by residents spread across the country.

Over a quarter (28%) of respondent households earn between £20,000 and £34,999 a year whilst around a fifth (19%) earn between £35,000 and £49,999, and around a further fifth (22%) earn under £20,000 a year. Half of all responses were provided by those in full-time employment, while a fifth were provided by retirees.

In terms of car ownership and ownership of National Entitlement Cards, 67% of respondents have access to a car for their own personal use, whilst a further 8% have shared access to a car with other members of their household. 15% of respondents are NEC card holders, while nine percent are Young Scot card holders. A further nine percent are blue badge holders.

Challenges & Limitations

These interim results are heavily caveated in terms of the limitations imposed on the survey distribution, the time available to survey the public, and understanding the scope of ScotRail’s additional research. In combination this impacted on the first wave of the public survey. The full impact of the pilot can only be assessed fully over the full nine month period. Learning has been taken from this interim assessment and will be used to refine the methodology for the final evaluation.

Understanding the Scope

The initial focus of research undertaken by Transport Scotland was to fill any primary data collection gaps, including surveys with non-rail users to capture consideration of new trips and/or additional trips that would otherwise be taken by another mode i.e. private car. As the research developed, the scope broadened to include existing rail passengers who do/and do not change their behaviour. This provided the data required to make an assessment on the pilot’s impact.

This resulted in an additional survey being conducted with the public alongside ScotRail’s survey with its customer base. Transport Scotland worked with ScotRail to ensure survey questions provided similar data outputs and where possible, avoided duplication.

The survey comprised of a number of questions on how often people travelled; the purpose of their trip; satisfaction with rail services; overcrowding/capacity on trains, and if they had switched from a different mode. As well as asking non-rail users about the barriers they faced.

Fieldwork Design

The fieldwork started in December 2023 and the survey was open for four weeks. This month typically sees more leisure trips in the lead up to Christmas and fewer commuting trips.

Despite this, a representative sample, covering all target groups identified was achieved. Group 3, non-rail users who have changed their behaviour as a result, has the smallest sample, but is also the hardest to reach group (as it is the smallest in actual size). For the next Wave, there will be more time to increase the sample size to understand the behaviour of this particular group.

Assessment of Value for Money (VfM)

The methodology for assessing the Value for Money (VfM) of the policy change was developed specifically for the pilot. Normally, appraisal of transport interventions is undertaken before they are in place, using standard tools and techniques to predict the impact of options and following delivery. This is followed up by evaluation of outputs and outcomes – did the appraisal accurately capture the actual outcomes of the intervention.

For the pilot of the removal of peak fares, an almost unique situation was faced in that a “natural experiment” is being undertaken i.e. the purpose of the pilot is to specifically test the impact it has on behaviour. This allowed a different approach to be taken. As is standard practice, the predicted outcomes of the pilot were assessed but the trial means that these predicted outcomes can now be adjusted to see what the actual impact is (see below). This is particularly useful in the context of the changing patterns of demand post pandemic and the fact that such a significant change in fares is unusual.

A detailed discussion of the VfM methodology will be published alongside the final evaluation report but simplistically the approach was as follows:

• Use the Transport Model for Scotland (TMfS) to assess the impact of the pilot with the expectation that the model would not necessarily capture the actual impact.
• Calibrate the results of the model to the actual impacts of the pilot, check that the detail of the results matched what was seen in reality and use these calibrated results to estimate the Value for Money in the usual way.

Simplistically, whist existing rail users benefit from reduced fares, the full impact of the pilot is measured by capturing the value to those who switch to rail from other modes or undertake additional journeys. The approach captures the wellbeing (or welfare) gained by those who switch to rail as well as the impact on existing users, the loss of revenue from existing users (and the gain from new users) to ScotRail, the impact on bus patronage, as well as an estimate of CO2 emissions impacts from changes in car use.

The model runs on an annual basis so the inputs are adjusted to reflect this. For example, the initial additional costs incurred by ScotRail are annualised.

As such, the assessment of VfM for the pilot combines the appraisal and evaluation methodologies within Scottish Transport Appraisal Guidance (STAG) to give the most accurate possible assessment of the impact. However, it should be noted that this paper deals with interim results over 11 weeks of the pilot (in VfM terms) and significant further work is required once the pilot is complete to produce a fully rigorous assessment of the impact. It would be unwise to place significant weight on these interim results.

Of particular importance is the establishment of a robust “counterfactual” – what would have happened if the pilot had not taken place. This is difficult over the interim period due to significant weather disruptions in October and November and the impact of the Christmas and New Year holiday period in December and early January especially when combined with changing patterns of travel demand – more leisure and fewer commuting trips, post-pandemic. Significant further work will be undertaken to ensure that the counterfactual used over the full extent of the pilot is as robust as possible.