Larkhall – Milngavie Railway Project Evaluation Study Final Report
4. outcome evaluation
4.1. Transport Scotland's guidance explains that 'an Outcome Evaluation should be conducted once the project has been in existence for a sufficient period to enable an examination to be undertaken of actual performance against identified targets'.
4.2. To conduct the Outcome Evaluation, the project objectives ( section 2.12) were considered in turn and an assessment made as to the extent they have been achieved (see chapter 5). Performance against STAG criteria was also assessed (see chapter 6).
4.3. Additionally, the project was assessed in terms of the extent to which it generated additional wider benefits including the impact on the local economy and WEBs. These non-quantifiable benefits were considered, in SPT's view, to be more than sufficient to offset the deficit in the monetised appraisal. These are discussed in chapter 7 and 8.
4.4. The Outcome Evaluation methodology is discussed below.
4.5. To conduct the Outcome Evaluation, data was gathered and analysed to provide an evidence base to establish:
- before and after service levels at all affected stations (not just newly opened stations);
- impacts on the improved attractiveness of Larkhall and Kelvindale, including newly-attracted investment and development;
- improvements in accessibility, particularly for areas with low social inclusion;
- rail users' travel behaviour before and after;
- observed modal shift (e.g. whether rail users are now making fewer car journeys and, if so, the outturn reduction in vehicle kilometres and accidents).
4.6. To achieve this, a variety of techniques were employed, using a combination of primary and secondary research.
Primary Data – User Survey
4.7. It was felt that the most effective way to determine change in travel behaviour was to carry out an online survey with passengers to understand their travel patterns before and after the rail project's opening.
4.8. The User Survey is discussed in detail in Appendix A.
Primary Data – Business Survey
4.9. To understand how the station re-openings and service frequency improvements impacted the performance of local businesses by improving accessibility and access to the labour market, an online survey of local businesses was undertaken.
4.10. The Business Survey is discussed in detail in Appendix B.
Secondary Data
4.11. Several secondary data sources were used to inform the evaluation as shown in Table 1.
| Data | Source | Evaluation Application |
|---|---|---|
| Station Demand | Office for Rail Regulation (ORR) National Rail portal | Establish actual station demand (see Chapter 9 and 10) |
| Station Revenue | LENNON rail ticket database | Establish actual station revenue (see Chapter 10) |
| Population | General Register Office for Scotland | Assess impact of project on local socio-economic indicators ( Chapter 7) |
| Key Benefits Claimants | Scottish Neighbourhood Statistics | |
| Jobseeker's Allowance Claimants | ||
| House Prices | ||
| House Sales | ||
| House Building |
Accessibility Analysis
4.12. TRACC – a multi-modal transport accessibility tool – was used to assess how public transport journey times have changed since the completion of the project. The outputs from this assessment are presented in chapter 5.
4.13. As noted in the draft Rail Evaluation Guidance, it is imperative that any evaluation is able to attribute any change to the transport intervention. In doing so for this evaluation, the following areas were considered.
Counterfactual
4.14. The counterfactual is what would have happened in the absence of the project and can assist in determining the extent to which the project was responsible for any change in behaviour.
4.15. Establishing the counter-factual for this project was challenging given no baseline information was gathered such as user surveys to establish travel behaviour before the project was completed. Instead, in the User Survey ( Appendix A) conducted for this evaluation, to deduce how respondents would have travelled before 2005:
- users of the four new stations (Larkhall, Merryton, Chatelherault and Kelvindale) were asked how they would travel in the future if rail was not available from their nearest station; and
- users of other stations which benefited from a frequency improvement were asked how they would travel in the future if the current train service frequency was reduced.
4.16. This information was then used to deduce the level of generated demand and the level of demand abstraction from other modes.
Control Group
4.17. For station demand, to isolate the net impact of the project, the counterfactual was established by assuming that growth at impacted stations would have been as per actual growth at stations within a control group. The control group selected was a group of stations that were not impacted by the project, but are within the same locality such that economic factors that influence rail demand are otherwise similar. This is discussed in more detail in chapter 9.
4.18. For the impact on the wider economy ( chapter 7), data for the project study area was again compared against various control groups to help understand whether any impacts that have occurred locally within the project area have been directly due to the station re-openings and frequency improvements. As data for such impacts is typically at a more aggregate level (usually by unitary authority), it was more challenging to select control groups against which the only key differentiating factor has been the Larkhall-Milngavie project. While the approach in this context is not perfect, it is regarded as reasonable within the principle of proportionality emphasised in STAG.
4.19. Due to the localised impacts of the Larkhall – Milngavie project, results of the Outcome Evaluation are typically presented by the different areas affected by the project. For example, the results of the User Survey are presented according to the four separate project components ( section 13.17) to evaluate the localised impacts of the scheme and the impacts of the new station openings. For the impact on the wider economy ( chapter 7), results are typically presented at a unitary authority level (South Lanarkshire, East Dunbartonshire and Glasgow) or, if the data were sufficiently disaggregated, at a settlement level (Larkhall, Hamilton, Milngavie).
4.20. Although the focus is on local or intra-area impacts, where possible, inter-area or 'two-way street' effects are also assessed to evaluate the success of the project in opening up scope for new economic interactions between different regions. For example, the re-opening of Larkhall station may have benefited local residents by increasing their access to employment opportunities but it may also have benefited businesses beyond Larkhall as people are increasingly spending their money in non-local locations due to the improved transport links.