Larkhall – Milngavie Railway Project Evaluation Study Final Report
11. RECOMMENDATIONS FOR RAIL EVALUATION
11.1. This chapter sets out the recommendations to be considered for inclusion in Transport Scotland's Rail Evaluation Guidance. The basis for these recommendations has been primarily SYSTRA's experience in conducting this evaluation study and in particular the challenges experienced undertaking the different aspects of the study including the Outcome and Process Evaluations, conducting surveys and reviewing passenger forecasts.
11.2. A number of the recommendations are already included in the current draft guidance but are repeated here, either to highlight their importance or to refine the relevant guidance text. Others are potential new guidance which are not covered in the current version of the draft guidance.
Project Objectives
11.3. The current draft guidance states that projects should be assessed against their original objectives and against the five STAG criteria. One of the issues with the Larkhall – Milngavie evaluation was that only general project objectives were set (e.g. 'to offer social inclusion benefits for residents'). In the absence of quantitative or measurable objectives, it was difficult to assess the extent to which each objective had been met.
11.4. It is therefore recommended that 'SMART' project objectives are established prior to the initial appraisal and that these objectives are then used in the subsequent evaluation:
- Specific – objective should specify exactly what the project will achieve in unambiguous terms;
- Measurable – objective should be quantitative and measurable, against which the project can be evaluated in the future;
- Achievable – objective should have a realistic chance of success;
- Realistic – objective should have a clear purpose and benefit;
- Time-bound – objective should specify the timescales for accomplishment.
11.5. Objective-setting should involve all project stakeholders' agreement. Consideration must also be given to the information and data that will be required to effectively measure the project against each objective. This should include details of the method and frequency of the data collection.
11.6. Establishing achievable 'SMART' objectives is a fine balance. Whilst objectives should be challenging and stimulating, they should also be realistic and take into account forecasting uncertainties. Failure to meet objectives by a large margin could potentially be politically embarrassing and damage public confidence in future transport projects.
Control Groups
11.7. Due to the wide geographical coverage of the Larkhall – Milngavie rail project it was difficult to establish a reliable control group that was close enough to the project area to have been subject to the same economic conditions but not affected by the project's impacts.
11.8. The project control group should be identified at the project outset and measures put in place so that any data collection is carried out in the control group area, as well as the main project area, to allow a subsequent comparison between the two.
Baseline
11.9. As noted in the draft guidance, establishing the baseline is essential to be able to understand the behaviour of users before the project intervention.
11.10. A challenge for the Larkhall – Milngavie Outcome Evaluation was that no baseline was established. The User Survey tried to recreate the baseline by asking respondents what they would do if there was no station or if the rail frequency was reduced. However, how passengers would travel in the future is not necessarily the same as how they used to travel before the project was completed and so approach may not reproduce the 'Before' behaviour accurately.
11.11. It is recommended that the baseline is established at the initial appraisal stage, for example through surveys of the users of existing stations within the study area and non-users who currently use other modes. Users and non-users could be asked about their current travel patterns and how they are likely to react to the improvements, in particular whether they would switch to rail.
11.12. The data from these surveys would give a better understanding of likely demand abstraction and mode shifting that can be modelled and lead to more accurate and more evidence-based demand forecasts.
Secondary Data
11.13. The draft guidance highlights the importance of secondary data sources that could be used to inform evaluations of rail projects.
11.14. It is recommended that these data sources, the frequency of collection and the stakeholder responsible for collection and analysis are identified at the project outset or as part of the STAG appraisal process. How this data will be used to inform whether a particular objective is being met should also be clearly established.
11.15. For certain localised impacts (e.g. the opening of a new station), data on socio-economic indicators such as population and employment may not be at a sufficiently disaggregate level to accurately assess the impact of the project. In such instances, alternative data sources should be considered and a process established to gather it.
Primary Data
i. Survey Method
11.16. On-board and at station surveys are referred to in the draft guidance. Whilst these are an effective technique for collecting information, they can be time-consuming and expensive.
11.17. An alternative is the use of online surveys, as demonstrated with the User Survey undertaken for this evaluation study. These can be a more cost-effective alternative to on-board or at station surveys. Although it excludes people who do not have internet access, this is becoming increasingly less of an issue, as more people get access to the internet, through a combination of home computers and smartphones.
11.18. However, one of the issues encountered on the Larkhall-Milngavie evaluation was the low response rate to the online survey. To increase the response rate for future online surveys, it is recommended that alternative channels for promoting surveys should be explored alongside traditional advertising. In particular, advertising on social media such as Facebook and Twitter is likely to generate a higher response rate from those who use the relevant social media. Such channels have the advantage of users been able to access the survey immediately by clicking on the link as opposed to users seeing the advert and then accessing the survey later.
11.19. The main disadvantages of using social media to recruit respondents are:
- it is a less targeted approach and could generate a significant number of irrelevant responses from respondents living or working outside the study area; and/or
- it may annoy some members of the relevant social media networks who receive the request to participate but who have little or no interest in the rail scheme or the survey.
11.20. To avoid these problems, it is recommended adding QR codes[44] to the flyers and posters being used to advertise the survey in the study area. These can then be used to enable the members of the travelling public who have the relevant smartphone/tablet technology to quickly access the relevant online survey webpage, while ensuring that the set of respondents remains within the targeted study area.
ii. Survey Content
11.21. Regardless of the survey method used, it is recommended that rather than bespoke surveys being developed for each rail project, a standard 'best practice' survey template is developed that could be used each time an appraisal or evaluation study is undertaken. Surveys could contain a common core set of questions, supplemented by further questions tailored to the individual scheme.
11.22. It is also recommended that surveys maximise the information gathered by asking information about multiple frequent journeys (e.g. the Larkhall-Milngavie User Survey asked respondents about up to three of their most frequent journeys). This is useful because many passengers use rail on a regular basis for more than one purpose (e.g. commuting and leisure), but their behaviour following an intervention may differ according to their journey purpose. It is therefore useful to collect information on more than one journey.
11.23. Transport Scotland could use the format of the travel purpose questions used in this evaluation study as the starting point for an (electronic) survey template which could be provided with the evaluation guidance. However, it is likely some modifications may be needed to this questionnaire, to ensure that it adequately covers the generic needs of a typical rail scheme evaluation.
iii. Timing of Surveys
11.24. In addition to a survey to establish the baseline (see section 11.11), it is recommended that information on users' travel behaviour is collected following project completion, typically between six and twelve months from the project opening. This will help establish the extent to which travel behaviour of existing residents/employees of the area changes as a result of scheme, including the level of abstraction from other modes and stations.
11.25. However, this 'just after opening' survey would not capture lifestyle changes as a result of transport improvements (e.g. moving house, changing job). A further survey should therefore be undertaken between 3 and 5 years after the scheme's opening, to pick up these medium/long-term impacts.
11.26. This recommendation will obviously have cost implications for the evaluation process, so it may be desirable to exclude this follow-up survey for any small/local schemes deemed 'unlikely' to generate significant land-use changes. These smaller schemes are likely to be appraised using simple fixed-demand matrices, with no WEBs included within their appraisal, so there will be a corresponding reduction in the need to evaluate these WEBs within the evaluation process.
Process Evaluation
11.27. In the Larkhall-Milngavie 'Lessons Learned' exercise undertaken in 2008, it was noted that as three years had passed since the project's completion, many of the personnel involved in the project had moved organisations and so were not available to participate. Additionally, much of the project documentation had been archived and was difficult to obtain. This limited the findings from the exercise.
11.28. This highlights the need for the Process Evaluation to be conducted soon after project completion. This will ensure that most of the key individuals involved in delivering the project are still available to provide their input and that all issues encountered can be easily recalled and recorded.
11.29. It is therefore recommended that the Process Evaluation is conducted within six months of project completion.
Project Documentation
11.30. When conducting the Larkhall-Milngavie evaluation it was at times difficult to gain access to key project documentation. Furthermore, given the change in personnel since the project was completed, it was sometimes unclear whether the information provided was comprehensive or the latest version.
11.31. It is therefore recommended that on project completion, all key project documentation is archived and transferred to the stewardship of one organisation (probably Transport Scotland).
Demand Modelling
i. Choice of Demand Forecasting Methodology
11.32. For the Larkhall-Milngavie demand modelling, no rationale was given in the SPT Modelling Report for the modelling approach chosen.
11.33. It is recommended that before any modelling begins, an evaluation of the various modelling approaches available and the pros and cons of each one is undertaken. Clear justification for one approach over alternatives should be documented. Best practice guidance as to which approach is likely to be the most suitable for a given set of circumstances is provided in PDFH.
ii. Documentation of Forecasts and Assumptions Applied
11.34. Little information regarding exogenous assumptions used was given in the SPT Modelling Report. Furthermore, demand forecasts were only provided for the four new stations, and for the predicted opening year (2001).
11.35. It is therefore recommended that a modelling Record of Assumptions is produced. This should include:
- a detailed description of the modelling methodology;
- all exogenous factors assumed including their source;
- the levels of demand abstraction and mode switching assumed, including justification and/or evidence for the assumptions made;
- all key modelling parameters including annualisation and ramp-up factors;
- a clear description of the rail service pattern (frequency and journey times) and fares assumed; and
- any assumptions made about the Reference Case and Do Something changes affecting the competing modes (e.g. journey time, frequency and/or fares).
11.36. Additionally, a full suite of model outputs (ideally in a simple comma-separated variable or equivalent 'flat' format) should be prepared. This should include comprehensive annual demand forecasts for all stations impacted by any change i.e. all stations in the study area, and for an extended time period e.g. for 15 years post-implementation. This will facilitate a future comparison of the forecasts against actual demand.
iii. Sensitivity and Risk Analysis
11.37. To account for the impact of potentially volatile drivers such as economic growth on demand forecasts, it is best practice to prepare a range of forecasts to reflect the inherent risk in using a particular set of exogenous drivers. In addition to the 'Central' forecast (based on the most likely or mid-range scenario), typically two sensitivity tests should be performed: 'Low' and 'High'.
11.38. The 'Low' sensitivity test would for example take the more pessimistic forecasts available for GDP, employment and population growth; the 'High' would take the more optimistic forecasts. These could be either drawn from different forecasters e.g. Experian, Oxford Economics or CEBR, or by adjusting the 'Central' forecasts e.g. +0.5% and -0.5% per annum for the 'High' and 'Low' scenarios respectively.
11.39. In addition to sensitivity testing the main exogenous drivers, tests should also be conducted around other factors that may influence the demand predicted for the scheme. For example:
- with and without park and ride facilities;
- the inclusion of a significant competitor response (from bus/coach operators);
- if the forecast demand at a particular station is significantly influenced by additional local housing or commercial development, what would be the impact if this development does not occur or is on a smaller scale than assumed in the core forecast?