7. Recommendations for rail evaluation guidance

Overview

7.1 This chapter sets out the recommendations to be considered for inclusion in the development of Transport Scotland’s Rail Evaluation Guidance. It is split into two sections. The first section lists a number of recommendations, many of which are already highlighted in the Draft Guidance. The purpose is to re-emphasise these points and the importance of these factors.

7.2 The second section presents potential new recommendations which are not, as yet, covered in the Draft Guidance.

Recommendations Part 1 – factors already covered in Draft Rail Evaluation Guidance

To carry out an effective outcome evaluation it is crucial that relevant data is available relating to the period pre and post project opening. While, in some cases it may be appropriate to use officially published socio-economic time series data, in the case of evaluating the impacts of relatively small and / or rural rail projects these types of data are not necessarily available at that appropriate spatial level, and therefore a bespoke data collection exercise may be required.

At the outset, or as part of the STAG appraisal process, consideration therefore needs to be given to the information that will be required to effectively measure the performance of a project against its objectives. If the information is not readily available, then measures should be put in place, following the decision to proceed with the project, to ensure it is gathered. This should include details of the method and frequency of the data collection exercise.

An outcome evaluation may not be able to rely on official published sources to inform pre and post project impacts. This is particularly so for projects that are small / or in relatively rural areas as the socio-economic data may not be available at the required spatial level. Careful consideration of data requirements therefore needs to take place early so that if the information is not available then measures can be put in place to gather it.

The data required to undertake the evaluation should be detailed in the Monitoring Plan and these aspects should run right through the logic of the STAG process from the identification of problems to the setting of SMART objectives onwards, i.e. the SMART objectives should be defined together with a specification of how they will be measured right at the outset.

Fully understanding the behaviour of users before the intervention and the counterfactual are crucial to effective evaluations. The Laurencekirk outcome evaluation largely depended on the memories of users and their answers to the online survey to establish their travel behaviour before the station was reopened, and what they would have done had the station not reopened. Whilst this worked well, there are inherent risks with the accuracy and reliability of the responses. It is important that this baseline information is collected ahead of project implementation, or at the very least as quickly as possible after project opening i.e. data should ideally be collected in advance of the outcome evaluation and not as part of it.

The process evaluation should be undertaken during the implementation and delivery i.e. at the early stage of a project. This will ensure that issues are fresh in the minds of those involved so that lessons can be learned. It will also allow opportunities to improve processes that are not going well during delivery stage. Importantly, if carried out early it also means that the key individuals involved in delivering the project are still accessible to provide inputs to the evaluation.

For the outcome evaluation, there needs to be a ‘bedding in’ period to ensure the impacts expected to be generated by the project have sufficient time to materialise.

To allow a meaningful assessment of the performance of a project against the Transport Planning Objectives, the objectives should be SMART or supported by a number of specific targets or indicators – Transport Scotland should consider if further guidance should be provided to practitioners in the setting of meaningful and consistent SMART objectives which will ultimately feed the evaluation process.

Control groups should be considered and agreed as part of the appraisal stage, i.e. when the monitoring and evaluation framework is being determined, and measures put in place to ensure the relevant data is gathered to allow a comparison with the area directly affected by the project.

Recommendations Part 2 – factors not covered in Draft Rail Evaluation Guidance

Details of the option appraisal should be recorded clearly, particularly the methodology and key forecasting assumptions made, so that the outcomes of the scheme can be evaluated against what was anticipated, and the forecasting process can be more readily revisited when outturn data becomes available.

If the outcome evaluation is considering the impacts on local business performance, e.g. wider impacts analysis or WEBs, then baseline information should be gathered from local companies in advance of the project opening. Similarly to the user survey, this eliminates the risk of relying on the memories of people at a much later stage.

STAG sets out a requirement for a Monitoring Plan and an Evaluation Plan. In practice, these aspects of the process perhaps receive less resource and attention than the main appraisal. These two linked processes should be given a greater prominence as they are key to ensuring lessons are learned and therefore more effective projects in the future.

There were no Monitoring & Evaluation plans detailed in the Laurencekirk STAG but this STAG appraisal may have preceded this requirement. As such this is not a good example in terms of following a Monitoring & Evaluation Plan through from STAG appraisal to implementation to evaluation.

In the case of Laurencekirk there did not appear to be any firm commitment from organisations involved (funders or stakeholders) post STAG appraisal to undertake the necessary data collection outlined in Transport Scotland’s guidance to ensure effective evaluation. If evaluation, as required, is to become an integral part of the ROAMEF cycle, it needs to be seen as important, rather than an ‘add on’ and of secondary importance to the role of project delivery.

There appears to be an increasing public acceptance of Web-based surveys. In some circumstances, particularly rural and / or small projects, if correctly thought through and designed, these surveys can provide a cost effective way of collecting a lot of detailed data as demonstrated with the Laurencekirk survey. This can particularly be the case when there is strong local feeling towards a project.

In the case of small stations, the evidence from the Laurencekirk survey suggests that the Wider Economic Benefits (WEBs) impacts are likely to be minimal. While there was some evidence of the station influencing people’s decision to enter the workforce, the numbers were small. It is possible that larger stations may have greater impact. However, in the spirit of the proportionate approach to appraisal highlighted in STAG, it is recommended that WEBs impacts are not required to be captured as part of future appraisals of small station re-openings similar to Laurencekirk.

The approach to, and information required for, evaluating rail projects will differ depending on the nature and size of the project. For example, an evaluation of a small rural station such as Laurencekirk will obviously differ from a new rail service such as the forthcoming re-introduction of Scottish Borders. Important consideration needs to be given to the transport planning objectives when designing the approach to the evaluation.

Evaluations can require a significant resource and need to be considered within the principle of proportionality as highlighted in STAG.