Section 16

STAG Technical Database

Section 16

Evaluation

November 2015

Transport Scotland

Once printed or downloaded this document is considered to be uncontrolled. For the current version refer to the Scot-TAG section of the Transport Scotland website.

Version History

Changes since STAG Refresh, May 2008

Change number	Section updated	Date
1	Entire Section 16 updated to reflect recent STRIPE guidance	December 2013
2	References to Section 17 amended	May 2014
3	Reference to the Rail Evaluation Guidance	November 2015

Contents

16. Evaluation
16.1 Introduction
16.2 Evaluation Plan
16.3 Proportionality
16.4 Evaluation Elements and Timing
16.5 Process Evaluation
16.6 Stage 1 Outcome Evaluation
16.7 Stage 2 Outcome Evaluation
16.7.1 From inputs to impacts – Logic Model
16.7.2 Establishing what is to be Evaluated
16.7.3 What would have happened otherwise - Defining the Counterfactual
16.7.4 Assessing the success of the project
16.7.5 Value for money assessment
16.7.6 Data sources
16.8 Reporting

16. Evaluation

The guidance which follows relates to the evaluation of all transport projects. Specific guidance exists for trunk road projects costing more than £5 million and rail projects. Guidance on the specific issues associated with the evaluation of trunk road projects with a cost greater than £5 million is contained in Transport Scotland's Scottish Trunk Road Infrastructure Project Evaluation (STRIPE) guidance, published on Transport Scotland’s website. Rail Evaluation Guidance, also published on Transport Scotland’s website, sets out the key issues to consider and suggested steps to follow, drawing on case study examples where possible. Both STRIPE and the Rail Evaluation Guidance follow the same broad principles of evaluation as outlined in this chapter.

16.1 Introduction

Evaluation forms an essential part of the policy cycle, demonstrating what has been achieved with public resources and providing evidence and learning points for future interventions and investments. As such, the Scottish Government and Transport Scotland require evaluation to be undertaken and documented for any project for which it provides funding or approval.

Evaluation is a specific post-implementation event designed to identify whether or not a project is performing as originally intended, whether, and to what extent, it is contributing to established policy directives and whether the implemented project continues to represent value for money.

Evaluation should comprise a robust analysis, conducted in a similar manner as appraisal. However, whereas appraisal takes place prior to implementation and requires a comparison to be made between forecasted outcomes of the intervention and forecasted outcomes of a ‘do-minimum’ scenario, evaluation takes place after implementation and involves comparison of actual outcomes with a counterfactual.

16.2 Evaluation Plan

An Evaluation Plan should be developed at an early stage to outline the general boundaries of the proposed evaluation including:

questions it seeks to answer;
resource requirements;
data collection requirements;
provisional timing and cost; and
who should be consulted.

Stakeholders should be consulted during the development and implementation of the Evaluation Plan. The support and acceptance of the Evaluation Plan by stakeholders is essential to ensure performance can be effectively evaluated and that the integrity of the process of STAG is maintained.

16.3 Proportionality

The thoroughness of an evaluation should reflect the scale of the project. Large scale, high impact, high investment projects, such as a new rail line, should therefore undergo a more rigorous evaluation than smaller scale projects, such as minor enhancements to a station.

16.4 Evaluation Elements and Timing

An evaluation should comprise three elements: Process Evaluation; Stage 1 Outcome Evaluation; and Stage 2 Outcome Evaluation.

Process Evaluation is conducted at an early stage in the life of a project and is primarily concerned with how well the project has been implemented. It may also be known as ‘Formative Evaluation’ or ‘Lessons Learned’. The precise point in time for Process Evaluation has to be judged carefully to ensure that sufficient evidence is available to assess performance but also to ensure that evidence is captured while relevant staff and stakeholders are still in post. To ensure a balance between the two, it is considered that Process Evaluation should take place approximately one year after opening. More information is provided at section 16.5.
Stage 1 Outcome Evaluation is also conducted at an early stage in the life of a project and aims to provide a high level, early indication of project performance against targets, drawing on monitoring information available. It is considered that Stage 1 Outcome Evaluation should take place alongside Process Evaluation one year after opening. More information is provided at section 16.6.
Stage 2 Outcome Evaluation is conducted once the project has been in existence for a sufficient period to enable a comprehensive examination to be undertaken of actual performance against identified targets. If undertaken too soon, final impacts may not have had time to “work through”, but if undertaken too late, evaluation results may lose relevance. To ensure a balance between the two, it is considered that Stage 2 Outcome Evaluation should take place approximately three to five years after opening. More information is provided at section 16.7.

16.5 Process Evaluation

Process Evaluation is concerned with how well a project has been implemented. It may also be known as ‘Formative Evaluation’ or ‘Lessons Learned’. The aim of Process Evaluation is to determine which aspects of a project went well and which could have been improved upon. The learning points taken from Process Evaluation can then be used both to improve the current project to ensure it runs more successfully going forward and to inform the implementation of future projects.

Particular issues which may be worth examining in a Process Evaluation include: project selection and planning; the application and funding process; the way in which funds are allocated; and the management of the project at national and local levels. It may be useful to undertake a series of interviews with those involved in the project and relevant stakeholders as part of this process. Interviews should be carried out in a constructive manner to encourage openness and honesty.

Process Evaluation should also highlight issues for the future Stage 2 Outcome Evaluation, including the extent to which the information being produced by the Monitoring process is likely to be adequate for subsequent evaluation needs.

16.6 Stage 1 Outcome Evaluation

The purpose of a Stage 1 Outcome Evaluation is to provide a high level assessment of the extent to which a project is on track to meet its objectives.

Particular issues which may be worth examining at Stage 1 Outcome Evaluation include: usage/patronage; journey times; journey time reliability; and abstraction/displacement from other modes and routes. An assessment of outturn cost versus predicted cost as well as the number of jobs supported through construction work may also be useful at this stage. Stage 1 Outcome Evaluation may also provide an opportunity to assess mitigation measures to prevent, reduce, remedy or offset any emerging adverse impacts of the project.

16.7 Stage 2 Outcome Evaluation

The purpose of a Stage 2 Outcome Evaluation is to assess the extent to which a project has met its objectives and whether it remains value for money.

The process used in a Stage 2 Outcome Evaluation may be set out as a series of sequential steps, as follows:

Outline project rationale and define scope and purpose of evaluation;
Establish what is to be evaluated;
Establish the counterfactual for comparison;
Assess the success of the project against Transport Planning Objectives, STAG criteria and other relevant policy objectives by comparing actual outturns with target outturns and counterfactual outturns;
Assess value for money; and
Report and make recommendations.

Where necessary, further information on these steps is provided in the paragraphs that follow.

16.7.1 From inputs to impacts - Logic Model

As a starting point, it may be helpful to map out the Logic Model of the project. A Logic Model describes the rationale for a project through linking the intended outcomes (both short and long-term) with the policy inputs, activities, processes and theoretical assumptions. Generally, a Logic Model will identify the following elements:

issues being addressed and the context in which the project takes place;
inputs (i.e. what is being invested in terms of resources and activities);
initial outputs (i.e. target groups reached, tracks built, products developed);
outcomes (i.e. short and medium-term results such as changes in traffic flow levels and modal shifts);
impacts (i.e. long-term results such as better quality of life, improved health and environmental benefits); and
the assumptions made about how these elements link together.

16.7.2 Establishing what is to be Evaluated

A key initial step in any evaluation is to establish exactly what is to be evaluated and how outturns can be measured. The objectives and targets of the project should be clear, and quantified where possible, to allow comparison with actual outcomes and outputs. The availability of output and performance measures and other monitoring data (and how they relate to the objectives) should be reviewed. If this information is inadequate, consideration should be given to the collection of additional data.

16.7.3 What would have happened otherwise - Defining the Counterfactual

The crux of any evaluation is determining to what extent any change in the outcomes or outputs monitored are a result of the intervention, as opposed to other external factors. This can be determined by comparing actual outcomes with those of a counterfactual, which is the most likely transport situation over the course of the period, had the project not gone ahead.

Establishing the counterfactual requires a degree of judgment. However the sources and methods noted below can assist in determining what would have happened in the absence of the intervention:

The original business case or appraisal should define and include estimates of the ‘do minimum’ case which will often form the basis of determining the counterfactual.
The use of control and comparison sites may help establish whether any changes associated with a project have occurred directly because of the intervention. However, it may prove difficult to find a sufficiently similar site which provides a useful and fair comparison.
Establishing and collecting data on the baseline (the ‘before’ position) and comparing with data collected after project completion (the ‘after’ position) may also assist in establishing the impact of a project. However, it is important to bear in mind that any change between the ‘before’ and ‘after’ positions may be influenced by factors other than the project itself. To ensure a baseline can be established, data requirements need to be considered at an early stage before a project commences to determine whether existing surveys will provide the information needed or whether primary data collection is required.
The collection of primary data (such as interviews or surveys with passengers on a new train line or ferry or cyclists using a new path) can help establish what individuals would have done in the absence of the intervention.

16.7.4 Assessing the success of the project

To assess the success of the project in achieving its objectives, we compare the actual outturn(s), quantified where possible, with:

the target outturn; and
the counterfactual.

Evaluation should be undertaken against indicators and targets derived from:

the Transport Planning Objectives;
the five STAG criteria (environment, safety, economy, integration and accessibility/social inclusion); and
other relevant policy objectives of the particular project.

The evaluation should utilise information gathered for monitoring purposes during the project as well as existing secondary data where appropriate. However, particularly for large schemes involving significant investment, it is also likely to require primary data gathering and analysis that is particular to the evaluation itself. See section 16.7.6 for further detail.

16.7.5 Value for money assessment

To assess whether the project continues to offer value for money, outturn costs and benefits should be compared with anticipated costs and benefits.

16.7.6 Data sources

An evaluation should utilise existing secondary data sources relevant to project and STAG objectives. Depending on project objectives, this might include the following (please see section 17.2 for source references):

Rail

Passenger Focus’ National Rail Passenger Survey which assesses passenger satisfaction with rail travel in the UK.
Department for Transport’s National Rail Travel Survey which is a survey of passenger trips on the national rail system in Great Britain on weekdays.
Office for Rail Regulation’s Data Portal which collects together a selection of secondary rail data sources.
Latest Earnings Networked Nationally Overnight (LENNON) rail ticket database which holds information on all national rail tickets purchased in the UK.
Identification of the type of rolling stock used can help assess CO₂ impacts.

Road (projects under £5 million)

Transport Scotland’s Reported Road Casualties Scotland, which provides data on road safety and accident numbers.
Transport Scotland’s Scottish Roads Traffic Database which provides traffic flow information.
Bluetooth and mobile phone tracking data.
Car ownership cost calculators.
Department for Transport’s Average Bus Operating Costs.
Local Authority automatic traffic counter data.
Local and Strategic Trunk Road Authority event, accident and incident logs.

Ferries

Ferry Operator data which can provide information on patronage, journey times, fares etc.
Transport Scotland’s Scottish Ferries Review Household Survey which gathers views of all island and peninsula communities served by ferries in Scotland to provide data on frequency of travel, mode of travel, reason for travel and the experiences of ferry users.

Cycling

Sustrans’ Hands Up Scotland Survey which provides a comprehensive oversight of children and young people’s journeys to school at a local authority level.
Scottish Household Survey (Travel and Transport chapter) which provides data at a local authority level on a variety of data including the proportion of trips made by Scottish adults by bicycle. It also contains a Travel Diary component which collects data on travel patterns.
Transport Scotland’s Reported Road Casualties Scotland which provides data on road safety and accident numbers.
Sustrans and Local Authorities provide information on the number and overall length of 20 mph streets in Scotland, the length of the National Cycle Network in Scotland and the lengths of cycle routes in local areas.

Multi-modal

Transport Scotland’s Scottish Transport Statistics which presents a comprehensive statistical picture of transport activity and covers a wide range of topics.
Transport Scotland’s Land-Use and Transport Integration in Scotland (LATIS) which has a robust database of transport, land-use and demographic data which is linked to a multi-modal transport and land-use model.

Non-transport

National Records of Scotland’s Small Area Population Statistics which provides estimates of population at the local authority level.
Scottish Government’s Labour Market Statistics which provide data for Scotland on a variety of labour market indicators including employment, unemployment, inactivity and earnings. Data is sourced from a variety of Office for National Statistics publications including Annual Population Survey, Labour Force Survey and Annual Survey of Hours and Earnings
Office for National Statistics’ Nomis portal which provides access to data on employment, unemployment, inactivity and earnings from a range of sources including Annual Population Survey, Labour Force Survey, Business Register Employment Survey and Annual Survey of Hours and Earnings.
Office for National Statistics’ Regional Accounts which provides estimates of Gross Value Added (GVA) at local authority level.
Office for National Statistics’ Business Demography publication which provides information on business births through analysis of VAT/PAYE registration statistics.
Scottish Government’s Businesses in Scotland publication which provides information on the business stock at local authority level.
Registers of Scotland’s Property Statistics and Scottish Government’s Housing Statistics for Scotland publications which provide information on a range of housing statistics including housing prices.
Local Authority planning permission applications
Local Authority road accident numbers (Key Reported Road Casualties Scotland)

For large schemes involving significant investment, primary data gathering and analysis that is particular to the project itself is generally required. Depending on project objectives, this might include the following:

On-board or at station surveys to collect information such as journey purpose; journey patterns in absence of new service; potential agglomeration impacts; whether an individual has moved job or house as a result of the new service; whether a business has found new markets as a result of the new service; and whether a business has access to an increased supply of labour as a result of the new service.
Manual or automatic traffic counts to assess congestion and carbon emissions and, in the case of rail evaluations, to assess how individuals are travelling to and from the station (e.g. by car, bicycle or foot).
Surveys/interviews with businesses in the relevant area to assess the impact of the project on firm location decisions, access to labour, proximity to suppliers and customers etc.
Surveys/interviews with local residents to ascertain the impact of the scheme on their lives and the community.
Travel diaries to collect in-depth information on travel behaviours before and after an intervention.
Focus groups to probe respondents on issues raised in earlier surveys or interviews.

16.8 Reporting

The project manager should prepare an Evaluation Report based on the Evaluation Plan. An Evaluation Report should take into account outputs from both the process and Outcome Evaluations undertaken.

The principal outputs forming the basis of the Process Evaluation Report will generally include:

A quantitative analysis of performance against the selected indicators and measures;
A qualitative review of factors underlying performance such as the quality of management, the provision of information, geographic factors and political factors; and
An interpretation of the above which offers comment on performance of the implementation of the project and draws lessons for on-going implementation and for the design, management and implementation of future projects.

The outcome Evaluation Report should set out and substantiate its findings on:

Whether the project achieved its objectives, fully or partially;
The reasons for and consequences of not meeting objectives;
A statement of the costs of the project;
A statement of those outputs and outcomes from the project which are assessed to be additional;
Analysis of performance measures indicators; and
Interpretation using the criteria of economy, efficiency, effectiveness and equity.

In summary, the Evaluation Report should show whether the project represents a good use of resources, whether value for money could be improved, and, if so, how best to achieve this.

The Evaluation Report should be disseminated to relevant stakeholders to ensure that any learning points are fed into future decision making.

For the purposes of the STAG Report it will only be necessary to provide detail of the proposed Evaluation Plan, outlining the scope and timing of the Evaluation to be undertaken together with an Evaluation Summary Table (EST) to be used to report outturn performance and impacts following implementation.