3. Process Evaluation
3. Process Evaluation
3.1 STAG explains that an important aspect of the Evaluation Plan should be a process evaluation. A process evaluation is concerned with how well the project has been implemented and delivered, with a view to identifying what went well and what lessons could be learned for the planning, implementation and delivery of future projects. In this specific context, it is intended that the findings of this process evaluation will feed into Transport Scotland’s appraisal and evaluation guidance for future rail investment projects.
3.2 The process evaluation was informed by a series of consultations with the project delivery partners and stakeholders, all of whom were involved in the implementation and delivery stages of the project. This included meetings with:
- Aberdeenshire Council;
- North East Scotland Transport Partnership (NESTRANS);
- Transport Scotland; and
- Network Rail.
3.3 The remainder of this chapter sets out the findings of the process evaluation. It also includes a series of key learning points, set out in grey boxes, which have been gleaned from the exercise and can be used to inform future evaluations. The main focus is in the areas of :
- Timing of Process Evaluation
- The Project Team;
- project scope;
- project management - budget, programme and risk management; and
- stakeholder engagement.
3.4 As explained in STAG, the process evaluation should be carried out at an early stage of the overall project. The benefit of this is that it will then be possible to amend a project early to make it more efficient and effective in terms of implementation and delivery. If left too late, then it may not be possible to change the delivery process and implement improvements.
3.5 Another benefit of conducting the process evaluation early is that the information is clear in the minds of those who have been involved. This is important if lessons learned are to influence future projects. If left too late then key lessons could be forgotten and missed. Perhaps more importantly, if left for a period of time then it is possible that those that were involved in the delivery stage could have moved on. During this process evaluation it was fortunate that those that were involved in the delivery stage were still employed in Government and were able to participate in the interviews and feed in their valuable comments and experience. While some no longer worked in Transport Scotland, they were still employed in the Scottish Government. Under other circumstances they could have left the organisation and it would not have been possible to glean their input. This would have made the process evaluation very much incomplete with important findings missed.
3.6 To maximise value from the process evaluation it is important that it is carried out at an early stage of a project. The exact timing will vary depending on the size of the project, but for small projects it should be no more than half way through the proposed delivery programme and for larger projects, with a lengthier delivery programme, this should be earlier. This will allow lessons to be learned early and changes / improvements made to the delivery process to reflect the findings. This will also ensure that individuals involved in the delivery phase can feed into the exercise and their experience recorded.
3.7 The chart below shows the team structure for the project to reopen Laurencekirk station:
Figure 1. Laurencekirk Station Reopening Team Structure
* DB Schenker and EWS (English, Welsh and Scottish Railway) are rail freight operators, although the former acquired the latter in 2007.
3.8 As noted in the introduction, the commitment to proceed with the reopening of Laurencekirk railway station was based on the STAG appraisal undertaken prior to the formation of Transport Scotland. Transport Scotland took over the project from Aberdeenshire Council when the agency was formed in 2005. The handover of the project to Transport Scotland occurred at the end of Network Rail’s ‘Governance on Rail Investment Projects’ (GRIP) 3 stage, the point at which a single preferred option to be taken forward is identified.
3.9 The role of Transport Scotland as project lead (or Promoter) was seen by those interviewed to be an important reason for the successful implementation of the project. As a delivery agency, Transport Scotland had a wide range of skills in-house and its scale meant that it was better able to mitigate the risks and liabilities associated with a new-build station.
3.10 At the outset consideration should be given to which organisation is best placed to manage the delivery the project, in particular managing the considerable risks associated with rail schemes.
3.11 Transport Scotland was given a fixed level of investment approval and therefore had the challenge of bringing the project in on budget. Its role included managing public expectations and clearly defining the roles of individual delivery partners.
3.12 Network Rail was tasked with the delivery of the reopening of Laurencekirk station, Transport Scotland being the overall client / promoter. Network Rail progressed the project through its eight stage GRIP process.
3.13 Aberdeenshire Council was responsible for the delivery of the station car park, although both it only had a stakeholder role on the wider project. NESTRANS was a funding partner for the project.
3.14 A key question in the process evaluation was the extent to which the scope was mutually agreed by all parties, both at the outset and throughout the duration of the planning and build.
3.15 One of the initial challenges was that the Laurencekirk project brought together a series of partners who were bedding into new roles. NESTRANS became a statutory partnership in 2006, albeit it had existed as a voluntary partnership prior to that. Transport Scotland was formed in 2005 and was only beginning to develop relationships with Network Rail, which itself had only been created in 2002 following the demise of Railtrack. It is also important to note that First, the train operating company, had also only assumed control of the ScotRail franchise in October 2004.
3.16 As would perhaps be expected, it was explained that there was an element of uncertainty at the outset of the project, particularly with regards to funding. However, both Transport Scotland and Network Rail believe that the new arrangements were perhaps to the benefit of the Laurencekirk project overall, as they allowed a review of existing practices and a fresh approach, something which was reflected in the Implementation Agreement between Transport Scotland and Network Rail. Given the overall budget for the new station (around £4.25 million in 2009 prices), there was a view amongst the partners that the Implementation Agreement had to be proportionate to the project scale. It was also explained that this led to the development of what can best be described as a ‘letters / memo’ approach (based on short, succinct dialogue rather than lengthy contract documents), which set out the key Terms of Reference in a concise fashion. This proportionate Implementation Agreement ensured that the scope and Terms of Reference were appropriate for the scale and level of risk associated with the project. Indeed, Network Rail explained that this agreement is now the standard template for rail projects in Scotland of less than £10 million in value.
3.17 The development of a proportionate Implementation Agreement ensures that the scope is appropriate, clearly understood by all partners and is focused on project outcomes.
Engagement with Network Rail and the Operator
3.18 One important outcome stated by all consultees is to ensure that Network Rail and the operator(s) are integral to the project from its outset. Allowing these organisations early participation in the delivery process provided useful insight on a range of technical and operational issues that helped to maintain the delivery programme. It also helped in discussions with stakeholders by making the ongoing consultation transparent and informative.
3.19 It is helpful to have both Network Rail and the incumbent operator(s) as key stakeholders at the outset and part of the Steering Group
Construction of the Station Car Park
3.20 Network Rail explained that one of the key successes of the Laurencekirk station reopening was the decision for Aberdeenshire Council to assume responsibility for the car park build. First and foremost, as the local Roads and Planning Authority, Aberdeenshire Council had much of the expertise in-house to ensure that the car park was successfully delivered. However, in addition to this, procuring skilled civil engineers in the North-East can be challenging given the demand for labour from the oil and gas industry in particular. Aberdeenshire Council’s local frameworks ensured that the most appropriate civil engineers could be procured at an affordable price. This prevented specialist Network Rail civil engineers being redirected onto the construction of the car park. Network Rail also explained that the key underlying message here is that the correct delivery partners are assessed and appointed on each project.
3.21 The findings from the consultation suggest it is important to undertake an objective assessment of which organisations are best placed to deliver each aspect of the infrastructure required in the project.
3.22 The first major scoping issue faced by Network Rail was the land required for the construction of the access road. The land for the road was part of a strategic freight site leased by Network Rail to EWS, a rail freight operator. The site was used as a freight marshalling area by EWS and was on a 125 year lease (which had 109 years left to run). Network Rail worked closely with EWS to negotiate an agreed position on changes to the design of the site.
3.23 Overall, the consultees agreed that that there was a clearly defined scope for the project at the conclusion of the GRIP4 stage. This included restoration of the listed station building, construction of two platforms and other elements related to rail operations. From this point onwards, the scope was understood by all parties and formed an effective basis for the design and build.
3.24 It is important, for effective delivery, that there is a clear scope for the project and that this is fully understood by all necessary organisations.
Project Management - Risks, Budget and programme
3.25 It was explained that the reopening of Laurencekirk station was delivered on-time and below-budget. A number of consultees also explained that the budget was well scoped, understood and agreed at the outset of the study. The authorised budget for the station reopening was £4.25 million but the outturn cost was around £3.50 million, some £0.75m under budget.
3.26 The key factors identified in the successful delivery of Laurencekirk station, in terms of time and cost, were:
- controlled aspirations, which were realistic, well-scoped and informed by expert opinion; appropriate lead-in times;
- consistent communication of the programme and budget to Ministers, elected Local Authority members, the media and the public; and
- succinct, accessible and light touch reporting of progress and issues.
3.27 The consultees explained that there were a number of budget-related items that had to be resolved at the outset of the study, particularly related to the provision of the car park and the footbridge. The initial set of tenders for the study all came in with a price over the authorised budget. However, one of the claimed successes of the Laurencekirk project was the manner in which this issue was handled at the implementation stage.
3.28 Transport Scotland utilised its governance procedure to begin a ‘challenge process’, aimed at reducing overall costs. Every aspect of the project was assessed for potential savings related to value engineering. It was noted that there were two key successes in this regard:
- Aberdeenshire Council assumed control of the car park construction, which it was better placed to do given that it was the local roads and planning authority; and
- The proposed steel ramps at the station were designed to have a very shallow gradient, but they were not in keeping with the overall build, were very large and would cost around £1.2m, or a quarter of the allocated budget for the station. Through a process of engagement with local accessibility groups, Network Rail successfully secured a change to the ramp design, saving £0.70m, which, was approximately the amount of the project underspend.
3.29 An effective tender challenge process and a pragmatic review of value engineering solutions can help to identify cost savings on a project.
3.30 Network Rail identified an innovative approach to procurement which contributed to cost reductions. The contract for the reopening of Laurencekirk station was of a design and build format. With contracts of this nature, tenderers will likely price the risk of changes in the design (impacting on the build) into their bids.
3.31 However, Network Rail split the design and build process. Whilst the winning tenderer would secure the full design and build process, bidders were initially asked to provide a fixed price for the design work and outline cost for the build work. There was a break clause in the contract which allowed Transport Scotland not to proceed with the build should it chose not to (with a penalty clause to Network Rail of £66k). Once the design was finalised, the winning tenderer was then asked to provide a cost for the build based on an exact scope of work. It was claimed that this approach potentially saved a significant sum of money as it helped to address the scoping risks in a design and build contract.
3.32 Careful consideration should be given to the method of procurement. A procurement process specifically tailored to a project can help minimise risks, reduce costs and ensure the project is delivered on time and within budget.
3.33 Network Rail noted that there is a need for all partners to be kept up to date with concise and realistic briefing. In particular, Network Rail believes that an end / opening date should not be announced before detailed planning takes place, as the company has to build in a variety of factors, including track possessions. Unrealistic dates can drive unsafe behaviour and poor design practices. This was claimed to be one of the successes of the Laurencekirk project – expectations were effectively managed from the outset, which helped minimise unfair negative press for the project.
3.34 Detailed project planning should be undertaken before milestone dates are announced to the public. Announcements and press releases should also be communicated between, and jointly agreed by, the project partners.
3.35 It was explained that there was significant stakeholder engagement undertaken throughout the delivery phase of the project. Network Rail stressed the high importance of early engagement with stakeholders and the sharing of information. It was also explained that the Laurencekirk scheme had a high profile locally and involved building a station in a small town, mainly at night. It was explained that during that period not a single complaint was received.
3.36 The high level of engagement was enshrined in the Stakeholder Plan, which had been developed as part of the early GRIP and STAG processes. The plan involved public meetings, engagement with community councils and joint press releases with local representative bodies.
3.37 The consultees explained that there were very few issues with stakeholders. The programme of community engagement ensured that local people were well informed, although it was acknowledged that the small size of Laurencekirk also helped in this regard. It was also clear from the discussions that there was considerable and overwhelming local support for the reopening of the station which, it was also acknowledged, made the consultation process smoother and less challenging than could be faced in more controversial projects. Nevertheless, it was explained that a well-planned engagement and consultation process did contribute to the successful delivery of the project..
3.38 One general area of improvement that was suggested was more community consultation with the operator so as to understand the development of the train service timetable. For example, Laurencekirk to Stonehaven is seen as an important local journey but there are very few services that call at both stops. It was suggested there was limited opportunity throughout the implementation phase to discuss the timetable with the operator.
3.39 Extensive community engagement, as recommended in STAG, beginning early and continuing throughout the implementation stage, can be a key factor in the successful implementation of a project. It is a useful mechanism for sharing information with stakeholders. It also ensures openness and transparency and can help generate buy in from affected parties.
3.40 The timetable is a particularly important issue for potential users of the station and opportunities should be available to for them to discuss and feed in their views to the operator.
3.41 From the evidence gathered during the consultations, it can be concluded that the delivery of Laurencekirk station can be regarded as a success. The project was delivered on time and on budget. There were a number of factors that contributed to that success which could be learned for future rail investment projects. The most important factors can be summarised as:
- Careful consideration being given to who is best placed to manage risks and take overall ownership of delivery;
- A well thought out scope that is clearly understood by all parties;
- Procurement methods specifically tailored to meet the needs of the project; and
- A carefully thought out, well-designed and transparent stakeholder and community engagement process to generate participation and buy in.
3.42 It is useful to note in concluding, however, that it is important that a process evaluation is carried out early in the process. This didn’t happen in the case of Laurencekirk and it is fortunate that a number of important lessons were not missed due to, for example, the turnover of key and experienced staff involved in the delivery.