Larkhall – Milngavie Railway Project Evaluation Study Final Report
3. Process Evaluation
3.1. Under STAG guidelines, the Process Evaluation is concerned with how well a project was implemented. Its aim is to establish which aspects of the project went well and which could have been improved upon. The lessons learned from the Process Evaluation can then be used both to improve the current project to ensure it runs more successfully going forward and to inform the implementation of future projects.
3.2. For the Larkhall – Milngavie railway project, the current guidance was not in place at the time the Process Evaluation should have taken place. In 2008 however, a Lessons Learned exercise[8], effectively a Process Evaluation, was conducted by the Nichols Group. This involved a review of the project's implementation and identified successes, issues and challenges. This was achieved through interviews with the key project personnel from the project delivery partners and stakeholders: Scottish Executive, SPT, Network Rail, Carillion Rail, ScotRail and South Lanarkshire Council.
3.3. The report noted that on the whole the implementation of the project was a success:
- the project was delivered on time;
- outturn cost was within 3% of the planned budget;
- the adverse impacts on train performance during implementation was less than anticipated; and
- passenger demand has exceeded forecasts.
3.4. However, a number of issues were identified which led to a set of comprehensive recommendations.
3.5. For this evaluation study, the objective was not to repeat these recommendations. Instead, the process involved a review of those findings and providing supplementary insight into the key project issues through further interviews with personnel involved in the project. Interviews were conducted with:
- Network Rail;
- SPT;
- Transport Scotland; and
- South Lanarkshire Council.
3.6. ScotRail were also contacted but decided there was no-one remaining within their team who had been sufficiently involved in the project to contribute (the Franchise Manager at the time of the project had since retired). It is noted however that ScotRail contributed to the 2008 Lessons Learned exercise.
3.7. Interviews were conducted in person, over the phone or by completion of a questionnaire, with the aim being to establish:
- what had worked well during the project?;
- what had worked less well?;
- what were the key project issues?;
- if the project were to be repeated, how might the delivery process be improved or refined?; and
- are any of the lessons learned from the Larkhall – Milngavie project already being put into practice?.
3.8. The findings from the first interview conducted with Network Rail generated a number key themes around which to structure subsequent interviews:
- Project Team & Project Management;
- Project Objectives & Strategy;
- Stakeholder Management;
- Risk Management; and
- Change Control.
3.9. In general, the themes discussed and issues raised were consistent with the previous Lessons Learned report. This chapter should be read in conjunction with that report.
Project Team & Project Management
3.10. Network Rail project managed the infrastructure works and appointed their own team to undertake this role. ScotRail project managed the operational elements. SPT had overall programme management responsibilities for the two workstreams.
3.11. Feedback from some respondents suggests there was a lack of clarity of project and programme management responsibilities, and that reporting lines were not sufficiently defined. It is also unclear whether Project Management Plans were prepared and maintained to ensure the project was well-executed, monitored and controlled.
3.12. During the implementation, no project office involving the three key organisations (SPT, ScotRail and Network Rail) from which to co-ordinate the project existed. Network Rail did have their own project office which provided the project manager and supporting staff. Feedback from interviews suggests that this separation of key personnel reduced the potential for collaborative working and led to slow resolution of issues.
At the outset, project and programme management responsibilities and reporting lines should be clearly defined. Consideration should be given as to which organisation is best placed to manage the delivery of the project, in particular in managing the risks associated in delivering rail projects.
Project Management Plans should be created and maintained to ensure that the project is well executed, monitored and controlled.
A multi-organisational project office should be established to house key personnel and facilitate collaborative working.
3.13. As noted in section 2.12, quantitative project objectives or success criteria were not defined at the outset, and little consideration was given to how the project's benefits would be monitored and reported.
3.14. Feedback suggests that efforts at a senior level were too focussed on time and budget rather than project outcomes, and that the focus on the outcomes was dominated by the engineering and infrastructure aspects, rather than on the desired train service and operational outputs. In effect, the engineering works 'took over' and the purpose of the project (i.e. the delivery of a robust and resilient train service was a secondary consideration).
The project strategy should be clearly defined at the outset, with quantitative project objectives and key success criteria set. As well as delivering the project on time and budget, there should be focus on ensuring that the desired project outcomes are achieved.
Project strategy and objectives should have the buy-in from all project stakeholders.
3.15. Although no formal stakeholder management plan was developed, the main stakeholders were identified and legal agreements were put in place.
3.16. However, issues were encountered when the new stations were handed over from Network Rail to ScotRail and South Lanarkshire Council refused to accept the re-fettled road bridges from Network Rail.
3.17. Feedback suggests that this was caused by failure to meet stakeholders' expectations, rather than quality control issues. Neither ScotRail nor South Lanarkshire Council were sufficiently engaged at an early enough stage, so their sight of initial designs was limited. Furthermore, there was no representation from ScotRail and South Lanarkshire Council at a Project Board level and so there was no opportunity to influence the project outcomes.
A formal stakeholder management plan should be in place with stakeholders engaged in the process and their input sought from project inception onwards in order to minimise the acceptance risk at a later stage in the project.
3.18. A risk management plan was developed and was perceived as being generally well-managed. However, a key issue raised by Network Rail was that of risk ownership. It is widely agreed that too much risk was placed under Carillion's responsibility who were not in a position to effectively manage the risk when it materialised (e.g. relocation of key utilities). This was exacerbated by a lack of an implementation agreement between Network Rail and SPT which could have reduced the risk to Carillion prior to them being engaged.
Allocation of project risks should be carefully considered with a greater emphasis on risk-sharing and ensuring the party accepting the risk is best placed to manage it.
A more comprehensive risk assessment should be undertaken to better pre-empt risks in particular those relating to public utilities, ground condition and mining issues which can create major engineering problems.
3.19. Although there were no major changes to the basic scope of the infrastructure project, it is evident that change requests could have been better managed and was one of the major issues throughout the project.
3.20. Feedback suggests that Network Rail were given little flexibility to vary the contract with Carillion. Any changes required SPT's prior agreement and often required the implications of the change to be comprehensively costed. This frequently caused delays and impacted the smooth delivery of the project.
To avoid project delays, a less onerous and restrictive change control process should be considered with a higher change threshold set so that only changes having major cost and time implications require detailed scrutiny.
3.21. The evidence gathered from the Lessons Learned exercise conducted in 2008 and the supplementary interviews carried out for this evaluation suggests that on the whole the delivery of the Larkhall – Milngavie project was a success.
3.22. It is also noted that the Larkhall-Milngavie project was the largest rail infrastructure implemented in Scotland for over 30 years and so few of the personnel involved had experience of delivering such a large-scale project. Taking this into account, the success of the project is even more commendable.
3.23. From the interviews conducted during this evaluation, there were however a number of learning points which should be borne in mind on future rail investment projects including:
- project and programme management responsibilities, and reporting lines should be more clearly defined;
- collaborative working should be promoted, perhaps facilitated by a multi-organisational central project office;
- project strategy, objectives and success criteria should be better defined with the buy-in of all stakeholders;
- formal engagement with stakeholders from the project outset;
- major risks should be more carefully considered and risk-sharing mechanisms adopted; and
- a less onerous change control process should be established to ensure the delivery of the scheme is not unnecessarily delayed.
3.24. It was also noted that several of these learning points are already being put into practice. For example:
- on the Borders rail project, Network Rail are now adopting a more transparent and shared approach to risk assessment which has ensured better price certainty; and
- in direct response to the issues experienced on the Larkhall – Milngavie project, there was greater expenditure on mining remediation works on the Stirling – Alloa – Kincardine, Airdrie to Bathgate and Borders rail projects. Whilst this meant greater upfront costs, it mitigated the risk of higher costs and project delays at a later stage in the project.
3.25. A further recommendation relates to the timing of the Process Evaluation. The current guidance was not in place at the time of the Larkhall – Milngavie project was conducted on completion in 2005 so no Process Evaluation was conducted following project completion. By the time the 2008 Lessons Learned exercise was conducted, it was inevitable that some project personnel had moved organisations and were unavailable to participate. Additionally, much of the project documentation had been archived and was difficult to obtain. This highlights the need for the Process Evaluation to be conducted soon after project completion. This will ensure that the key individuals involved in delivering the project are still available to provide their input and that all issues encountered can be easily recalled and recorded.
To ensure all evidence is captured and that the relevant personnel and still available to participate, the Process Evaluation should be conducted as soon as possible after project completion and ideally within six months.