Borders Railway Baseline Study - Final Report

3 Research Methodology

Introduction

3.1 As discussed above, the research included a primary data collection exercise consisting of a number of surveys. These included a Household and Business Survey directed at the Borders and Midlothian; a Control Area Survey aimed at East Lothian and Dumfries & Galloway; and a Panel Survey of residents of Edinburgh and East and West Lothian. In order to ensure the research can be repeated in the future on a like for like basis, it is important to document the methodology adopted. This Chapter therefore provides a detailed discussion of the approach for each element of the research including how the samples were selected and the survey methods used.

Household Survey

Content

3.2 The purpose of the Household Survey was to gather information on the current travel choices, patterns and behaviour of residents in the Scottish Borders and Midlothian ahead of the reopening of the line. The survey which is included in Appendix A included questions on:

demographic characteristics and location;
car availability;
intention to use the railway;
lifestyle choices made prior to the reopening of the line;
frequency of commuting to work and education and mode of transport used;
frequency of business and other purpose trips to Edinburgh and mode of transport used; and
willingness to take part in future research (to enable the creation of a panel survey at a later date).

3.3 In addition to the above, the survey also included a oneday Travel Diary. This utilises the Scottish Household Survey (SHS) Travel Diary methodology where by one (random) adult from each household is asked to complete a detailed record of travel undertaken the previous day. This provides a consistent oneday record of travel behaviour and allows for the early monitoring of the impacts of Borders Railway via the ongoing Scottish Household Survey (Travel Diary).

3.4 In addition, the travel diary included questions on journey times, fares and parking costs. If individuals completing these questions optin to future research, it will therefore be possible to compare their ‘outturn’ travel times and fares with the preopening values to derive a true estimate of the benefits accruing to these individuals. This could be used as part of the evaluation to estimate the actual, as opposed to that estimated for the purpose of the business case, benefit cost ratio and outturn value for money.

3.5 For the journeys identified in the travel diary, this includes:

if car to final destination, parking arrangements & costs – typical commuting time;
if car to P&R, location and PT fare – typical total commuting time;
if bus – typical costs and travel time; and
if active – typical travel time.

3.6 It should also be noted that respondents to the Household Survey were also asked whether they would like to take part in a follow up survey or future research on behalf of Transport Scotland and if so, to supply their contact details. Thus, while not all respondents indicated that they would like to take part, there is the potential to approach some of the same people as part of the future evaluation.

Sample Identification

3.7 The alignment of the Borders Railway is such that the residents of the Scottish Borders and Midlothian will be affected in different ways given the different travel behaviours and patterns. The railway line, for example, will have an impact on communities that are not necessarily close to the stations e.g. through the opportunity to use the park and ride facilities. In addition, not all residents of the Scottish Borders and Midlothian are likely to be impacted by the line.

3.8 To account for this TRACC accessibility planning software was used at the Census Output Area level to determine three distinct geographies (‘Tiers’) for the purpose of devising sample targets for the household survey. The three geographies identified were as follows:

Tier 1: areas where walkin access to the new stations is possible (<15minutes), taking account of the walking network, including off street, footpaths and any new active travel based links to the new stations;
Tier 2: excluding Tier 1, areas where reasonable busbased public transport access is possible (i.e. along bus routes serving the stations) – e.g. by bus within 15 minutes in both an AM and PM peak period; and
Tier 3: excluding Tiers 1 and 2, areas where only carbased access to stations is realistic (within 20 minutes), and the new stations will become the closest P&R option for accessing Edinburgh. For example, Penicuik residents would not be expected to use the Borders railway to access Edinburgh, despite being approximately 20 minutes’ drive from a station.

3.9 An image showing the geographic extent of each of these three tiers is shown in Figure 3.1 below.

Figure 3.1: Borders Rail Survey Catchment Tiers

3.10 The number of households in each tier was then used to determine a set of ‘target’ samples as shown in Table 3.1 below. Applying the Transport Scotland target rate of 1500 in each local authority gave a sample size of 6.5% for Midlothian and 8.7% for the Scottish Borders as shown under Target Sample 1. The totals were therefore weighted according to the population in each tier to achieve a 7.4% target sample in each local authority area as shown under Sample 2 below.

Table 3.1: Target Samples
	Local Authority	Tier 1 Walk	Tier 2 Bus	Tier 3 Car	Total
Total Households	Midlothian	4015	10967	8076	23,058
Total Households	Scottish Borders	3399	5760	8124	17,283
Target Sample 1	Midlothian	261 (6.5%)	713 (6.5%)	525 (6.5%)	1500
Target Sample 1	Scottish Borders	295 (8.7%)	500 (8.7%)	705 (8.7%)	1500
Target Sample 2	Midlothian	299 (7.4%)	816 (7.4%)	601 (7.4%)	1715
Target Sample 2	Scottish Borders	253 (7.4%)	428 (7.4%)	604 (7.4%)	1285

3.11 The benefit of this approach was that it provided a systematic representation of the three main groups of users of the new railway (walk in, busbased potential and carbased) in each local authority area. If the sample had not been targeted in this way, a good representation of potential Borders Railway users may not have been obtained, and the findings could be skewed by an over or under representation of some groups. The three ‘tier’ groups also form a natural categorisation for analysing the results of the survey and the later evaluation, as they will be expected to show different behavioural responses.

Survey Method

3.12 In order to achieve the required sample of households there were a number of possible survey methods. These are outlined in the table below, along with the pros and cons of each approach.

Table 3.2: Potential Survey Approaches
Method	Pros	Cons
Phonebased interviews	Ongoing control over sample Responses are coded in real time Complex routing can be included	Length of questionnaire may be limited by acceptable length of call. Increasing reluctance by some people to accept cold calls may lead to bias in sample. Fairly labour intensive assuming a circa 20% response rate.
Doortodoor interviews	Ongoing control over sample Lots of detailed data can be collected Anecdotal findings are also obtained	Very resource Intensive – interviewers may only achieve circa 10 completed interviews per day. Responses need to be manually coded. Potential bias depending on time of day of calls.
Door to door selfcompletion questionnaire mail drops	Can access a very large number of households Respondents can complete at their own pace and in their time	Surveys could be mistaken for ‘junk mail’. No control over sample. Responses need to be manually coded. Response rates can be very low. High volumes of printed material and return envelopes required.
Door to door ‘knock and drop’ selfcompletion questionnaires	Can access a very large number of households Raises awareness of the survey on a personal basis Respondents can complete at their own pace and in their time	No control on sample. Responses need to be manually coded. High volumes of printed material and return envelopes required.
Online	Very low cost – set up and publicity only Responses are automatically coded Survey can include complex routing Can be promoted and liked via social media Respondents can complete at their own pace and in their time Many more responses may be received than the target, at zero additional marginal cost	No control on sample responses. Difficulty in reaching the nononline community [phone support line can be offered].
On street intercept interviews in key centres	Ongoing control over sample Lots of detailed data can be collected Anecdotal findings are also obtained	Very labour intensive and time consuming. Responses need to be manually coded. Potential bias depending on interview location strategy.
Onstreet distribution of selfcompletion questionnaires at key centres	Easy to administer Can access large number of people.	No control over sample. Responses need to be manually coded. Potential bias depending on distribution location strategy. High volumes of printed material and return envelopes required.

3.13 As the table shows, there are both positive and negative aspects for all of the approaches and no single approach necessarily meets all the research requirements. Given the short time frame available for the data collection (July 2015 until the opening of the line on 6th September 2015), it was reasoned that conducting doortodoor surveys would be too time intensive. Similarly, as a result of the length of the survey it was felt that achieving the target response rate in the timeframe required using a telephone interview approach alone would be problematic.

3.14 As a result a Mixed Method Approach was adopted; consisting of an online survey supplemented with a phonebased booster survey should the online responses fail to reach target sample rates. It is acknowledged that an online survey carries the risk of potentially underrepresenting some groups that do not have access to the web. However the ONS report that in 2014, 84% of households had online access, and this proportion is continually growing, so households with no online access are now very much in the minority. Nevertheless, for those without access to the internet a telephone number was provided on the literature advertising the survey.

3.15 In addition, in order to minimise any potential bias, the demographic mix of the responses received was monitored throughout so that if certain groups were found to be underrepresented, these could be targeted as part of the booster phone based survey.

3.16 For the booster survey, households within the three tiers discussed above were targeted, with telephone numbers derived using zonal searches provided by www.zonesearch.com. The telephone surveys were conducted using a Computer Assisted Telephone Interview (CATI) approach, using similar bespoke software to that used within the online survey. This enabled responses to be coded at the point of interview and totals logged on a daily basis to ensure accurate monitoring of the sample rate.

Survey Promotion

3.17 As part of the promotional strategy for the Household Survey over 40,000 letters and leaflets advertising the survey were produced and distributed. Copies of these are included in Appendix B and Appendix C. The letter was delivered to households within the three tiers as identified above while the leaflets were delivered to public buildings across the area. A complete list of the buildings to which leaflets were delivered is included in Appendix D.

3.18 The survey was also advertised within The Borders Telegraph, The Midlothian Advertiser and The Southern Reporter and was included in the South Scotland Newsfeed on the BBC news website. All three local news agencies also regularly promoted the survey via their social media outlets, including Facebook and Twitter. A copy of the newspaper advert used is included in Appendix E.

3.19 Scottish Borders Council, Midlothian Council and the Borders Railway also publicised the survey via their websites and social media outlets, with Borders Railway also sending the link via email to all members of their mail group. In order to encourage participation in the survey, respondents were also offered the chance to be entered into a prize draw for one of five £50 shopping vouchers of their choice.

3.20 All publicity material for the household survey, including the introduction to the survey, explicitly mentioned the Borders Railway. It is acknowledged that this had the potential to introduce an element of bias into the results, with those most interested in the rail line more likely to be motivated to complete the survey. However, it was felt that capitalising on the positive publicity and general interest surrounding the opening of the line would assist in achieving a larger sample and that this outweighed any negative consequences associated with potential bias.

Survey Administration

3.21 The online survey went live on the 26th July 2015. Following, a slowdown in the number of responses received after the first month, a booster telephone survey was undertaken during the period 26th August 7th September 2015. Despite this, the number of responses received slightly remained below the 3,000 target at the date of the reopening of the line (6th September 2015). As such, the survey deadline was extended to the 22nd September. While this had the potential to introduce an element of bias into the results since the motivations driving people to complete the survey pre and postopening are likely to differ, it was felt that this was outweighed by the benefits of achieving a larger sample.

Control Area Survey

Control Area Selection

3.22 The purpose of the control area is to help understand whether any changes which occur in the study area are directly due to the new rail line. This is achieved by comparing impacts within the study area to that of the control area, thereby isolating the effect of the railway and using it as a differentiating factor.

3.23 In selecting a control area it is important to choose a location which shares as many of the characteristics of the study area as possible but which are not affected by ‘spill over’ effects from the intervention. With this in mind, two control areas were selected: East Lothian and Dumfries & Galloway. For the former, only households within those areas without a rail station were targeted while the latter included the whole local authority area.

Content

3.24 The Control Area Survey was based on the Household Survey and shared many of the same questions on household characteristics and travel patterns. However, the Control Area Survey did not include the Travel Diary element as it was felt this would add unnecessary length. In addition, any questions on or reference to the Borders Railway were removed as these were not felt to be relevant and had the potential to confuse and deter potential respondents. Finally, for the Dumfries survey, rather than asking about the frequency of business and other purpose trips to Edinburgh, respondents were asked about their frequency of business and other purpose trips to the nearest large towns (namely Dumfries, Stranrear or Glasgow). A copy of the Control Area Survey is included in Appendix F.

Method

3.25 In contrast to the Household Survey, the Control Area Surveys were conducted entirely by telephone. This was because of the additional difficulty associated with reaching areas where the Borders Rail is not relevant using an online survey. Telephone numbers for the Control Group samples were derived using zonal searches provided by www.zonesearch.com.

3.26 In addition, references to the Borders Railway in the introduction to the survey were removed as these were not felt to be relevant and had the potential to confuse and deter potential respondents. Instead, respondents were informed that research was being undertaken to investigate how people travel in Dumfries / East Lothian. As with the household survey, respondents were asked if they were willing to undertake future research and therefore there is potential to approach some of the same respondents during the future evaluation.

Survey Administration

3.27 The Control Area Survey was undertaken between 5th August 2015 and 25th August 2015. As with the Household Survey booster survey, responses were coded at the point of interview and totals logged on a daily basis to ensure accurate monitoring of the sample rate. Respondents to the Control Area Surveys were also offered the chance to be entered into the prize draw for one of five £50 shopping voucher of their choice.

Business Survey

Survey Purpose

3.28 The purpose of the business surveys was to determine business conditions and business use of the transport network prior to the opening of the Railway. Given the evaluation will ultimately consider the Wider Economic Benefits (WEBs) generated by the railway, the survey gathered baseline information on a number of business metrics, such as those on turnover, employment, productivity and access to markets (e.g. suppliers and customers).

3.29 In addition, while the new line will likely result in new and positive opportunities, the opening of the railway may also bring threats to existing businesses through, for example, greater levels of competition for both staff and customers. The business survey therefore explored current recruitment and competition so that the impact of the line on these issues can be systematically evaluated at a later date. A full copy of the survey is included in Appendix G.

Survey Method and Administration

3.30 The Business Survey was conducted primarily by phone between 6th August 2015 and 24th September 2015. Businesses in both Midlothian and the Scottish Borders were contacted directly and asked if they would like to complete the survey. Contact details for an initial list of businesses were provided by Scottish Enterprise. These were supplemented with details obtained from the website Zone Search and google based web searches.

3.31 An online version of the survey was also produced. This was circulated to a number of membership based consultees who were asked to circulate the survey to their members. The consultees included the Federation of Small Businesses, Midlothian Chamber of Commerce, Scottish Borders Business Gateway, and Scottish Borders Business Forum who were asked to share it with their members. A link to the survey was also sent to all businesses who requested to complete the survey online rather than by telephone.

3.32 As with the Household Survey, the introduction to the Business Survey explicitly mentioned the Borders Railway as it was felt that this would assist in achieving a larger sample. During the future evaluation, consideration should therefore be given to the different motivators driving businesses to opt into the research.

3.33 Achieving the required response rates to the survey was difficult, with large numbers of businesses choosing not to take part. While the survey was initially scheduled to close prior to the opening of the railway, as a result of the low number of responses, the deadline was extended to the 24th September.

Panel Survey

3.34 The Household and Business Surveys were largely focussed on the perspective of Midlothian and Scottish Borders residents and businesses, i.e. those that live and/or work in the area. However, the new rail line will also open up access to Midlothian and the Scottish Borders for those who do not live or work there. In order to gauge the potential inbound effects from easier access by rail, a panel survey of residents in Edinburgh, East Lothian and West Lothian was undertaken.

3.35 The Panel Survey ran between 18th and 25th August 2015 and aimed to explore the perceptions of both the Borders and Midlothian as business and/or leisure destinations and the propensity to visit each location. It is intended that the Panel Survey be repeated as part of the evaluation in order to determine if attitudes or behaviours have changed as a result of the opening of the Railway. As with the Household and Business Surveys, the introduction to the Panel survey explicitly mentioned the Borders Rail line. A full copy of the Panel Survey is included in Appendix H.