1 Introduction

1.1 Background

In September 2015, KSO Research was commissioned to undertake an independent evaluation of Road Safety Scotland’s Early Years Resource: ‘Go Safe with Ziggy’.

Launched in 2010, Go Safe with Ziggy is Scotland’s main road safety programme for children in their early years. Focussed around a series of six books, the approach targets three key age groups: 0-3, pre-school and the transition into Primary 1, in line with the Scottish Government’s policy on early intervention and Curriculum for Excellence (CfE)^[3]. The Ziggy books are made available for free to all early years establishments across Scotland each year including all public, private and public-private nurseries. Each of the books is available in large copy format (‘big books’) for use as a teaching resource by professionals in early years’ establishments and primary schools, as well as small books for the home to be used by children and their parents/carers. The hope is that the small books can be used at home to complement the work done in class and build strong links between home and school, promoting consistent road safety messages in both environments. The books are provided to all who submit an order, however, take-up remains far from universal across the country.

Given that the Ziggy books have now been in operation for just over five years, it was considered timely to evaluate whether they are delivering against their aims and to explore potential for further improvements based on users’ experiences of the resource to date. This report details the learning from the research that was commissioned and considers how the evidence may be used to contribute to the onward development, promotion and use of the Go Safe with Ziggy resources.

1.2 Aims and Objectives

The main aim of the research was to evaluate the content, distribution and use of the Go Safe with Ziggy series of books in order to understand whether they are designed, distributed and utilised in an effective way, offering value for money and to identify where potential improvements might be made.

Specifically, the research sought to provide an understanding of:

how the books and online resources are being used across the country;
how the books are being distributed to parents/carers;
how the resources are perceived by users, in terms of content and design; and
suggestions for improvements to the content, distribution and/or use of the Go Safe with Ziggy series of books.

Although the main focus of the work was on the Go Safe with Ziggy books, consideration was also given to the wider offering (including the online and audio resources that accompany the books, and associated collateral), to provide a more holistic view of the resource overall.

1.3 Research Methodology

The research methodology had four key strands, as follows:

a familiarisation exercise: including interviews with those involved in the resource distribution process, a literature and data review, and a technical review of the Go Safe with Ziggy books;

engagement with early years practitioners: including an online survey of early years practitioners and telephone interviews with a sample of teaching professionals;

engagement with parents/carers: through small focus group sessions held in four different sites across Scotland; and

workshops with road safety professionals and early years practitioners: designed to bring together representatives from different backgrounds to share early learning from the research and discuss suggestions for any future refinement or improvements.

The approach was formative, with findings from the early stages of the work informing the subsequent development of surveys, interview schedules, focus group topic guides and the workshop content.

Desk Research and Technical Book Review

An early task for the research was to complete a technical review of the Go Safe with Ziggy books. This was designed to provide the researchers with a comprehensive overview of the content, style and overall nature of the books in order to inform the subsequent development of questionnaires and topic guides for engaging with the books’ users. The technical review was undertaken at two stages: first, on inception and secondly, at the end of the research, following analysis of feedback from professionals, parents and carers with a view to better understanding and contextualising the feedback that was received. Each of the core books were reviewed separately against the same criteria to allow comparisons to be made between the different resources. Both the large and small copies of the books were included, along with the audio and online versions of the books. Although the Gaelic versions of the books and the accompanying multi-media resources/activities were not included in the initial review, views on these resources were sought in subsequent stages of the research.

Online Survey of Early Years Practitioners

The online survey provided the main data to inform the evaluation, and shaped significantly the overall evaluation findings that are presented here. The survey was designed to be completed online, with invitations to take part issued by email. Permission to approach schools and early years establishments in each local authority was sought from Directors of Education. Establishments from all local authorities were then invited to take part in the research, except for those in Moray where the council asked to be excluded from the research. To facilitate survey distribution, covering emails were designed for Head Teachers or other equivalent heads of establishments. The link to the survey was included in the covering email alongside assurances of anonymity and contact details for the research team.
Email addresses for primary schools were sourced from the Scottish Government’s school contacts database (a total of 2196). Addresses for early years establishments (including public, private and public-private partnership nurseries) were sourced from the Care Inspectorate database. These were cross-referenced with the 3130^[4] establishments listed on the Education Scotland early years database that is used for the distribution of the Go Safe with Ziggy order forms (which does not hold email addresses) to ensure that only those who had previously been invited to order the books were included in the survey.

A large proportion of establishments listed on the Education Scotland database could not be linked to email addresses on the Care Inspectorate database. Some of the establishments had specifically asked for their email address not to be included on the Care Inspectorate database and some groups did not fall under the remit of the Care Inspectorate (e.g. some playgroups and mother and toddler groups). Given that these groups are still eligible to receive the Go Safe with Ziggy books, independent efforts were made to trace their email addresses so that they were represented in the survey sample. In cases where email addresses were missing, online searches were carried out to try and find up-to-date email addresses. It was considered that the time and resources required to telephone establishments to ask for their email addresses was disproportionate to the likely returns.

The online search revealed that some of the establishments listed on the Education Scotland database had moved premises, changed name or had closed. Several of the early years establishments listed had the same email addresses as schools where the two were co-located (1186). There were also a number of duplicate entries (63 in total). Following removal of primary schools from the list (to avoid contacting them twice), a total of 1467 separate early years establishments were emailed.

Invitations to take part were issued by Transport Scotland’s Analytical Services instead of being distributed directly by the research team. It was felt that issuing invitations from a central, recognisable government organisation would help to highlight the legitimacy of the work and hopefully maximise response rates. Respondents were initially given between 2-3 weeks to respond, and the survey deadline was extended by a further 1-2 weeks following a reminder email^[5].

The questionnaire was also made available in paper format on request, and participants were offered an opportunity to take part by telephone, if preferred. There was one request for a paper copy of the questionnaire, and this was issued but was not returned.

A total of 279 valid responses were received from the 3663 email addresses that were contacted. It is important to stress that some of the email addresses may have been incorrect, have changed or been updated, may have been addressed to staff who were no longer at the establishment or may have been linked to establishments that were no longer open (e.g. community based playgroups). Given the mode of distribution, it is also likely that some of the emails were undelivered, may have been blocked or not opened by the intended recipients, and so it is not possible to know exactly how many of the invitations were actually received. Using the figure of 3663 as a guide, however, the indicative response rate was 8%.

The majority of respondents indicated that they worked with either ante-pre-school (78%) or pre-school aged children (76%)^[6], with most working with both age groups (73%). Only one in four respondents said that they worked with Primary 1 aged children (25%). A total of 21 respondents (8%) said that they worked with all age groups (i.e. ante-pre-school through to Primary 1). Where respondents indicated that they worked with ‘other’ children, this was typically those aged 0-5 or included children up to Primary 2. Some of the smaller rural or island based establishments worked with composite classes up to and including children in Primary 7.

Overall, the proportionate breakdown of email invitations issued was 60% to primary schools and 40% to early years establishments. With only 25% of the final sample indicating that they worked with Primary 1 aged children, the proportionate response from early years establishments seems to have been considerably greater than that of schools^[7].

There was considerable variation in the proportion of responses received from those invited to take part between local authorities and this should be borne in mind when interpreting the data presented in this report. It should also be noted that participation in the survey cannot be taken as a proxy for engagement with the Go Safe with Ziggy resources and some of those who use the resources most may simply not be represented in the survey.

Telephone Interviews with Early Years Practitioners

Following the survey and analysis of early responses, a topic guide was developed for semi-structured telephone interviews with early years practitioners, which sought to explore in more detail the use and perceptions of the Go Safe with Ziggy resources.

Volunteers to participate in the telephone interviews were identified via the survey, with respondents being asked to indicate if they would be willing to take part in a telephone interview and, if so, to provide follow-up contact details. From 53 volunteers, a total of 39 were invited to take part. The selection was based on geography, SIMD rank, establishment type and broad differences in survey responses (favourable and less favourable feedback), to ensure that a good mix of views was canvassed. A total of 20 interviews were carried out.

All interviews lasted around 10 minutes. Detailed notes were written up immediately following each interview, and qualitative feedback is provided alongside the survey results presented in this report. As the selection of respondents was not proportionate to the overall survey sentiments expressed, but was designed to seek a range of views, some of those with less favourable views may be over-represented in the qualitative feedback overall, and this should be remembered when interpreting the data presented.

Focus Groups with Parents and Carers

A total of four focus group sessions were held with parents and carers of children in their early years. Meetings were held in four separate geographical sites, a mix of rural and urban, which were chosen on the basis either of their local authority being well represented or poorly represented in the annual distribution/uptake figures for the books. The groups were held at neutral venues, and in all but one case this was at the venues where parents/carers and their children already attended regular groups together (e.g. at playgroups in church/community halls). Across the groups, the range in age of children cared for was 0-12 years and there was a mix of parents/carers who had only one child, up to 4 children. This meant that, in many cases, parents were able to reflect on their experiences of receiving the books not only for their youngest children but also older siblings. A total of 37 people took part in the groups and this included mothers, fathers, grandparents and childminders. Unlike professional participants in the research, parents/carers were given a small cash incentive to thank them for their time and to cover cost of travel to the meetings. Sessions were audio recorded with permission and detailed notes were written up after the sessions, including illustrative verbatim quotes for inclusion in this report.

Workshops with Professional Stakeholders

Towards the end of the research, two separate half-day workshops were convened to bring together road safety professionals and early years practitioners. The purpose of the workshops was to share learning from the research to that point, and seek views on how to take forward the emerging findings. Participants were recruited via existing networks (e.g. all Road Safety Officers across Scotland were invited to attend), via the online survey (respondents could indicate if they wished to attend) as well as via open invitations issued to early years establishments in the areas surrounding the workshop venues. Both events took place in neutral locations over a period of three hours in the afternoon. Across the two workshops, a total of 25 delegates took part (11 in Aberdeen and 14 in Glasgow). The sessions included a mix of presentations from the researchers and interactive group discussions and activities to share and document views and ideas. Participants were invited to give verbal feedback in response to group questions on the day and these were also documented on flip charts and sticky notes which were taken away for later analysis.

1.4 Research Caveats and Report Presentation

It is worth noting that, given the relatively small response rate to the survey overall, and the small numbers of people involved in the qualitative elements of the work, the findings cannot be generalised too widely. In particular, the disproportionate and varied level of response from around the country means that comparisons between areas should not be made.

While the overall response rate was lower than anticipated, it is worth emphasising that the online survey approach, with email invitations, represented the most cost effective means of ensuring that all primary schools and known early years establishments were contacted. Alternative postal or telephone survey approaches would have been considerably more resource intensive and would have required a sampling approach to have been taken instead of an all-inclusive approach. There is no way of knowing if the resultant sample would have been any larger than that achieved using the online approach, and so, although the response rate was low, it most likely represents the most effective and efficient means of generating the data that was required for the research.

It is also worth reiterating that the findings presented here represent only the views of those who opted to take part in the research and so there will also be a potential bias in the views expressed (i.e. we do not know the views of those who chose not to take part, and these may or may not differ significantly from the individuals who did respond).

Finally, the under-representation of schools should also be borne in mind. While the low level of response precludes any in-depth testing for statistically significant differences between groups of respondents (e.g. schools compared with pre-schools), the volume of data generated overall, and the level of detail provided by research participants still provides a relatively robust basis on which to evaluate the resources. The mix in different respondent groups reached across the research stages also means that the evaluation encompasses feedback from a wide range of users.

The remaining chapters detail the findings from across all of the research stages and are structured to provide learning in relation to the design, distribution and use of the Go Safe with Ziggy resources. Any differences in the views of different stakeholder groups are highlighted and, where qualitative evidence is cited, the strength or prominence of the sentiments expressed is detailed.