Evaluation and Reviews - College Awareness Week

Introduction

This is a summary of a longer report that presents the findings of an independent evaluation, undertaken by the Centre for Effective Services, of College Awareness Week (CAW). The evaluation primarily focused on the contribution of CAW to promoting and supporting access to post-secondary educational options in Ireland. Additionally, data was collected regarding the operation and implementation of CAW. The evaluation team is indebted to all stakeholders who were interviewed or participated in surveys as part of the evaluation process, including staff in the Trinity Access Programme, the Higher Education Authority (HEA), SOLAS, the National Association of Principals and Deputy Principals (NAPD), and personnel in schools, further education institutions and access offices. CES would particularly like to thank the staff in CAW and the Trinity Access Programme (TAP), particularly Kathleen O’Toole-Brennan, who kindly provided the authors with relevant documentation, data, information and contacts to complete the report.

College Awareness Week

CAW is an annual, week-long, national campaign that promotes the importance of post-secondary education through a wide range of events and activities around the country. It was established in 2014 with the objective of increasing the number of people from underserved groups who take up a college place. It seeks to increase the overall national college-going rate, and to address the fact that while in some communities the progression rate to Third Level is 100%, in others the progression rate is less than 10% (CAW, 2014, p.3). It aims to promote the benefits of post-secondary education and to raise awareness of the range of post-secondary options, diverse entry routes to college and supports available within college. CAW take a broad definition of the term ‘college,’ encompassing higher education, further education and apprenticeships. It seeks to create a college-going culture in all communities, particularly lower-income communities where the ‘expectational culture’ around attending college is lower.

CAW emulates model of other national weeks in Ireland like Seachtain na Gaeilge, Maths Week and Engineers Week, with a diverse range of events organised at a local level within schools and communities aligned nationally over the course of a week in November. The CAW team promote the week and provide resources and merchandise, with individual events organised at grassroots level. Anyone is free to organise and promote events as part of CAW and to log them on the CAW website. CAW involves a wide range of different activities such as campus tours, taster lectures, past pupil visits to schools, mentoring, games and competitions, careers or college fairs and informational talks.

Methodology

CAW was evaluated through a mixed method retrospective evaluation conducted from November 2018 to February 2019. The methodology employed involved:

Documentary analysis including:

Review of CAW documentation including annual reports;
Review of research on awareness raising initiatives for educational access;
Review of relevant documentation pertaining to the projects and bursaries.

Primary research involving:

Semi-structured interviews with internal and external stakeholders;
Qualitative analysis of interview data;
Survey with Access Officers in Higher Education Institutions and Institutes of Technology;
Survey with personnel in schools and Further Education Institutions;
Qualitative and Quantitative analysis of survey data.

Secondary research involving:

Analysis of CAW events and evaluation data.

Evidence Base

Evidence from other access initiatives that focus on awareness-raising internationally, along with early evaluations of CAW indicated positive outcomes in the areas of: college progression; college attainment; parental knowledge, involvement and expectations; student knowledge of postsecondary options; student aspirations; student attitudes towards higher education; student intentions; college-going culture within school; and teacher expectations. While the results showed the potential benefits of awareness initiatives focused on raising awareness of the benefits of postsecondary education and increasing students’ knowledge of options, key differences existed between the international initiatives and CAW, and so the same results cannot be expected. The evaluations that have been completed on CAW to date, while only having taken place in TAP-linked schools in Dublin, were promising as they suggest that CAW has a positive impact on the college-going culture within schools and on the attitudes of students and teachers. Research carried out by the CAW founder and manager Kathleen O’Toole Brennan in six second-level schools showed a range of positive outcomes among students and teachers. It showed that following CAW the number of students who felt that going to college was an expectation in their school increased from 50% to 58.3%. Further research into CAW among five DEIS schools identified significant positive correlations between the number of CAW events attended and students’ perceptions of the importance of college, and the level of college-related conversations in the home. In addition, participation in CAW events was found to be significantly positively related to students’ plans to complete a college course with 53% of students who participated planning to complete a course in a University or Institute of Technology compared to 47% among those who did not participate. This result was highly statistically significant.

Main Findings

Activities

CAW involves a wide range of activities, primarily taking place in schools, higher education institutions and FET colleges but also in other locations such as libraries, museums and community organisations. Interviews highlighted a range of events from high-profile guest speakers to various creative arts projects, as well as more traditional college fairs, campus visits and talks. The surveys indicated that among access officers, talks, campus tours, school visits and workshops were the most commonly held events. For the respondents who worked in schools or FET colleges, talks and campus visits or tours were the most popular events. The majority of events run by access offices focused on alternative entry routes to college such as HEAR, DARE and mature student entry schemes or college life. Over two thirds of the events run by access offices focused on courses and how to apply for grants and supports. Events run by schools and FET colleges focused on a wide variety of topics, with courses and how to apply the most popular, with over 80% running events on this topic. Over two thirds ran events on careers and college life. Guidance counsellors drive activity within schools. 56.1% of the respondents of the school and FET survey were guidance counsellors. The wide range of events reflects the grassroots approach of CAW, with event organisers having freedom to determine the design and focus of their event. All events are awareness-raising in nature, informing students of their postsecondary options.

There has been a huge growth in the number of events run as part of CAW. In 2014, a total of 363 events were logged on the website with an estimated 35,000 participants (CAW, 2014, p.6). In 2017, 1,555 events were logged, and 155,000 people were estimated to have participated (CAW, 2018, p.8). The growth among second level schools is particularly striking. In 2014, events were logged in a total of 36 second level schools and has grown hugely since then (CAW, 2014, p.11). The growth from 2017 to 2018 alone was from 147, 20% of all second level schools in Ireland to 243, 34% of all second level schools. The participation of DEIS schools increased at an even greater rate during the same period, from 73 or 37% of all DEIS schools to 114 or 57% of all DEIS schools. This growth was commented on multiple times during the interviews, with interviewees seeing it as a major achievement of the campaign, particularly considering that CAW is run by a single, part-time staff member.

The need for fresh ideas each year was raised in the surveys and interviews. Variety is needed to ensure that students participating in CAW from 1^st to 6^th year of secondary school remain interested. Survey respondents expressed a desire for fresh ideas and up to date resources on the website. It was acknowledged that CAW sends out an email with 50 ideas each year which was praised in survey responses. An idea that was raised in an interview was that CAW could have an overarching yearly theme. It is also important that the ideas for events are reflective of the diversity of event organisers and participants. CAW have made an effort to ensure that this is the case, with resources on the website reviewed by SOLAS, NALA and disability offices to ensure that they are accessible. Nonetheless, one survey respondent felt there were not enough resources aimed at FET colleges and it was also commented that the 50 ideas did not include any ideas for events aimed at mature students.

Target Groups

Engaging with students who are underrepresented in higher education remains a priority of CAW. One interviewee was concerned that CAW was not managing to reach National Access Plan (NAP) target groups, however this view was not shared by all interviewees with some stating that there was a high level of engagement with all target groups, including people from low-socioeconomic backgrounds, Irish Travellers, people with disabilities, mature students and migrants.

All but one of the access officers surveyed felt that CAW plays an important role in promoting college among disadvantaged groups. 47.37% believed it played a very important role, with 26.32% selecting very important and 21.05% feeling it played a somewhat important role. 94.74% of the access officers believed that participation in CAW increases the likelihood of members of disadvantaged groups applying for college, and 90% of the access officers felt that CAW aligned well with their access office’s goals and outreach activities. Over two thirds (68.42%) of access officers stated that CAW helped them to reach their target groups. Access officers felt that members of target groups who attended CAW events benefit greatly from them and that organising events under a national framework led to greater buy-in from schools, facilitating higher participation of target groups.

48.68% of respondents to the school and FET college survey indicated that they were DEIS schools. However, that figure is as a percentage of total respondents, 37.8% of whom were not from schools but rather FET colleges or other institutes so within the amount of schools clearly a much greater proportion were DEIS schools. The vast majority of respondents had conducted events that had had target groups present, with only 7.41% of the respondents having none of the target groups at their events. Students from lower socio-economic backgrounds were particularly highly represented, being present at over 87.65% of events. 61.25% of the respondents felt that the CAW team and resources helped them to reach disadvantaged groups. All but 4 of the respondents felt that CAW was at least somewhat effective in promoting college among disadvantaged groups, with a fairly even split finding it either somewhat (35.44%) or very effective (34.18%) and over a quarter finding it extremely effective. Respondents in this survey commented on the positive impact of students engaging with role models from their own community as part of CAW, and on the value of CAW’s message that college is realistic and achievable for everybody in breaking down barriers.

The importance of early intervention in promoting access to postsecondary education is widely recognised. The access officer survey showed that 80.95% of events were with senior cycle students. However, there were events were run with younger students also, 61.90% with Transition Year students, 42.86% with junior cycle students and 19.05% with primary students – a combined total that is greater than the number of events with senior cycle students. Respondents to the school and FET survey and interviewees valued the opportunity that CAW provides to conduct college awareness-raising events with younger students. The access officer survey also indicated good engagement with students in the 18-23 years age group and in the mature student age group, with 47.62% and 52.38% of access officers respectively running events aimed at these groups. Access officers were very positive about the impact of CAW on potential mature students.

CAW has clearly been successful in increasing college awareness-raising activities with Junior Cycle and Transition Year students, however less has been done with primary school students. Primary school students were the least common age group for access officers to conduct events with and none of the survey respondents worked in a primary school. In 2017, 25 primary schools participated in CAW, a much lower figure than the number of second level schools (CAW, 2018, p.11). It would be desirable for a greater number of events to take place within primary school, in recognition of the fact that expectations and desires around college and careers set in early. CAW have event ideas aimed at primary schools, and the events that do take place within them, such as a ‘graduation ceremony’ for 250 children from TAP-linked DEIS primary schools in 2017 (CAW, 2018, p. 11) show that it is possible for primary school students to engage with CAW.

Benefits

Responses to the surveys and in interviews were largely very positive about the benefit of CAW to participants. Only one access officer felt that CAW had no benefit on the participants, with the rest believing it had an excellent (45%), very good (30%) or good (20%) benefit on attendees. Responses to the school and FET survey were more positive. 62.96% rated the benefit to students as excellent, with 32.10% rating it as very good. Three respondents rated the benefit as good (3.7%) and only one rated it as fair.

Both interviewees and survey respondents felt that CAW was encouraging students to consider college who otherwise might not have. Most of the respondents in the school and FET survey believed that CAW plays either a very important (41.98%) or extremely important (38.27%) role in promoting college. 16.05% felt it plays a somewhat important role, and a small number- three in total- felt it plays either a not so important or not at all important role. Respondents from schools believed that CAW was improving progression rates within their schools. However, interviewees acknowledged that there was no empirical evidence to prove this and that it would be difficult to attribute a student’s decision to progress directly to CAW.

Respondents also felt that CAW greatly improves people’s knowledge of the options available to them. An increased self-belief among students and potential mature students was highlighted in surveys. Respondents felt that CAW events helped to break down barriers and show that college was an achievable option, and that there were multiple routes for people to get where they want to go. Another important benefit, that should not be overlooked, is the enjoyment that students got from CAW. 100% of respondents to the school and FET survey believed that students who participated in CAW events enjoyed them. Some of the interviews and qualitative responses to the school and FET survey emphasised the positive atmosphere CAW creates within schools, with respondents commenting on the buzz and excitement within their school. Benefits of this kind should be communicated and celebrated, as well as more quantitative aspects such as growth of events.

There was consensus that the use of role models during the CAW campaign was very effective and positive. Communicating CAW’s key messages through young people from students’ own schools or communities who they can relate to was seen as a highly positive element of the campaign. CAW increased its use of student mentors in 2018 through its collaboration with the community mentoring path and this was viewed as successful.

Purpose

There was broad agreement that CAW’s purpose is to raise awareness of options available and promote the benefits of postsecondary education. A couple of interviewees commented on a lack of clarity about CAW’s purpose and mixed messaging. This criticism was not reflected in the survey responses. This perhaps reflects the fact that CAW operates primarily as a grassroots campaign, with event organisers at a local level therefore more likely to be clear on the purpose of the events that they are running.

There was some disagreement as to whether CAW should be a universal campaign or aimed more explicitly at target groups. There were also differing opinions on how prominent a role further education should play in CAW, with some interviewees feeling that CAW should focus primarily on higher education and others that CAW should promote all postsecondary options equally. Respondents to the school and FET survey expressed positive views about the fact that CAW raises awareness of all options, including FET and apprenticeship options

It appeared that CAW’s two statutory funders, the HEA and SOLAS, understandably have their own organisational objectives and policy goals, and this results in some differing viewpoints on what CAW should prioritise as a national week. The HEA have a remit to promote higher education, and SOLAS to promote further education and training. This could place CAW in a somewhat difficult position of attempting to meet the differing requirements of both of their funders and to try to strike a balance between both viewpoints. It is desirable that CAW, SOLAS and the HEA come to a shared agreement on what the purpose and focus of CAW should be.

Model

CAW utilises the same model as initiatives such as Seachtain na Gaeilge, Engineers Week and Maths Week with activities being organised at a local level under an overarching framework. The success and growth of other national weeks such as Seachtain na Gaeilge would suggest a merit to this approach. However, there were some doubts expressed about the merit of this approach, with some respondents feeling that CAW events would take place anyway and having them concentrated in a week creates undue pressure on organisers without bringing extra benefit. This view was not widely shared. Many respondents noted the added excitement and festival-atmosphere that a week-long celebration brings, particularly within schools. Some respondents were motivated to put on events that would not have taken place without CAW. Access officers felt that CAW led to a greater level of support from other departments and increased engagement from schools. Respondents also noted that CAW fosters a whole school approach, where ordinarily work around college awareness would be solely consigned to guidance counsellors. Many survey respondents also appreciated the increased media attention that the CAW campaign brings. Another clear benefit of the model is that it allows CAW’s budget to be harnessed effectively to reach a greater number of people. Considering all of this, the view of the reviewers is that the model of CAW is a good one, and that shared activity under an umbrella campaign brings many benefits.

There was a criticism in one interview and by one access officer responding to the survey that the CAW campaign was not sufficiently national and was viewed primarily as a Trinity campaign. This criticism was not widespread however. The geographical spread of CAW has increased over the years. In 2014, events took place in 22 counties but the majority of events, 236 out of 363, took place in Dublin (CAW, 2014, p.13). In 2017, events took place in 26 counties with a greater spread outside of Dublin, with 407 out of 1,555 of the events taking place in Dublin (CAW, 2018, p.16). Given that this means that over 70% of events took place outside of Dublin, a growth from just over a third in 2014, CAW clearly is a national campaign. It is apparent from analysis of the data that there is good geographical spread nationally, in both urban and rural areas and CAW should continue to pay attention to regional spread to ensure that this continues. There were suggestions in one of the interviews that CAW’s governance should be less Trinity-based however the question of governance is beyond the scope of this evaluation.

Monitoring and Evaluation

To date, there have been limited evaluations of CAW and the decision to commission this evaluation illustrates the positive contribution CAW is making to a number of outcome areas. The limited independent evaluations of CAW to date has been primarily due to a lack of resources to commission evaluators. It was highlighted in interviews that CAW is implemented on a limited budget and available resources are prioritised for project delivery, not evaluation.

The main mechanism for monitoring of CAW is through the CAW annual reports. Review of the CAW annual reports suggest that the current emphasis is on outputs (primarily in the form of events, activities etc. logged to the CAW website, numbers of participants etc.). This is important information and is helpful in determining the community reach and engagement with CAW. However, there is very limited cross-initiative information on the contribution CAW has made to students and prospective postsecondary students in relation to some of their specified core objectives (CAW, 2018, p.5) such as: increase awareness of the multiplicity of entry routes to college, raise awareness of benefits of going to college and increase awareness of the range of courses in college.

In a number of interviews, some respondents stressed that they would like CAW to show/prove its’ impact on third level progression rates among under-represented groups. However, this would be a difficult outcome area to draw direct attribution to, as by the time students access third level they may have received a variety of supports, services or interventions. For future evaluations and reports, CAW should focus on outcomes directly related to the types of events held during the week such as information, access to positive role models, educational aspirations, awareness of entry routes, courses, etc.

Some recommendations were made in interviews as to how the impact of CAW on progression rates could be captured. The primary recommendations were for access officers to query about participation in CAW at induction to third level or to include a question in the CAO application process. However, these are oriented at third level and CAW is focused on the broad spectrum of post-secondary options, including FET.

Support Received from CAW

The work of the CAW team and supports received were commended in interviews and surveys, with the helpfulness and positivity of the staff highlighted by several respondents. Just over 76% of all access officers surveyed were at least satisfied with the supports received from the CAW team, with just a small minority not expressing satisfaction. Over 80% of access officers also stated they were happy with the CAW online resources available. Looking at schools and FET institutions, just over 86% were satisfied with the supports received and 84% were satisfied with the online resources made available.

There is also effective dissemination of CAW merchandise by the CAW team, with 95% of access officers receiving CAW merchandise. Just under 90% of access officers reported satisfaction with the materials received. 90% of schools and FET institutions responding to the survey reported receiving CAW merchandise, with 77.5% indicating they were satisfied with the materials received.

There were some helpful suggestions from survey respondents including providing digital templates of CAW promotional materials to allow local organisers to personalise materials and providing more ideas for events with mature students and audiences outside of schools. Other suggestions included providing more robust selfie frames and not dating all merchandise, so it can be reused in future years. While it was acknowledged that resources are an issue, several survey respondents stated they would like to receive more merchandise. The need for new ideas for local organisers was stressed and there was also a suggestion that ideas for events are updated and disseminated earlier, to help local organisers plan events in advance.

Funding

The issue of funding and CAW being run on a limited budget was a common theme across interviews. Limited financial resources and staffing appeared to be inhibitors to the expansion of CAW and it was suggested that the absence of multi-annual funding threatens the sustainability of the campaign. The continued financial and in-kind support of the HEA, SOLAS and Perrigo were seen as vital to CAW’s continued success. AIB’s withdrawal of funding was identified as a loss.

It is clear that CAW is achieving a significant amount given its funding and staff limitations. This is possible due to volunteer work and passion of staff. Yet there are necessarily limits on what this can achieve, and there was an impression that, with multi-annual and increased funding, CAW could become a more prominent campaign and plan more strategically. There is a clear need for CAW to attract more diverse funding to ensure security and stability, from both statutory and private bodies.

Timing

In general, participants in this evaluation felt that the timing of CAW was appropriate, but there were some concerns raised. It was highlighted that having the national week in November was helpful for students considering their CAO applications and those considering scholarship applications, which generally open in late October. There is also a European level awareness week for Vocational Education and Training (VET) in November which was not currently posing an issue, but it was indicated as important for SOLAS to strike the right balance in November between European and national initiatives.

However, several respondents stated that November can be a busy time academically, especially for third level institutions and semesterised third level students and that this can impact negatively on third level student participation. A small number of respondents stated a preference for holding the week in October, however November appeared to be broadly suitable based on qualitative responses from personnel in schools.

Media

The media attention on CAW was valued by survey respondents. One interviewee expressed the concern that a huge proportion of CAW’s budget appears to be spent on PR. However, it became clear through the evaluation that the PR company CAW works with, PR360, work with them on a pro bono basis. It was seen as desirable for CAW’s national media exposure to increase. Some interviewees spoke of a desire for CAW to take over national airwaves during the week. One interviewee suggested raising awareness of the use of ambassadors and mentors undertaking courses would attract media attention.

Recommendations

A number of recommendations were made in light of the findings. CAW is currently being run by one part-time staff member who also has other project commitments outside CAW, and there are obvious resource implications, both from a financial and personnel perspective, to support implementation of these recommendations.