1 Professor Christina Hughes Sheffield Hallam University Professor Christina Hughes Sheffield Hallam University

2 LEGACY - Learning and Employability Gain Assessment CommunitY18 Russell Group Universities Birmingham, Bristol, Durham, Cambridge, Exeter, Imperial, Kings, Leeds, Liverpool, Manchester, Newcastle, Nottingham, Queen Mary, Sheffield, Southampton, UCL, Warwick, York HEFCE Pilot Programme 3 Years Oct 2015 – Sept 2018 4 Work Packages Measurement of Learning Gain Employability Higher Education LEGACY is a 3 year research project involving 18 Russell Group Universities, it consists of 4 WP’s – one which determines LG measurement factors and 3 which measure employability of LG & is part of a HEFCE Pilot programme investigating learning gain measures

3 Timeline Year /16 Year /17 Year /18 Measurement of Learning Gain WP1 R2 Strengths WP2 Career Adaptabilities WP3 Activity of the WPs start and complete at differing interaction points with the target GROUPS AND ALONG ITS LIFESPAN IT WILL PRODUCE A NUMBER OF REPORTS AND OUTCOMES WHICH WILL ALSO BE AVAILABLE VIA OUR WEBSITE ONCE IT HAS BEEN COMPLETED International Experience & Employability WP4

4 What is learning gain? Distance travelled (HEFCE)The improvement in knowledge, skills, work-readiness, and personal development made by students during higher education (RAND, 2014). The change (potentially, progress) in the knowledge, skills, and competencies that are relevant across disciplines made by students during higher education (Vermunt, Vignoles & Ilie, 2016).

5 Which dimensions are we measuring in LEGACY?Cambridge – cognitive, meta-cognitive, affective, socio-communicative plus cross-cutting dimensions – openness, research, moral Warwick – Realise 2 Strengths Birmingham – International experience Nottingham – Career Adaptabilities

6 Cambridge: Knowledge, skills, and competenciesCognitive Critical thinking Analytical thinking Cognitive abilities Synthesising Analysing Evaluating Problem solving Meta-cognitive Self-regulation Life-long learning attitude and motivation Learning to learn Need for cognition (information seeking) Affective Attitudes towards own discipline and learning/studying in general Motivation Engagement Professional and academic interest Socio- communicative Levels of belonging in social learning networks Social embeddedness Communication skills

7 Cambridge: Cross-cutting dimensionsOpenness dimension Open-mindedness; View of intelligence; Epistemological stance Research dimension Curiosity; Interest in research ; Interest in knowledge; Attitude to sharing ideas Moral dimension Moral reasoning

8 Warwick: Strengths Theoretical underpinning Base-Positive PsychologyR2 Strength Evidence base Strengths-based recruitment is increasing Students struggle to demonstrate self-awareness so can ‘undersell’ Strengths awareness helps develop supporting, authentic evidence Method Training in R2 Strengths of careers professionals Two cohorts of students – one engaged in full R2 programme; the other in strengths identification only. This strand seeks to increase students’ understanding of their strengths so they are able to demonstrate a greater degree of self-awareness and better articulate what they have to offer recruiters Led by University of Warwick Six HEIs partner: Sheffield, Queen Mary University of London, Manchester, Newcastle, Durham. Self-assessed, changes in students knowledge of their strengths, before questionnaire, intervention, after questionnaire Strengths is an established instrument that has been validated and implemented by a number of global organisations. Around 40% of larger graduate recruiters have moved to Strengths-based recruitment and selection IDEA: Students who have a clearer understanding of their strengths are able to demonstrate a greater degree of self-awareness and can often articulate what they have to offer recruiters more effectively Students identify the kinds of opportunity and potential employer will be able to provide supporting evidence of their relevant strengths in order to enhance their applications and interviews. ability to understand and describe their strengths will build long-term confidence and self-efficacy

9

10 Birmingham: InternationalisationImpact of international experience on students’ employability (Wright and Jones, 2014) Study abroad Work abroad On-campus experiences How can internationalisation enhance students’ employability skills? Method: Semi-structured interviews pre and post experience Identify self-perceived employability learning gains This strand seeks to identify students self-perceived employability gains from international experience and help develop more effective interventions Seven different Russell Group HEIs (Unis Birmingham, Southampton, Liverpool, UCL, Kings College, Cambridge, Exeter) – led by Uni Birmingham. Current reports focus primarily on overseas experience and graduate destination data. Strand will include on-campus cohort findings will identify activities and approaches that enhance employability learning gain from on-campus opportunities (particularly important for the widening participation agenda) How can internationalisation enhance students’ employability skills? Team working & learning how to work with, different people, Confidence and communication, Resilience, Leadership, Creativity/Imagination, Awareness/appreciation of different ways of thinking & doing, Problem Solving, Adaptability, Ability to use both narrow & wide, perspectives (of work), Using & making networks & contacts, Broaden Opportunities, Demonstrable ability to cope internationally. Similar to the Priority Global Competencies as Top 10 as reported in the AGR/CFE/CIHE report, 2011.

11 Birmingham: InternationalisationStudy Abroad: Motivation: academic related, employability related, personal interest & development Work abroad Extrinsic factors (CVs, experience) Intrinsic motivations Gain global competencies (Diamond et al, 2011) On-campus experience Risk averse behaviour Perceived positive impact Study abroad 33 transcripts, all pre experience - 6 institutions (saturation reached) Three main areas identified in terms of motivation and hoped for gains: academic related, employability related, personal interest & development Work abroad 8 transcripts , all pre experience - 3 institutions Extrinsic factors (CVs and experience) were mentioned, but intrinsic motivations were more frequent and more emphasised in the accounts. Hoped for gains tended to match global competencies in the literature (Diamond et al, 2011) but were not related to employability by participants On-campus experience 4 transcripts pre & post - one institution Reasons for not undertaking international experience tend to suggest risk averse behaviour Despite no reported change in quantitative perception of global competencies (op cit), all reported a perceived positive impact (even if only slight) on at least one competency post experience.

12 Nottingham: Career Adaptability‘The capability of an individual to make a series of successful transitions where the labour market, organisation of work and underlying occupational and organisational knowledge bases may be subject to considerable change’ (Bimrose et al, 2011) Stages of Careers Adaptability Source: Frigerio and Wright (2014) Four Career Adapt-abilities (Savickas, 1997) CAAS-International validated in 13 countries (Savickas & Porfeli, 2012) Adaptive Readiness Adaptability Resources Adapting Adaptation as outcome Career Adaptability – addresses progress in employment Led by Uni Nottingham, Partners are Unis York, Bristol, Birmingham, Kings College, Imperial College and Durham Uni. Tracking a cohort of students over the 2016/17 and 2017/18 academic years Based on the knowledge, that most graduates will need to change job or employers in the course of their working lives, this strand addresses the following aspect of employability - How trainable, how able to teach themselves new skills. - Abilities to adapt to different jobs and different employers - How ready, willing and able graduates are to deal with change or challenge in life and work FOUR STAGES of Careers Adaptability: Adaptive Readiness (personality traits) Adaptability Resources (psycho-social competencies – can be developed and measured) Adapting (action when required ) Adaptation as outcome Four Career Adapt-abilities (Savickas, 1997) Concern (Developing a positive optimistic attitude to the future) Control (Exerting a degree of intrapersonal influence on situations) Curiosity (Broadening horizons by exploring social opportunities and possibilities) Confidence (Believing in oneself and in the ability to achieve a goal0

13 Nottingham: Career Adaptabilities – initial findingsCareer Adaptability Scores Mean concern Mean control Mean curiosity Mean confidence Mean overall adaptability 80.0 Means are very similar to the means found in the original UK study Score ranges showed a normal distribution with varying degrees of skewness to the right) Careers questions 15% (N=64) have spoken to careers about ideas/plans 27% (N=116) have attended a careers fair 12% (N=50) have attended a career skills workshop 16% (N=67) have had a position of responsibility 32% (N=137) have done a CV 26% (N=111) have some work or volunteering experience Method: survey, based on the UK version of the CAAI (Careers Adapt-Ability Inventory) The CAAI measures 4 dimensions of Career Readiness via a questionnaire to self-assess against 4 dimensions: Concern, Control, Curiosity and Confidence. The study will use mixed methods, administering the questionnaire twice in each year and conducting a small number of interviews once in each year. Respondents Phase 1 N = 431 Gender Female 60% , Male 40% Age Mean age 19yrs Domicile 80% home students Ethnicity 75% white, 25% BEM Relationship between Career Adaptability and career related activity – Correlation matrix between career related activity/no career related activity and career adaptability factor scores showed significant relationships but extremely small effect sizes and a maximum shared variance of 4%, indicating no real relationship between career related activity and career adaptability scores at this first time of testing.

14 How does the work of LEGACY contribute to teaching excellence?Empirical Evidence of the Validity of a Range of Models to Measure Learning Gain, via A longitudinal explanatory model of student learning gains by a range of personal and environmental factors within disciplinary and employability domains; Analysis of core variables related to widening participation, gender, ethnicity, tariff entry scores and so forth Empirical assessment of a range of tools developed to measure learning gain, and establishment of their validity and reliability; A better understanding of the potential to measure learning gain at scale; Knowledge Exchange and Critical Awareness of debates and issues related to the measurement of learning gain – check ;

15 Will there by one way to measure learning gain?Our evidence, and that coming out of the overall programme, is that the complexities and dimensions of what is measured required a range of tools and techniques. All measures have limitations ie measuring employability gain based on credentials or skills provides measures of different dimensions All measures have limitations in measurement (cf self-reporting and random control trials) We need to critically explore the notion of whether there is a holy grail of one instrument/one measure that provides a common point of comparison across subjects and institutions We need to ask what measures are measuring – are they in effect proxies for something else that becomes invisibilised We should be measuring what we value not valuing what we are required to measure.

16 Find out more... Programme Professor Christina Hughes Lead: Project Sunil Maher Manager: Research Dr. Heike Behle Fellows: Dr Toni Wright Dr. Sonia Ilie Research Programme Leaders: Cambridge: Professor Jan Vermunt Professor Anna Vignoles & Birmingham: Eluned Jones Nottingham: Dr. Nalayini Thambar Warwick: Anne Wilson