1 Meeting of the Pedagogical CommitteeEvaluation „Where are the Young Trade Union Leaders whom we trained in the last 5 years and what are they doing“ Meeting of the Pedagogical Committee Simonas Gaušas, Visionary Analytics 10th March 2015, Firenze © Visionary Analytics, 2015

2 Content Introduction Aims and objectivesSubject and scope of evaluation Approach: Evaluation criteria and questions Evaluation methods Evaluation phases, tasks Risks Timetable Next steps © Visionary Analytics, 2012

3 Visionary Analytics Research based policy advisory, specialising in education and innovation policy areas Founded on March 2012 Education assignments: Thematic Country Review of Apprenticeships in Lithuania Thematic studies in education in the contract „Development of the Lithuanian Progress Strategy “Lithuania 2030” implementation monitoring system“ ETUC study ‘Young, Green and Union: addressing sustainable youth employment for Europe’ Innovation assignments: Ex post evaluation of Cohesion Policy programmes financed by the ERDF and CF on support for increasing research and innovation in SMEs Preparation of evaluation methodology and consultations regarding return on investment to R&D Supporting EC and national government with regard to smart specialisation strategy © Visionary Analytics, 2012

4 Aims and objectives of evaluationOVERALL aim – to evaluate the relevance, coherence, added value, efficiency, effectiveness and impact of the activities provided by the ETUI Education to train young European trade union leaders over the last five years SPECIFIC objectives: the degree to which the participants saw their responsibilities increase in the organisations, particularly a European dimension (+how many ETC participants moved outside their trade union) the relevance of the contents and the efficiency and effectiveness of the programmes the impact and added value for the affiliated organisations, as well as other key stakeholders and, by extension, its contribution to the development of Social Europe and the European social dialogue © Visionary Analytics, 2012

5 Subject and scope of evaluationEuropean Training courses (ETCs) for young TU leaders and their effects Duration: October 2008 to June 2013 Evaluation focuses on the following groups: Participants (main target group) – approx. 100 persons Non-participants Training-trade union organisations (includes some ETUFs) Non-training trade union organisations Stakeholders (ETUFs + NETYL, ETUC, ETC experts, EESC and EP members) ETCs include 3 residential weeks and 3 preparatory work periods © Visionary Analytics, 2012

6 Approach: evaluation criteriaImpact Stakeholders Organisations Participants Other similar programmes at national level Needs & challenges Results ETC programme Objectives Inputs Outputs ETC evaluation Relevance Efficiency If we can drop any evaluation question, we should do it (less is better) Effectiveness (results) Added value Impact Internal & external coherence © Visionary Analytics, 2012

7 Preliminary indicators Sources of informationApproach: Relevance The extent to which the objectives of ETC activities match the needs of target groups. Evaluation will measure relevance of ETC for both the ETC participants and their environment (i.e. trade union organisations) Evaluation questions Preliminary indicators Sources of information Has the ETC helped you to identify knowledge and skills gaps? Whether the gaps, identified as relevant for work during the ETC, remained relevant for you after the ETC? Relevance of ETC content in terms of identifying knowledge and skills gaps of participants Survey P Clarifying interviews P Desk research How useful for your career and for your organisation was the knowledge received during the ETC? Usefulness of ETC content for personal career and his/her organisation ETC has facilitated identification of development needs of my organisation ETC has contributed to satisfaction of the key needs / challenges Relevance of ETC in terms of identifying and satisfying the development needs of participants’ organisations Survey NC Clarifying interviews NC Knowledge that our colleagues have gained at ETC was highly useful to my organisation Usefulness of ETC content for participants’ organisation © Visionary Analytics, 2012

8 Preliminary indicators Sources of informationApproach: Added value The “value” resulting from ETC activities that is additional to the “value” that would have resulted from no intervention or an intervention at national or regional level by public authorities, trade union(s) and/or NGO(s) Evaluation questions Preliminary indicators Sources of information What additional value is created by implementing the ETC young leaders course at European level rather than national, sectoral, regional, local or organisational level? Reinforcement of European trade union identity Surveys P + NC Clarifying interviews P+NC Thematic case studies? Desk research Are you aware of other education programmes for young trade union leaders at national, sectorial, regional, local or organisational level? Existence of other education programmes for young trade union leaders at national, sectorial, regional, local or organisational level In this evaluation the European Trade Union identity is understood as the set of features (especially values and forms of action) shared by the European Trade Union movement (set of unions members of ETUC), as representatives of a social class, the workers, which distinguish it from other social groups © Visionary Analytics, 2012

9 Preliminary indicators Sources of informationApproach: Coherence The extent to which interventions reinforce objectives and create synergies Training activity can be considered coherent if, for example, the same objectives are sought with different activities or by targeting different groups Types Evaluation questions Preliminary indicators Sources of information Coherence with other ETUI training activities (internal coherence) To what extent the ETC young leaders courses in terms of their objectives and content are complementary to other training courses provided by the ETUI? Evidence of synergies or overlaps with other ETUI training courses Survey P Clarifying interviews P Coherence with other training programmes for YTUL (external coherence) Are you aware of any other training courses exclusively addressing young trade union leaders that are carried out by any organisation in your country at national, sectoral, regional, local or organisational level? Survey P+NC Clarifying interviews P+NC Desk research © Visionary Analytics, 2012

10 Preliminary indicators Sources of informationApproach: Efficiency The extent to which the desired effects are achieved at a reasonable cost (or as the best relationship between resources employed and effects achieved) Evaluation questions Preliminary indicators Sources of information To what extent effects of training programmes were delivered with reasonable cost? Adequate distribution of funds by type of ETC activity (e.g. residential weeks) Desk research Costs per ‘typical’ output/ result (e.g. person trained) as compared with other ETC trainings To what extent implementation of training was well functioning and efficient? The management system is well-developed (structure, resources, processes, procedures, tools, risk management) incl.: pre-course information; working methods; language support; work of training staff; division of time; the location/ facilities; organisation of the course Survey of P Clarifying interviews P Interviews ETUI Cross-check of different results High levels of satisfaction among participants/ stakeholders Is the direct monetary contribution for registering one member of your organisation to the ETC training course acceptable for your organisation? Adequate requirement for cost-sharing of education programme costs Survey of NC Clarifying interviews NC © Visionary Analytics, 2012

11 Approach: EffectivenessThe extent to which objectives set and intended results are achieved. Effectiveness measures the achievement of the primary objectives – results immediately or shortly after the intervention Effectiveness also relates to the extent to which the target groups are being reached Evaluation questions Preliminary indicators Sources of information What trade union organisation has sent you to the ETC? Appropriate selection of participants Survey and interviews of P To what extent do you personally agree with the statements (information on opportunities, barriers…)? Survey and interv. of NC To what extent announced objectives of the ETC were implemented in practice? Evidence that ETUI has delivered what it promised Desk research What skills did you acquire or improve during the ETC young leaders course and to what extent? Reported acquisition/ improvement of skills To what extent the ETC has helped you to fill in your most relevant knowledge and skills gaps, i.e. provided knowledge and skills that you have identified as particularly lacking in your work? Reported filling in of most relevant knowledge and skills gaps of participants Survey of P Clarifying interviews with P © Visionary Analytics, 2012

12 Preliminary indicators Sources of informationApproach: Impact Focus is on net effects of the ETC. The extent to which the intervention changes outcomes. The goal is to verify the effect on longer-term outcomes (e.g. use of skills, use of networks or increase of responsibilities of participants, better performance of trained staff of organisations or development of wider social dialogue) caused by the intervention Unit of analysis Evaluation questions Preliminary indicators Sources of information Impact: participants During a typical working week, does your work involve the following skills? Reported use of skills Survey P Clarifying interviews P Survey NP Thematic cases Has participating in the ETC young leaders course helped you in your career? Reported increase of responsibilities regarding European dimension and/or change of participants’ job positions Did you keep in touch with other participants-YTUL from other countries after the end of the ETC? Reported use of networks built during ETC activities Impact: organisa-tions To what extent and in what skills do trained YTUL in your trade union organisation perform better than their colleagues – YTUL who did not attend the ETC? Reported differences in performance of young trade union leaders who attended ETC and those did not Survey NC Clarifying interviews NC Impact: stake-holders To what extent the ETC training course has contributed to the expertise of participants on… ((European) social dialogue, Social Europe, organising of TU) Views of representatives of national confederations and identified major stakeholders Survey of NC Interviews S © Visionary Analytics, 2012

13 Methods: Desk researchCarried out July 2014 – January 2015 What we have: Strategic and operational documents (e.g. training programmes and journals, ETC seminar dossier); Monitoring and financial data (e.g. previous reports & surveys, financial information, lists of target group members); Physical data includes outputs that have been produced (e.g. participants’ evaluation sheets; participants’ e-profiles (e-portfolios); videos of participants) © Visionary Analytics, 2012

14 Methods: Surveys Four web-based surveys of: Implementation:ETC participants Non-participants Representatives (of Education departments) of national confederations that have sent their young members to the ETC (training TU organisations) Non-training TU organisations Implementation: Piloting for all four types of surveys (one interview per survey) Distribution of survey via Visonary Analytics (Fluid Surveys®) Participants survey questionnaire translated in 10 languages The remaining ones (except non-participants?) are in English Unclear/ interesting answers are subject for interviews © Visionary Analytics, 2012

15 Methods: Interviews Aims:To better understand the programme To pilot questionnaires To clarify answers To get more information on the selected cases Up to 40 semi-structured telephone interviews with: ETUI representatives (incl. Education officers and ETC trainers) (Non-)participants Representatives of national confederations Representatives of stakeholders Questions will be based on survey/ case study templates © Visionary Analytics, 2012

16 Methods: Case studies 4-6 brief (up to 1 page) illustrationsPrimary selection criteria – interesting or prominent ETC effects or added value Primary source of thematic cases – surveys of participants and training trade union organisations Piloting: first case study to verify focus/ structure/ style © Visionary Analytics, 2012

17 Approach (by phases) © Visionary Analytics, 2012Stage 1: Planning and methodology incl. interviews with ETUI Stage 2: Conducting web-based surveys Stage 3: Clarifying interviews, thematic case studies Stage 4: Analysis and preparation of evaluation report Inception phase Data collection phase Data analysis phase © Visionary Analytics, 2012

18 Approach (by tasks) © Visionary Analytics, 20121.1. Literature review 1.2. Interviews on ETC 1.3. Drafting questionnaires 1.4. Piloting questionnaires 1.5. Finalising questionnaires Output 1.0: Questionnaires 2.1. Identification of survey respondents 2.2. Surveying all target groups Output 2.0: Survey responses 3.1. Clarifying/ in-depth interviews 3.2. Identification of themamtic case studies Output 3.0: Clarified data, identified cases 4.1. Analysis of data 4.2. Preparation and integration of thematic cases 4.3. Presentation of findings Output 4.0: Evaluation report © Visionary Analytics, 2012

19 Project specific risksMeasure Respondents cannot remember their specific experiences in relation to the ETC (the earlier the year, the less memories) Ensuring sufficient number of responses to reduce impact of any individual’s failure to recall Appropriate level of detail in survey questionnaires Clarifying interviews s of some ETC participants are no longer valid; failure to reach those who moved out from the trade unions Contractor will seek the actual addresses of all participants Possible ETUI assistance in obtaining actual s Language difficulties (e.g. participants may need some clarifications regarding particular questions) Piloting of all survey questionnaires Participants questionnaire in 11 languages Skype, and telephone support line Relevance of the ETC in terms of filling the knowledge and skills gaps of participants may be problematic as they tend to usually have low or no expectations for the course Will also rely on information in participants’ evaluation sheets filled just after the end of the ETC Information in few available e-portfolios may also help to better assess this aspect Low response rate Follow-up reminders by and, if needed, by telephone Difficulty to identify cause & effect relationships in tracing impacts and effects Triangulation of information Reliance on control group of non-participants Emphasis on limitations in the evaluation report © Visionary Analytics, 2012

20 Timetable © Visionary Analytics, 2012

21 Next steps What When Who Finalising participants survey April 2015 VALaunching remaining surveys March 2015 ETUI/ VA Finalising remaining surveys April-May 2015 Interviews March-June 2015 Identification of thematic case studies May-June 2015 First progress report (reporting on data collection) July 2015 Drafting thematic case studies August-September 2015 Second progress report (data analysis) October 2015 Final report November 2015 © Visionary Analytics, 2012

22 Thank You Simonas Gaušas Visionary Analytics [email protected]