1 A Systematic Review of Learning Analytics Intervention Contributing to Students Success in Online Learning Kew Si Na and Zaidatun Tasir Faculty of Education, Universiti Teknologi Malaysia, 81310 UTM Skudai, Johor, Malaysia.
2 -the number of students enrolling (growing)INTRODUCTION -the number of students enrolling (growing) -different technologies (implemented) [1] such as e-learning a massive volume of students-generated data can be collected through their online learning activities Learning Analytics (LA) -in order to analyze and transform the data into more actionable information to improve learning environment and enhance the teaching and learning practices
3 -the attention of researchers on LA -the increases of data quantity -to understand what has happened in e-learning. LA is ‘‘the measurement, collection, analysis, and reporting of data about learners and their contexts, for purposes of understanding and optimizing learning and the environments in which it occurs’’ [8]
4 -the literature related to LA has only focused on the use of tools-less attention has been given on intervention design for students [7] -LA Intervention plays an important role (the impact and effectiveness of LA intervention) [4] [5] -faculty members’ perception (a useful tool in identifying at-risk students and delivering the intervention) [6]
5 -LA Intervention are needed (at-risk students) [21]-there is also a need to review of the use of the Learning Analytics Intervention implemented in various educational institutions and how its development level.
6 This paper will focus on(i) what is the purpose of, and how and where, LA Interventions are applied in educational institutions; (ii) what type of LA Intervention research is conducted; (iii) what the effects are of LA Interventions on learning and teaching; (iv) what theories or principles are applied in developing LA Interventions; and (v) which intervention strategies or tools are chosen to enhance students’ success, retention and engagement in their learning
7 Learning Analytics LITERATURE REVIEWStudents’ behaviors in e-learning can be easily tracked and collected in Learning Management System (LMS) by using Learning Analytics. These data are then analyzed and reported for the purpose to improve teaching and learning practices LA “uses analytic techniques to help target instructional, curricular, and support resources” to investigate the learning behaviors of the student and intervene the learning environments of the students” [8]
8 Learning Analytics The useful LA will bring benefits to the education field -one of the objectives of LA is prediction and intervention Eg1: Number of log-ins to the LMS by students, the frequency with which students engage with learning and teaching materials, and the results of assignments effectively predicted students’ course performance [11] Eg2: Final grades in class can be significantly predicted by the performance of students on course assignments and tests [12] System JACK -Nils
9 Learning Analytics Intervention-importance of the development of LA Interventions is emphasised (students at risk) [21] -“the surrounding frame of activity through which analytic tools, data, and reports are taken up and used”. - the educators should implement an intervention [16] [22]. (i) A traffic signal indicator is posted and displayed on the student’s LMS home page; (ii) messages or reminders are sent; (iii) Text messages are sent; (iv) Appointments are made with academic advisors or academic resource centres; or (v) Personal meetings are held with the educator. [23]
10 Methodology METHODOLOGY (Khan et al, 2003) 1) Frame the question ----what purpose and what intervention 2) Identify relevant work ; electronic databases; keywords, 30 3) Assess the study quality associated to LA Intervention 4) Summarize the evidence ----summarized in Table 5) Interpret the findings ----5 tables
11 Interpretation Flow what is the purpose of, and how and where, LA Interventions implemented in educational institutions what type of LA Intervention research is conducted Empirical Conceptual what intervention strategies or tools are selected to enhance students’ success, retention and engagement in their learning what are the effects of LA Interventions on learning and teaching what types of theory or principles are applied in developing LA Interventions;
12 Summaries of the research articles
13 c The learning analytics cycle: closing the loop effectively (Clow, 2012).
14 e Improving Undergraduate Student Achievement in Large Blended Courses Through Data-Driven Interventions (Dodge et al, 2015) Identify methods and intervention to reduce students who fail early on in their academic career. Logins, exam and clicker points
15 c Designing Pedagogical Interventions to Support Student Use of Learning Analytics (Wise,2014). Integration means that the use of analytics should be an integral part of the learning activity Agency aims at the goal that LA interventions shall support self-regulation of learners instead of distracting them. Reference Frame postulates a point for comparison a learner can refer to when using analytics. Dialogue creates a space for negotiation and discussion around the interpretation of analytics A preliminary model of pedagogical learning analytics intervention design
16 The Design of Intervention Model and Strategy Based on the Behavior Data of Learners: A Learning Analytics Perspective (Wu, 2015) Intervention model
17 e Learning Analytics for Online Discussions: Embedded and Extracted Approaches (Wise, et al, 2014)
18 e Investigating student motivation in the context of a learning analytics intervention during a summer bridge program (Lonn et al, 2015) Face-to-face meeting
19 Recommend strategies and provide feedbacksA Conceptual Framework linking Learning Design with Learning Analytics (Bakharia et al, 2016) Recommend strategies and provide feedbacks
20 Reviewing three case-studies of learning analytics interventions at the Open University UK (Rienties et al, 2016) evidence-based framework for learning analytics (make decisions about which types of interventions work well) an effective evidence-based framework in learning analytics needs to adhere to five conditions: 1) accurately and reliably identify learners at-risk/needing for support; 2) identify learning design improvements; 3) deliver (personalised) intervention suggestions 4) operate within the existing teaching and learning culture; and 5) be cost-effective.
21 Predictive modeling to forecast student outcomes and drive effective interventions in online community college courses (Smith et al, 2012) Direct and informal contact via telephone Automated course welcome s (their start date to encourage them to log in) Retention rate
22 Course Signals at Purdue: Using Learning Analytics to Increase Student Success (Arnold and Pistilli, 2012) Posting of a traffic signal indicator on a student’s LMS home page; messages or reminders; Referral to academic advisor or academic resource centers; or, Face to face meetings with the instructor Students more proactive
23 e An enhanced learning analytics plugin for Moodle: student engagement and personalised intervention (Liu et al, 2015) Enhanced Moodle Engagement Analytics Plugin (MEAP+) Gradebook data, assessment submissions, login metrics, forum interactions Personalized s
24 Integrated representations and small data – towards contextualized and embedded analytics tools for learners (Harrer and Göhnert, 2015) Integration Agency Reference Frame Dialogue + Scope that takes into account which information is relevant and useful for analysis and interpretation Representation Consistency shows the effects of representations used by learners on the quality of their learning.
25 Framework for Learning Analytics Intervention in e-learning (Kew and Tasir, 2016)
26 Result 1: Purpose of LA InterventionFINDINGS AND DISCUSSION Findings and Discussion Result 1: Purpose of LA Intervention Purpose of LA Intervention Frequency Percentage Cognitive engagement, engagement and participation 5 27.77% Cognitive retention and retention 4 22.22% Performance outcome and success Motivation 2 11.11% Satisfaction, Collaborative Learning, Learning process 1 (for each feature) 5.56% TOTAL 18 100% Number of views, replies and posts Number of log in More researches are needed
27 Result 2: Types of LA Intervention ResearchStill at a developing stage Type of Research Author Total Theoretical/ Conceptual [27], [21], [7], [31], [35], [36] 6 46% Empirical [28], [29], [30], [11], [33], [34] Review [32] 1 8% TOTAL 13 100% is or will be conducting
28 Result 3: Effects of LA Intervention Research on Learning and TeachingArticle Effect of Learning Analytics Intervention [28] Limited impact on student achievement [29] Supported changes in students’ discussion participation [30] Students’ mastery orientation decreased over the programme [11] (i) Faculty-designed outreach delivered: did not generate significant improvements in success rates (but students who received direct contact succeeded more often than students who received non-direct contact) (ii) Automated welcome system: a significant increase in early log-in rate [33] There is an impact on students’ grades and retention behaviour [34] It is useful for staff Mostly show that LA Intervention has impact
29 Result 4: Type of Theory/ Principle used in LA InterventionArticle Type of Theory/Principles Used the Intervention [27] Kolb’s cycle, Schön (single-loop learning and double-loop learning), Diana Laurillard’s Conversational Framework [21] Four principles of pedagogical learning analytics intervention design (Integration, Agency, Reference Frame and Dialogue); Three core processes (Grounding, Goal-Setting and Reflection) [7] Intervention mechanisms (intervention means, intervention contents and intervention implementation) [31] An open source Loop tool (a data warehouse and a web interface to display metrics and visualisations) [35] Adding two more principles of Scope and the principle of Representation Consistency to the existing principles proposed by Wise (2014) [36] LA Cycle, Kolb’s Experiential Learning Theory (Felder-Silverman model), Expectancy-Value Theory (ARCS model) Expanded the existing LA Intervention principles
30 Result 5: Strategies/ Tools used in LA InterventionArticle Posting the Signal ing Texting Arranging a meeting Telephone Teaching materials Instructional materials Tutoring Providing guidelines [27] / [28] [21] [7] [29] [30] [31] [32] [11] [33] / [34] [35] [36] TOTAL 3 5 1 2 6
31 Other findings: Examples of InstitutionsInstitutions and Learning Analytics Tools Open Universities Australia Personalized Adaptive Study Success (PASS) Rio Salado Community College PACE (Progress and Course Engagement) Purdue University Course Signals System Edith Cowan University Connect for Success (C4S) University of Michigan Coach Marist College Open Academic Analytics Initiative (OAAI) Nottingham Trent University, UK Dashboard Northern Arizona University Grade Performance Status (GPS) Open University, UK Structured Programme of Intervention Central Queensland University Early Alert Student Indicators project Have an impact on students’ success Inspiring more researches and developing
32 As a conclusion, LA Intervention is important to help at-risk studentsshows us how the current LA Intervention development status gives a clear overview for educators to make informed and successful interventions a good reference for other LA researchers more research on LA Intervention should be conducted
33 limitations and future researchthe characteristics of at-risk students, the attitudes, perceptions and roles of stakeholders, such as teachers, students and faculties, in developing LA Interventions, and the readiness of institutions to establish LA Interventions.
34 ACKNOWLEDGEMENT Acknowledgment The authors would like to thank the Universiti Teknologi Malaysia (UTM) and Ministry of Higher Education (MoHE) Malaysia for their support in making this project possible. This work was supported by the Matching Research University Grant (Q.J M68) initiated by UTM and MoHE.
35 THANK YOU