1 Click-level Learning Analytics in an Online Medical Education Learning PlatformMATT CIRIGLIANO, MS CHARLIE GUTHRIE, BA MARTIN PUSIC, MD, PhD May 5TH, 2017 Hi everyone, my name is Matt Cirigliano and I am a Doctoral Candidate at NYU Steinhardt working on research in medical education and the learning sciences. With me I have… Charlie Guthrie - I just finished my Master’s in Data Science at NYU, with the Center for Data Science. My background is in statistics with a focus on learning analytics. And we're currently working with Dr. Martin Pusic -- who [you’ve just met in the last talk] is the Director of the Division of Learning Analytics at the Institute for Innovations in Medical Education at the NYU School of Medicine -- on click-level learning analytics in MedU, an online medical education learning platform. [26 sec]
2 OBJECTIVES Exploratory Analysis: Understand the relationships between measures of learner engagement and learner achievement in historical MED-U databases using results from focus group discussions and learning analytics. Our objective was to understand measures of learner engagement… …meaning what learners clicked on, interacted with, and for how long… …and how these behaviors related to learner achievement. MedU's historical database of learner interactions and learning analytics allowed us to do that. [14 sec]
3 LEARNING ANALYTICS …is an emerging field in which analytic tools adapted from computer science, math, and statistics are used to improve learning and education by extracting usable information from very large datasets. Briefly, learning analytics uses the power of large datasets and analytic tools to understand how learners engage with material and improve approaches to achieving educational goals. [11 sec] SOURCE: Elias, T Learning Analytics: Definitions, Processes and Potential.
4 CONCEPTUAL MODEL FOR ENGAGEMENTAnd engagement incorporates the complex network of interactions a learner has with content―if they become engaged or disengaged by material. Learning analytics and predictive models can help us identify what content is most/least useful. [14 sec] SOURCE: O'Brien, H. L., & Toms, E. G. (2008). What is user engagement? A conceptual framework for defining user engagement with technology. Journal of the American Society for Information Science and Technology, 59(6),
5 ENHANCING ONLINE LEARNING THROUGH FEEDBACKLEARNING ANALYTICS ENHANCING ONLINE LEARNING THROUGH FEEDBACK Instructor & Instructional Designer Students Overall, learning analytics can help us generate feedback systems to help stakeholders improve learning content and strategies. [9 sec] Learning Analytics User & Log Data Adapted from Bienkowski, M., Feng, M., & Means, B.,(2012). Enhancing Teaching and Learning Through Educational Data Mining and Learning Analytics: An Issue Brief.
6 CLICK-LEVEL DATA Features: Multiple choice questions HyperlinksPage progression clicks Enlarging images Checking answers Time spent on images, pages, etc. Scope: 2,806 North American medical students From June 1st, May 5th, 2015 So with Med-U, we applied learning analytics to click-level data to reveal how learners interacted with the content. That includes [riff] MedU itself is an online suite of case-based learning systems and courses accessed by over 150 different medical schools across North America―you can see some more information at the bottom—there were over 2800 med students who contributed (only 6). One feature of MedU is the CORE Radiology series, which has 18 modules total. We focused on one on musculoskeletal trauma. [32 seconds]
7 A RELEVANT ASSESSMENT Content Relevant MultipleSo in sum, we wanted to understand if engaging with relevant content impacted performance on assessment questions in the module. [7 sec] Content Relevant Multiple (Images, links, etc.) Choice Question
8 MODULE PROGRESSION The module was broken down into units, where a set of cards featuring content was followed by an an assessment card, which featured a relevant multiple choice question. [9 sec]
9 And the whole module was 23 cards longAnd the whole module was 23 cards long. This is a screenshot of one card. [5 sec]
10 CANDIDATE ANALYTIC MEASURESBut...which features would be worth exploring with learning analytics? [3 sec]
11 FOCUS GROUP RESULTS Six experts grouped and ordered candidate analytic measures (CAMs), revealing which were considered the most useful: Thumbnail Click Post-Answer Expert Feedback Use Supplementary Link Click Magnifying (Zooming-in on) Images Time Spent on Cases To identify Candidate Analytic Measures, we performed a focus group with experts in medicine and instructional design to see what they thought would be most important to know about learner behavior. A set of 12 analytics were ranked and the top five were selected for further study. [18 seconds]
12 CAM-A: THUMBNAIL CLICKIn an image set where there is one dominant image along with several supplementary clickable thumbnails, does the rate of clicking through the thumbnails correlate with learning? What we need is: (a) Whether users clicked each thumbnail, yes/no (b) Whether they got the relevant MCQ correct (Card 13, MSK Trauma) The first was the thumbnail click, and whether clicking on relevant thumbnails impacted assessment outcomes. Sadly, because this data was unavailable in the database, it wasn’t incorporated in the model. But there’s more… [11 sec] (Card 15, MSK Trauma)
13 CAM-B: POST-ANSWER EXPERT FEEDBACKDoes learning correlate with how often a user clicks on links in the expert window? What we need is: (a) Whether users clicked links in the expert window (b) Whether they got the relevant MCQs correct The next measure is clicking on “expert links”, which showed how experts might respond to questions. [6 sec] (Card 5, MSK Trauma)
14 CAM-C: SUPPLEMENTARY HYPERLINK CLICKIn the presence of supplementary links/hyperlinks to external content, does the rate of clicking through the links correlate with learning? What we need is: (a) Whether users clicked links (b) Whether they got the relevant MCQ correct Third was clicking on hyperlinks, and whether this predicted better assessment outcomes. [5 sec] (Card 15, MSK Trauma)
15 CAM-D: MAGNIFYING IMAGESDoes the rate at which one magnifies an image or images correlate with learning? Our hypothesis might be that those who magnify the images had a higher rate of correct answers on relevant MQCs… What we need is: (a) Whether users clicked magnification button (b) Whether they got the relevant MCQ correct Fourth was magnifying or zooming in on images, and whether this behavior predicted better outcomes. [6 sec]
16 CAM-E: TIME SPENT Does the length of time on an image or case correlate with learning? Our hypothesis might be that those who spent more time on images/cases have a higher rate of correct answers on relevant MCQs… What we need is the duration of time spent viewing each: (a) image/case/webpage (b) success on MCQ (Card 5, MSK Trauma) And finally, the fifth measure was time spent on each card, and whether this behavior had a relationship with performance. [7 sec]
17 EXAMPLE EXPECTATIONS NUMBER OF HYPERLINK CLICKSQUARTILE 4 QUARTILE 3 QUARTILE 2 QUARTILE I In terms of expectations, we might expect that more clicks on links and images would predict better outcomes, since those learners engaged with more content. [9 sec] SUCCESS RATE ON MULTIPLE CHOICE QUESTIONS
18 EXPECTATIONS CAM-E: TIME SPENT SUCCESS RATE ON MCQSQUARTILE 3 QUARTILE 2 SUCCESS RATE ON MCQS QUARTILE 4 QUARTILE I And more time spent on a card would also be expected to result in better assessment outcomes, with the exception of very long times, as these might indicate off-task behavior. So, what did we find? [Hand off to Charlie] [15 sec] AVERAGE DURATION OF TIME ON CARDS/CASES
19 MODELING & ANALYSIS Having hypothesized about which activities would correlate with assessment performance, we set out to build models to test them.
20 MODEL OVERVIEW CAM Engagement Measures: Assessment Performance:Expert links clicked Hyperlinks clicked Magnifier buttons clicked Time spent CAM Engagement Measures: There were two investigations. Both involved using engagement measures to predict assessment performance Assessment Performance: Question answered >50% correctly
21 COURSE CONTENT MAP Here is a map of the content for the course we studied, with each card’s number and topic category Assessments are highlighted in green.
22 ENGAGEMENT ACTIVITIESHere is where those engagement activities were distributed. Not every engagement activity was available on every card
23 ENGAGEMENT ACTIVITIESFor example, card one only had an external hyperlink on it, so we only had insight into that click and time spent
24 ENGAGEMENT ACTIVITIESBut card 16 had all three types of activity, plus time spent
25 EXAMPLE: CARD 16 Magnification External Hyperlink Expert Opinion linkShown here Expert Opinion link
26 Is there a relationship between student engagementINVESTIGATION 1 Is there a relationship between student engagement and performance? First investigation was to test our assumption that more engaged students performed better
27 BUILDING A MODEL INVESTIGATION 1To answer that question, we broke up the course into units...
28 INVESTIGATION 1 BUILDING A MODEL And for each unit...
29 BUILDING A MODEL INVESTIGATION 1 =The model predicts whether a student will pass the end-of-unit assessment given...
30 BUILDING A MODEL INVESTIGATION 1 =Given whether the student clicked on any of the available links,
31 INVESTIGATION 1 BUILDING A MODEL = Any of the magnify image buttons,
32 INVESTIGATION 1 BUILDING A MODEL = Any of the expert links
33 BUILDING A MODEL INVESTIGATION 1 =And how much time the student spent on each card. But since we expected a nonlinear relationship between time and performance, we split time spent on card into bins: and had indicators for each
34 Top half of students spent 23% longer per card than the bottom halfINVESTIGATION 1 RESULTS Independent Variable Odds ratio 95% Conf. Interval Hyperlink clicked 1.21 (1.12,1.31) Magnify image clicked 1.20 (1.11,1.31) Expert link clicked (1.05,1.39) < 20 seconds per page 0.74 (0.66,0.83) >100 seconds per page 1.38 (1.27,1.51) We tried several models, including decision trees and logistic regression for various transformations of the data, but the best-performing model was logistic regression, [with AUC of 0.594]. *** All features statistically significant As expected, engagement and performance were related. Students that rushed through the cards had lower performance on assessments. But that’s relatively obvious. What we really want to know is, which of the materials provided are useful to the students? Top half of students spent 23% longer per card than the bottom half
35 INVESTIGATION 2 Which engagement activities impacted assessment score?Yes studying helps pass tests, but which materials are useful and which are not? Which materials should be removed and replaced with others?
36 EXPERT PREDICTIONS INVESTIGATION 2First Dr. Pusic provided his expert opinion, predicting which materials would be most useful to students in answering subsequent assessment questions. Darker colors are expected to be more relevant For example he predicted that card 16’s materials, which were about the Hip, would not be useful for the assessment on card 19, which is about the shoulder.
37 ORIGINAL MODEL INVESTIGATION 2 =Like before, we broke up the course into units...
38 NEW MODEL: UNIT 1 INVESTIGATION 2= But this time built a separate model for each unit. Unlike the first model, which lumped together any engagement activities, Now we are looking for specific activities that contribute to performance To do that, we consider each event separately so that we can see its impact Investigation 2: Procedure Run lasso-regularized logistic regression using all activities before assessment card Find largest regularization parameter that is close to maximum cross- validation AUC Re-run logistic with remaining variables Return variables that have significant impact with p-value < 0.05 Which features are significant predictors of passing assessment?
39 RESULTS INVESTIGATION 2 =Then after running a model, we record which activities were significantly correlated with passing the assessment In this unit’s model, we find that students who clicked the magnifier on card 5, or spent more time on cards 2,3,4,5, were more likely to pass the assessment on card 5. But the other engagement activities were not significant predictors of passing probability
40 RESULTS INVESTIGATION 2 =Comparing the model’s results to Dr. Pusic’s predictions shows where the predictions were and were not supported by the data.
41 RESULTS INVESTIGATION 2Repeating that step for all units, we have this chart. We can use these insights about what materials are NOT predictive of good performance, and can recommend that instructional designers replicate what is working and replace what isn’t Notable Observations:...
42 RESULTS INVESTIGATION 219. Dr. Pusic’s predictions for card 19 were consistent with model results. You need to engage with that material in order to answer the question correctly But 17 - Card 17, which is about the wrist, should be predictive of performance on card 21; but perhaps there was too many cards between it and card 21 for students to see the relevance 1 - Card 5 Cards 1 and 2 comprise general content, and do not cover the assessment topic of ankles. These were understandably not predictive. 2 - Card 9 Engagement with card 6 and 8 was not associated with improved performance. Instructional designers should reconsider their inclusion. 3 - Card 12 Card 11 was expected to be fully relevant by the content expert, but the model did not consider its content predictive. 4 - Card 15 Few predictive variables were observed in this unit. Only time spent on the assessment card proved significant.
43 CONCLUSIONS Expert predictions not always supported by dataFeedback empowers instructional design Evidence that studying works Limitations: Retroactive study, no control over data collection This is a retroactive study on MedU historical data; database designs could benefit from prospective research goals, to avoid limitations on data collection. Conclusions/Strength of Innovation: Our intention was to demonstrate the merits of learning analytics within the online context, giving educators a new tool for improving experiences in educational online learning environments. Results of this analysis, where the data from thousands of learners are summarized, can serve as feedback to instructional designers as to which interaction elements are effective. It may also be useful to show students themselves evidence that there is a statistically significant relationship between engaging with the material and performing well on assessments.
44 Grant/Research support and historical MedU data provided by:DISCLOSURE Grant/Research support and historical MedU data provided by:
45 for research support and historical data.THANK YOU Special Thanks to MedU, for research support and historical data. For further information, feel free to contact us at: Matt Cirigliano, MS NYU Steinhardt - CREATE Lab Charlie Guthrie, BA NYU Center for Data Science Martin Pusic, MD, PhD NYU School of Medicine Division of Education Quality and Analytics (DEQA)