1 Real-Time Visual Analytics to Evaluate Online CollaborationJohn McCormick | Sarah Krongard

2 Agenda Learning analytics defined Discourse and collaborationSocial Networks Adapting Pedagogical Practice (SNAPP): A tool for visualizing interaction Future of discourse learning analytics tools Q and A

3 Horizon Report: Emerging Technologies

4 Mobile Learning Tablet PCs Learning Analytics (1-2 yrs) (1-2 yrs)iPad Learning Analytics (2-3 yrs)

5 Definitions The measurement, collection, analysis and reporting of data about learners… for purposes of understanding and optimising learning and the environments in which it occurs - 1st international Conference on Learning Analytics and Knowledge (LAK 2011) …selection, capture and processing of data that will be helpful for students and instructors at the course or individual level. …able to adjust content, levels of support and other personalized services by capturing, reporting, processing and acting on data on an ongoing basis in a way that minimizes the time delay between the capture and use of data. -Elias, T. (2011). Learning Analytics: Definitions, Processes and Potential

6 Definitions (cont.) Learning Analytics:is the analysis of many kinds of learner-produced and learner-related data; seeks to monitor learner activity and progress and to predict learner outcomes; and enables interventions and decision making about learning by instructors and students. - From Educause/ELI Focus Session on Learning Analytics (2012)

7 Example 1: Signals: PurdueSource:

8 Signals: Results “When instructors follow recommended Course Signals best practices, grade performance improvement has been consistently demonstrated at both the course and departmental level.” 10% increase in numbers of As and Bs awarded in semester-long courses, and a 6.41% decrease in Ds, Fs and withdrawals. Greatest improvement seen in students who were initially receiving Cs and Ds in early assignments, and pull up half a letter grade or more to a B or C. “…the overwhelming response from the students is that Course Signals is a helpful and important tool that aids in their overall academic success” Arnold and Pistilli (2012)

9 The seven dimensions of ‘Learning Power’Example 2: Effective Lifelong Learning Inventory (ELLI) The seven dimensions of ‘Learning Power’ Critical curiosity Changing and learning Meaning making Creativity Learning relationships (interdependence) Strategic awareness Resilience

10 Source: http://www. slideshare

11 Source: http://www. heacademy. ac

12 Spider Diagram

13 ELLI Research Findings from research and development projects indicate that learners who are aware of their learning power profile using ELLI and who have structured learning opportunities to improve it, become: More reflective and aware of themselves as learners More confident and motivated to learn More willing and able to take responsibility and ‘own’ their learning More keen to carry on learning More likely to attain higher levels and grades More likely to fulfil their potential More sociable and able to learn with and from each other More able to manage change and transitions Source: My Learning Power: Using ELLI in Higher Education

14 Enquiry Blogger Source:

15

16 Instructor Dashboards

17 ELLIMent Source:

18 Academic vs. Learning AnalyticsLong, P. & Siemens, G. (2011). Penetrating the Fog: Analytics in learning and education.

19 Initial Thoughts about Learning Analytics?

20 Discourse & Online CollaborationWhat is the value of collaboration in an online environment? What are the challenges & opportunities do instructors face in facilitating online collaboration? To what extent have the roles of instructor and student changed?

21 The Importance of CollaborationTo refine and confirm personal understanding with a community of learners To develop, foster, and practice higher-order thinking skills To share and co-construct knowledge to support emerging collectives Research in face-to-face & mediated educational contexts confirms the benefits of collaborative learning in supporting higher-order learning (Garrison, 20) A collective is a “collection of people, skills, and talent that produces a result greater than the sum of its parts

22 Online Collaboration Challenges Opportunities Increased WorkloadDifficulty of Encouraging and Sustaining High-Level Discourse Asynchronous Interactions Lack of F2F Connections Challenge of Self-Motivation Technical Difficulties Time Allocated for Thoughtful and Reflective Dialogue Permanency of the Written Word Tools Available to Facilitate Interaction and Engagement Opportunities for Student Agency and Efficacy

23 Making the Shift in Roles from…… Traditional Instructor to Guide and Facilitator … Presumed Authority to Collective Credibility … Student as Passive Recipient to Active Producer and Co-Creator

24 (Dooley, Lindner, Dooley, & Murphy, 2005, p. 49)“Learning is in social relationships – situations of cooperation with others… Both the learner and the community in which they participate are transformed by the community as a complementary unit that work in harmony to promote learning…” (Dooley, Lindner, Dooley, & Murphy, 2005, p. 49)

25 (Garrison, 2011, p61) Teaching: Interaction and discourse play an essential role in a community of inquiry and it has been shown that teaching presence is crucial to ensure participation and quality of responses (An, Shin, & Lim, 2009; Bliss & Lawrence, 2009; Gorsky, Caspi, Antonovsky, Blau & Mansur, 2010); Strong teaching presence “is related both to students’ sense of connectedness and learning” (Shea, Li, & Pickett) Social Presence: “Not only is teaching higher-order cognitive skills more successful when co-operatively based (Resnick,1987), cognitive presence is also enhanced and sustained when social presence is established.” (Akyol, & Garrison, in press; Fabro & Garrison, 1998; Gunawardena, 1995; Liu, Gomez, & Yen, 2009) Once social presence is established, learners can feel safe and comfortable enough to build cognitive presence, allow opportunities to asking questions and connecting intellectually with peers. Important to achieve a balance among social/cognitive/teaching.

26 SNA Concepts: CentralitySource:

27 Node Size: Number of InteractionsSource:

28 Information Brokers

29 Frequency of Interaction

30 SNA and Online DiscourseVery little SNA research on online discussion for learning General Findings: More evenly distributed activity beneficial Low hierarchical structures better (groups interacting, teacher not playing central role) More interaction not necessarily better Toikkanen , T. & Lipponen, L. (2011). The applicability of social network analysis to the study of networked learning. The Journal of Interactive Learning Environments, 19 (4),

31 Social Networks Adapting Pedagogical Practice (SNAPP)

32 Other SNAPP metrics (1) Additional SNAPP metrics include the number of total number of posts by participant and a variety of centrality measures. First version features- SNA terminology Network Density- total number of interactions (not necessarily useful) Degree Centrality- posts in and out (different types of centrality- degree seems to be most important in terms of discussion interaction) Betweeness-number of times a student acts as bridge along shortest paths between to others. Eigenvector- influce of that node overall

33 What SNAPP can tell you Dawson, S., Bakharia, A., Lockyer, L., & Heathcote, E. (2011). ‘Seeing’ Networks: Visualising and evaluating student learning networks. Australian Learning and Teaching Council, Canberra, Australia.

34 Dawson et al findings: Instructor use“extremely effective in promoting reflection on teaching activities and rapidly assessing overall effectiveness..” “However, teachers require additional support and training to better understand the relationship between their implemented online learning activities and the resultant online student behavior…” Dawson, S., Bakharia, A., Lockyer, L., & Heathcote, E. (2011). ‘Seeing’ Networks: Visualising and evaluating student learning networks. Australian Learning and Teaching Council, Canberra, Australia.

35 Interpretation and Learning Design“Future research and professional development should seek to merge learning design with learning analytics and data interpretation. A sound understanding of learning design principles will greatly assist academics in better interpreting the learning analytics data into informed pedagogical action”.

36 Action Research Steps Create visual taxonomy Choose instructorsCollect data Send “Visual Key” and “Custom Report” Interview instructors Follow- up with instructors

37 Visual Key

38 Visual Features (filtering)

39 Instructor-Led

40 Learning Community

41 Combined Patterns

42 Learning Community with strong instructor presence

43 Emerging Learning Community

44 Emerging Learning Community with strong instructor presence

45 Weak Learning Community

46 Learning Community Development

47 Two Facilitation Styles

48 Facilitator Role: Central vs PeripheralInstructors critical to success of discussion Blignaut & Trollip, 2003; Ferdig & Roehler, 2004; Greenlaw & DeLoach, 2003; Love, 2002; Roblyer & Wiencke, 2003; Kearsley, 2000; Arnold & Ducate, 2006; Darabi etal., 2011; Ertmer et al., 2011; Liu & Yang, 2012; Instructors should take a back seat, let students construct own knowledge Burstall, 2000; Li, 2003; Mazzolini & Maddison, 2003; Poole, 2000; Rourke & Anderson, 2002; Paloff & Pratt, 2001; Wang & Chen, 2010, Baram & Correia, 2009

49 Learner Population: Undergraduate BusinessTypical Pattern: Learning Community with strong instructor presence Intervention Suggestions: Reduce number of instructor postings, add summary or “weave” following discussion, use student-led facilitation, organize discussions with use of titled posts. Intervention Plans: Reduce instructor posting in some discussions and better organize initial posts.

50 Instructor: High Centrality, Low In-Degree

51 Future learning analyticsSource:

52 Meerkat-ED

53 Group interaction over time

54 Networks of Terms

55

56 Individual Discussion Thread: Click on edges

57 Natural Language Processinghttps://www.coursera.org/course/nlp

58 Dashboards: Desire2Learn

59

60 “The ties between people make the whole greater than the sum of its parts”Nicholas Kristof, Sociologist

61 Questions, Comments, Answers?