1 UT System: Using Data to Drive ChangeEDUCAUSE Dr. Stephanie Bond Huie Vice Chancellor ad interim for Strategic Initiatives February 12, 2013
2 Begin with the Data Collection Processing DeliveryStopped collecting some data but began to systematically collect a whole lot more Processing Still adapting and transitioning, but new tools will ultimately streamline operations Delivery Major transition in methods of data delivery Major expansion in analytics options Key to creating a culture that uses data to drive decision-making and change In data collection and processing we have seen substantial internal changes in our operations that have come with the new, more sophisticated tools we have adopted. We are collecting more data, but we are being smart about it. Streamlining and automating. And that includes our processing of the data. But it is in the delivery of the data that most of the changes are visible.
3 Data Delivery: Old vs. NewThe Old Way = Before 2012 Annual publication of a statistical handbook (or similar) Periodic publication online of special reports or briefs (PDF/HTML) On request, provide Excel spreadsheets and/or create graphs The New Way = After 2012 No more statistical handbook! Continue to publish special reports/briefs but now create supporting topical dashboards Data updated as it becomes available New tools deliver online, dynamic and interactive reporting Our office has always delivered data. In printed fact books or PDFs you could view online. Now, it’s much more consumable. Get what you want when you want it. Updated as data becomes available versus on a once-a-year schedule.
4 Data Delivery: The New (for us) ToolsBI/Information Delivery Portal data.utsystem.edu Dashboards Web reports Online, publicly accessible Visual Analytics exploredata.utsystem.edu Data visualizations Analytics Mobile BI, publicly accessible We actually call the entire project “The Dashboard Project” even though dashboards are only one of the tools involved. As we have implemented them, dashboards provide a current-year view at a high level across a variety of metrics. Although we have not used the traditional stoplight indicators so far, as we finalize goals set by the institutions, that approach will get more use. Each indicator on the dashboard links to a web report and an online report that allows the user to dig deeper into the data. This can mean the ability to drill down into the data. Or it may mean presentation of related data or different views of the same data. The user can sometimes build their own graph or table, or filter those already built. The user can also export the data to excel or to PDF. With this tool, we have provided our user with a basic data delivery, analytics, and visualization tool. Visual Analytics is our newest tool. These reports are online, and, picking up where the other tools leave off, a mobile app is due for release. VA takes the basic analytic tools available in our web reports and brings them to the next level. We anticipate that this tool will be widely used, particularly when the app is available.
5 Data Delivery: What’s on the DashboardExecutive Dashboard CORE INDICATORS Topical Dashboards Student Success Faculty Productivity Research & Tech Transfer Finance / Productivity & Efficiency Health Care Institutional Profiles College/Department Profiles Special Topic / Specific Need Dashboards Graduation Success Affordability Emerging Research Universities Diversity Information Dashboards for Specific Needs
6 Data Delivery: New in Visual AnalyticsFirst reports just went live on the web Total Operational Revenue and Average Revenue per FTE Student Average Debt for Graduating Students (bachelor’s) Research Expenditures by Source Tuition and Fees exploredata.utsystem.edu iPad app to be delivered soon
7 Using Data
8 Using Data: Peer Selection, Benchmarking & Goal-SettingUsing Data to Select Peers Baseline Peers – institutions statistically similar to you now Aspirational Peers – institutions that are now what you plan to be in the long-term (10-15 years) Benchmarking is critical to evaluating institutional performance Benchmark as starting point (you must know where you started so you can evaluate how far you have come) Benchmark as context (performance relative to what) Goal-Setting Top quintile of baseline performers as mid-range goals Long-term goals set at bottom quintile of aspirational peers
9 Need to reduce variablesUsing Data: Peer Selection A New Model for 2012 Original Model Factors New Factors More than 40 Variables Need to reduce variables Correlation matrix created Carnegie Classifications 15-19 3-year averages SAS PROC FACTOR VARIMAX option Undergrad % minority Graduate % minority Undergrad % Hispanic Graduate % Hispanic Operational revenue per FTE % part-time enrollment 12 Rotated Factor Loadings Full-Time Instructional Faculty Professor Salary (3-yr Avg) Total Enrollment Operational Revenue per FTE (3-yr Avg) 75th Percentile SAT Undergrad % Minority (3-yr Avg) % Pell Eligible % High Cost Degrees (3-yr Avg) Degrees Awarded as % of Total Degrees: Bachelor's Undergrad Enrollment as % of Total Enrollment Full-Time Enrollment as % of Total Enrollment Degrees Awarded as % of Total Degrees: Graduate Distance Scores & Finding Similar Institutions: Computed distances between all universities. Can order all 378 universities relative to their distance from a chosen one. Data is part of almost all aspects of our work--it’s not just about the data presented at the end. For instance, because peer comparisons are central to evaluating institutional performance, it was critical that the peer selection process be data-driven to create a statistically similar baseline against which to make those comparisons. [Original model factors include more than two dozen variables related to institutional size, student population, program mix, and research focus.] SAS PROC DISTANCE
10 Using Data: BenchmarkingPutting Delta Cost in Context: The Importance of Peer Comparisons Once a set of peers is statistically selected, you can provide the context—the benchmarking. Benchmarking against the institution’s baseline peers makes all the difference when evaluating performance. For example, in this first graph you can see the cost per degree at our institutions. But what does it mean? These are the data points, but the information they provide is limited.
11 Using Data: BenchmarkingPutting Delta Cost in Context: The Importance of Peer Comparisons Adding peer data adds information and provides the context that allows for a more meaningful evaluation.
12 Using Data: Goal-SettingFurther application of the peer data can be used for goal setting. Here, you can see UT Austin’s graduation rate over time in comparison with their peers: the bottom quintile, average, and top quintile. This kind of visual allows you to see the future goals in relation to peer performance—are you getting closer to the average, and, eventually to the top performers? For a series of 6-7 metrics that includes graduation rates and bachelor’s degrees awarded, the institutions were required to set goals that would bring them to the top quintile of their baseline peers by 2020.
13 Using Data: Informing Policy-MakingSupport regental task forces Task Force on University Excellence and Productivity Student Debt Reduction Task Force Task Force on Engineering Education Provide data for ad hoc requests Internal – board, chancellor, officers External – legislature, media, others Not just data Data is just data Research and analysis transform the data into information Visualization and presentation make that information consumable Examples follow
14 Supporting Regental Task Forces: Student DebtExtensive data was provided to the task force on student debt. This chart and the next are two examples of that information. National and state averages were provided as context. This is valuable, although less meaningful than more focused comparisons. Sometimes, however, the availability of the data limits the comparisons that can be made.
15 Supporting Regental Task Forces: Student DebtReal policy implications. This task force presented an objective, evidence-based discussion of the issues and recommended solutions for consideration. There was real support in terms of dollars for tools that could help students. Trends in this area will be monitored.
16 Supporting Regental Task Forces: Engineering EducationThe task force on engineering education was announced in November. The goal of this task force is to determine the current state of engineering degree programs in Texas, study current and future demand for engineers, and identify strategies that will foster student success in the field and support economic growth. Already there has been extensive data collection and basic analysis. Size of bubble is determined by % of entering cohort that was in the top 25% of HS class.
17 Supporting Regental Task Forces: Engineering Education
18 Supporting Regental Task Forces: Engineering Education2.8 million holders with engineering degrees Work-Life Earnings of $3.6 million Bachelor’s = $3.3 million Master’s = $3.9 million Doctorate = $4.2 million We are only just beginning to dig deeper on the information we’ve collected so far, as well as determining what other data exists out there that might be useful.
19 Using Data: Informing Policy-MakingBe Data Smart Data is just data Research and analysis transform the data into information Visualization and presentation make that information consumable Beware Data Marketing Data should tell a story, but only in the sense that the visualizations presented should accurately reflect underlying patterns Not all data consumers are data savvy Use good data practices and be consistent
20 Thank you