1 The Challenges and Opportunities of Information Lifecycle Management (ILM)Bob Rogers Application Matrix Founder & CTO Co-Chair SNIA ILM Initiative
2 Abstract Information Lifecycle Management (ILM) does not come in a box; nor is that likely any time in the near future. The automation of information management policies regarding protection, security, and retention require a different way of thinking about information, its value to the organization, and the business needs for the use of the information. This presentation will touch on current issues and opportunities and suggest technologies poised to benefit from ILM tools and techniques.
3 Information Lifecycle Management Defined*The policies, processes, practices, and tools used to align the business value of information with the most appropriate and cost effective IT infrastructure from the time information is conceived through its final disposition. Information is aligned with business processes through management policies and service levels associated with applications, metadata, information, and data. * Storage Networking Industry Association (SNIA) ILM Technical Work Group
4 What Does Information Lifecycle Management Mean?Automation of information management policies ILM is Multidisciplinary Management based on Service-Level objectives/requirements The Storage industry has focused on ILM because of the persistent nature of data
5 History ILM concepts are more than 25 years oldIEEE Mass Storage Reference Model dates from the early 80’s GUIDE “Futures of Storage Management” White Paper – 1983 Rate of Information Growth and retention is accelerating Merrill-Lynch study UC Berkeley study SNIA 100 Year Archive study
6 Information Growth Storage TCOExternal disk storage purchase projected to grow at 52% annually Capacity is #1 storage issued driven by , unstructured data Significant transition to disk-based archival storage Digital archive capacity will increase nearly tenfold between 2005 and 2010 54% CAGR 79% CAGR 68% CAGR * Sources: Merrill Lynch storage forecast & views from CIOs, Enterprise Strategy Groups 2006 Digital Archive study
7 SNIA 100 Year Archive Task Force HighlightsLong-term retention needs are real and that many organizations have long-term requirements. 80% of respondents declared they have information they must keep over 50 years 68% of respondents said they must keep it over 100 years. Long-term generally means greater than 10 to 15 years – the period beyond which multiple migrations take place and information is at risk. Database information (structured data) was considered to be most at risk of loss. Over 40% of respondents are keeping records over 10 years. is not just a short-term problem. Physical migration is a big problem. Only 30% declared they were doing it correctly at 3-5 year intervals. The rest of the sample group is placing their digital information at risk. 60% of respondents say they are ‘highly dissatisfied’ that they will be able to read their retained information in 50 years. Help is needed – current practices are too manual, too prone to error, too costly and lack adequate coordination across the organization. Collaboration and classification were recognized as very important practices to get the organization working together setting requirements for the management of their information. Report available at:
8 Information is data with context ILM Principles Information is data with context Information is competitive advantage The business unit understands information And its value – or cost if it is missing The infrastructure will eventually be virtual So ultimately, it’s all about the data Resources & processes don’t scale with growth Replace the processes Automate the infrastructure
9 An ILM Framework for the DatacenterInput Requirements Regulatory Corporate guidelines Information Services Budget Rationalize Goals Apply policies Balance Risk vs. Cost Deliver cost-effective infrastructure Manage the data Monitor Adjust Report Business Requirements Requirements Define Goals Management Requirements Business Process Business Framework Information Policies, Instrumentation, Filters One of the first issues we were faced with when we started this effort, was to address how ILM related to business processes. How do we show a relationship between what individual lines of business want to achieve – their goals and requirements, and the information that is used to support those objectives. So we started with Business Requirements, and we said that in the end, we will be able to illustrate the relationship between business requirements and ILM. The next thing we knew we needed to do was abstract policy management into a higher layer function that could provision, coordinate, and manage the operations of the data center. This is “Goals Management” and there’s lot s more to say about that later. Then came my favorite discussion: how big is ILM? How much of the IT Infrastructure will ILM apply to. There turned out to be relatively little debate about the scope of Information Lifecycle Management. The scope of ILM is such that you could eventually manage your entire data center with the policies, practices, and tools that ILM will provides. Right about now, you probably have visions of vendors trying to boil the ocean again. Let me assure you that there is no silver bullet here, there is no magic pill, and it will be a lot of work. However, on the plus side, each of the vendors involved is committed to finding practical solutions to this problem for the simple reason that today’s data centers will not scale to the requirements of tomorrow. Automation needs to be brought to bear. I’m going to walk you through the rest of this in more detail in the next slides, but first let me complete the high level picture. The ILM Framework is comprised of Goals Management which provides management and control of the IT Infrastructure. We based our IT Infrastructure on three legs of a stool: Network, Compute, and Storage infrastructures. Since we are “Information” Lifecycle Management, we built the other layers on top of those three legs from an information perspective. Data Management Services provides data-related functions which requires knowledge about data in a context-free way. Functions such as backup and restore, data migration, replication reside at this layer. Information Management Services, on the other hand, is all about content and context. Functions at this layer provide index and search capabilities, records management, etc. Finally, you have the applications themselves. Applications are not just databases, per se, but rather, they are the higher level business applications that may make use of databases, file systems, etc. It was mentioned earlier that we needed to tie ILM back to the lines of business. That’s where we made the last changes to this model. Note that we defined information to be data conveyed within the context of an application. It has meaning when it is created and when it is used. Business processes are what use Information. And while Business Requirements may always drive some aspects of IT Goals Management, such as for corporate governance, the fact of the matter is that each business process must define the IT service level requirements for information that it creates and uses. This completed the tie-in between ILM and the lines of business. This is where the relationship needs to be forged between IT and business, and it’s where automation needs to come into play to enable IT staffs scale with growing technology requirement. Before we move on to the next slide, let’s take a moment to talk about something that may be obvious to some of you already. ILM is an overlap with Utility Computing when it comes to managing the data center. While UC is approaching the same problem of automation from the compute perspective, ILM is approaching it from the information perspective. It’s also worth noting that for either UC or ILM to succeed, we need to achieve success with introducing service level management to the data center. Applications Information Management Services Data Management Services ILM Framework Network Infrastructure Compute Infrastructure Storage Infrastructure IT Infrastructure
10 Challenges Information Classification Metadata ManagementAutomated application of management policies
11 Policies Arise from Service Level ManagementService Level Agreements (SLAs) and Operational Level Agreements (OLAs) defined by users Availability Performance Security Reliability Policies carried out by systems
12 Classification and Metadata ManagementInformation Classification vs data classification Multiple disjoint taxonomies Need for ontological approach Information does not recognize system or organizational boundaries Public Metadata repositories Preservation Metadata efforts Other Metadata
13 Automation of Management PoliciesNeed to separate “policy engine” from “policy execution” Automation tools tend to be very specific Resource-centric vs Business-service-centric No commonality of policy “decision points”
14 Information Formats and MediaSNIA project to develop a “self describing data format” Long term retention without media refresh or reformatting
15 Effects of low adoption rate of ILM technologyImpedes “Green Data Centers” Jeopardizes information security Increases likelihood of lost data Inflates IT costs and decreases productivity
16 For More Information… SNIA Whitepapers SNIA EducationInformation Lifecycle Management Roadmap Storage Service Management: The Foundation for Information Lifecycle Management 100 Year Archive Requirements Survey SNIA Education SNIA End-User Community Collaboration Projects Maturity Model Storage Service Metrics End-User Experiences