1 Patch Warfare & Security Incident ResponseThis presentation was created in October 2003 (and updated in April 2004) to provide a comprehensive overview of Microsoft’s Update Management solutions and roadmap. The slides in the appendix section of this presentation contain additional detail and pointers to resources for various aspects of patch management and related security topics. The main section of the deck has several slides with builds so it’s recommended that you do a first viewing of the deck in slideshow mode. Note: This presentation addresses the broader area of Patch and Update Management, i.e., the management not just of patches, but of different types of software updates. For space saving reasons the shorter term ‘Patch Management’ is often used in these slides. All references to ‘Patch Management’ should be interpreted as references to ‘Patch and Update Management.’ Microsoft Corporation Presented by Robert Hensing - PSS Security Specialist © 2002 Microsoft Corporation. All rights reserved. This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.
2 Agenda Situation Solution Components RoadmapSecurity Incident Response We’ll begin by discussing the current situation with update management for Windows environments. I’ll talk about the feedback customers have given Microsoft and outline what Microsoft is doing – and has done thus far – to address the situation. Next, I’ll provide an overview of Microsoft’s current patch management offerings. The goal here is to clarify what the offerings are, what capabilities they provide for the various steps in the patch management process, the benefits they provide, how they work, and how to choose the offering that best meets your needs in cases where more than one offering provides similar functionality. In the roadmap section, I’ll discuss the various short-term and longer-term milestones and deliverables in each of the areas impacting patch management, from communications and training, to improving the patching experience, to the roadmap for Microsoft’s patch management products. © 2002 Microsoft Corporation. All rights reserved. This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.
3 Customer Feedback Reduce Frequency, Quantity of PatchesInadequate Communications, Guidance, and Training Inconsistent Patching Experience Reduce Frequency, Quantity of Patches Let me start by sharing with you the feedback Microsoft has received from customers. We’ve collected this feedback – both solicited and unsolicited – from hundreds of customers and have categorized it into 5 areas. The first relates to communications, guidance, and training. Customers tell us that we need to do a much better job of providing accurate, discoverable, simplified, and timely information on new patch releases. They are also looking for better guidance on how to enable effective patch management for their Windows environments and want better training on how to effectively use Microsoft’s patch management tools. The second area of feedback relates to the patching experience. Customer tell us that the experience of discovering, applying, and rolling back patches lacks consistency across Microsoft products and update types. We have too many different patch installers, each with different usage switches, capabilities, etc. There’s also no consistent way for installing a patch or update – no consistent set of registry setting or files that are changed, no repository to tell which updates or patches are installed, etc. The third area that’s causing customers pain is patch quality – the size of released patches, the proportion of patches that require a reboot on installation, and the number of patch recalls Another area of feedback is patch management products & tools. We have clear feedback that we have too many overlapping offerings, that none of them provides end-to-end functionality for patch management or effective support for patching all Microsoft products, and that it’s not clear what criteria customer should use in choosing between the products. Finally, the area of feedback that customer have been most vocal about is that Microsoft needs to reduce the frequency and number of patches it releases As you can tell, the customers we’ve spoken to have been very direct about what they want us to do. We are taking their input very seriously. Now, let me share with you what we’re doing in response to these customer needs Multiple, Incomplete Patch Management Tools Inconsistent Patch Quality © 2002 Microsoft Corporation. All rights reserved. This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.
4 Addressing The SituationSecurity and patch management priority #1 – bar none – at Microsoft Microsoft problem Industry problem Ongoing battle with malicious hackers Microsoft taking a comprehensive, tactical and strategic approach to addressing the situation Patch Management Initiative The first thing I’d like to make crystal clear is that addressing the situation around security and patch management is the number one priority at Microsoft. There is nothing that comes close in term of the sense of urgency and the level of focus. It’s a top priority at the highest level at Microsoft, starting with CEO Steve Ballmer and down to people at every level in the company. We recognize that this is a Microsoft problem and that we have to rise to the challenge and address it to the satisfaction of our customers. We also recognize that it’s an industry problem – vendors other than Microsoft are also being impacted – and that we need to work with other industry participants to address this situation going forward. We also see this as an ongoing battle with malicious hackers and we understand that we need to address the situation on multiple fronts to thwart the ability of these hackers to gain the upper hand in this battle. The other point worth noting is that Microsoft is not just taking a reactive approach. We are keenly aware that an intelligent, comprehensive approach to address the situation is essential to success. Yes, we have to take tactical short-term measures to the ease the pain our customers are experiencing today, but we are also taking a long-term approach in terms of rethinking product architecture and design, new options for securing the IT environment and mitigating potential vulnerabilities without having to patch systems, putting in place longer term alliances, programs, etc., to enable a quantum leap forward in addressing the situation. This work is part of the Patch Management Initiative, an initiative focused on delivering dramatic improvements in the various areas of patch management. The initiative is being driven by a cross-divisional, cross-functional team of experts in the various areas of patch management. I’ll talk about the goals, accomplishments to date, and future deliverables of this initiative in the next couple of slides and the roadmap section of this presentation. © 2002 Microsoft Corporation. All rights reserved. This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.
5 Progress to Date (July 2004)Informed & Prepared Customers Rationalized patch severity rating levels Better security bulletins and KB articles Security Guidance Kit; Patch Management guidance, etc. Security Mobilization Initiative – 500K IT Pros trained Consistent & Superior Update Experience Standardized patch and update terminology Standardized patch naming and installer switch options* Installer consolidation plan in place – will go from ~8 to 2 Reduced patch release frequency from 1/week to 1/month Superior Patch Quality Improved patch testing process and coverage Expanded test process to include customers Reduced reboots by 10%; reduced patch size by up to 75%** Best Patch & Update Management Solutions Released SMS 2003 which delivers expanded patch and update management capabilities Released MBSA 1.2 which integrates Office inventory scanning Windows Update Services in development The boxes on the left represent the high level goals for the Patch Management Initiative, and you’ll notice that they relate directly to the areas of customer feedback. The first goals relates to doing all we can to ensure that customers are adequately equipped to manage the security of their Windows based environments. It’s about providing the right information, at the right time, and providing the required levels of process and best practice guidance as well as product training. The second goals is to dramatically enhance the patch management experience, to deliver a consistent and superior patching experience The third goal is to achieve dramatically improved patch quality – quantum improvements in patch size reduction, fewer patches requiring reboots, and a minimal number – if any – of patch recalls. Finally the initiative is focused on rationalizing the set of patch management offerings to provide end-to-end patch management functionality that covers the spectrum of Microsoft products, minimizes the cost of enabling effective patch management, and enables third party products to leverage the core patch management infrastructure in Windows. Now let’s take a look at some of the results of the Patch Management Initiative to date. This is a summary of the results, there’s additional information in the roadmap section of this presentation. [Review & provide additional context for the progress bullet points] More on the deliverables of the Patch Management Initiative in the Roadmap Section of this presentation… *Update.exe now using standardized switches; Windows Installer will use these in MSI 3.0 **75% for Windows Update installs, more than 25% for other patches © 2002 Microsoft Corporation. All rights reserved. This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.
6 Terminology Private Fix Hotfix Update Critical Update Security PatchName Description Distribution Private Fix An unofficial fix which may not be fully tested or packaged. It is released to the customer to verify that it solves the problem before final testing & packaging. Limited to the customer who reported the problem. Hotfix A single cumulative package composed of one or more files used to address a defect in a platform. Limited to customers who contact Microsoft Product support services and are experiencing the specific problem. Update A broadly released fix for a specific problem addressing a non-critical, non-security related bug. Publicly available for download. Critical Update A broadly released fix for a specific problem addressing a critical, non-security related bug. Security Patch A broadly released fix for a specific platform addressing a security vulnerability. Update Rollup A cumulative set of hotfixes, security patches, critical updates and updates packaged together for easy deployment. A rollup targets a specific area such as "security" or component of the platform such as "IIS". Service Pack A cumulative set of all hotfixes, security patches, critical updates, and updates created and fixes for issues found internally since the release of the platform. Service packs may also contain a limited number of customer requested design changes or features. Service packs are broadly distributed and therefore tested heavily. In April, 2003 the sustained engineering cabinet created both the standardized terminology and naming standards. More information can be found here: © 2002 Microsoft Corporation. All rights reserved. This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.
7 Naming Standards Description of the File Names That Are Used for Microsoft Product Updates, Tools, and Add-ins The standardized file naming schema that Microsoft is adopting for packages that contain product updates, tools, and add-ins uses the following format: ProductName-KBArticleNumber-Option-Language.exe WindowsXP-KB IA64-ENU.exe - An update for the English (US)-language version of Microsoft Windows XP for computers with 64-bit Intel processors. The update is associated with Microsoft Knowledge Base article OfficeXP-KB Client-ENU.exe - An update for the English (US)-language version of Microsoft Office XP. The update is associated with Knowledge Base article SQL2000-KB JPN.exe - An update for the Japanese-language version of Microsoft SQL Server 2000 Build The update is associated with Knowledge Base article © 2002 Microsoft Corporation. All rights reserved. This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.
8 Bulletin Severity Rating SystemDefinition Customer Action Critical Exploitation could allow the propagation of an Internet worm such as Code Red or Nimda without user action Apply the patch or workaround immediately Important Exploitation could result in compromise of the confidentiality, integrity, or availability of users’ data, or of the integrity or availability of processing resources Apply patch or workaround as soon as is feasible Moderate Exploitability is mitigated to a significant degree by factors such as default configuration, auditing, need for user action, or difficulty of exploitation Evaluate bulletin, determine applicability, proceed as appropriate Low Exploitation is extremely difficult, or impact is minimal Consider applying the patch at the next scheduled update interval Revised November 2002 More information at © 2002 Microsoft Corporation. All rights reserved. This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.
9 Prioritizing and Scheduling the ReleaseThe priority and resulting schedule for a security release should be determined by taking into consideration the defined MSRC severity level of the vulnerability along with aspects that are unique to your environment. A simple mechanism for determining priority involves mapping the MSRC-defined severity level of the vulnerability to an initial priority level for the release (Table 3.2), then raising or lowering the priority level depending on the needs of your organization and unique aspects of your environment (Table 3.3). To ease the impact on resources performing release management, multiple security changes for the same category of asset can be combined into a single release. This is most appropriate for priorities 2 through 4, as listed in Table 3.1. © 2002 Microsoft Corporation. All rights reserved. This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.
10 Decreasing time in which to deploy a patchA Serious Problem Decreasing time in which to deploy a patch
11 Decreasing Time To Patch (Blaster)July 1, 2003 July 16, 2003 July 25, 2003 Aug 11, 2003 Vulnerability reported to us / Patch in progress Bulletin & patch available No exploit Exploit code in public Worm in the wild Report Vulnerability in RPC/DDOM reported MS activated highest level emergency response process Bulletin MS delivered to customers (7/16/03) Continued outreach to analysts, press, community, partners, government agencies Exploit X-focus (Chinese group) published exploit tool MS heightened efforts to get information to customers Worm Blaster worm discovered –; variants and other viruses hit simultaneously (i.e. “SoBig”) Here is a recap of the events around Microsoft Security Bulletin, MS03-026, commonly known as “O-twenty six” Note: Points made are business centric. RPC / DCOM explanation if needed is below: RPC: Remote Procedure Calls Used to communicate API calls between machines or across processes within a machine When intermachine, runs over port 135 DCOM: Distributed COM Used as the activation mechanism for RPC servers Runs on NT4, Win2K, WinXP, WS03 These technologies are broadly used in the OS, hence the degree of vulnerability due to the bug MS03-026: a new class of loop-based buffer overflow not caught by prior tools Mike’s transcript on the events: The four phases that we’re looking at are the time between when we were first notified of the vulnerability by the Security Research Group, Last Stages of Delirium. That was on July 1st. That period ended July 16 when the patch was first available. The second period of time is the time immediately after releasing the patch before the time that there was a public release of the exploit code. That’s the period of time between July 16 and July 25 of last month. The third period of time is the time between the public release of the exploit code on July 25 and the first discovery of the Blaster Worm on August 11. Lastly, the fourth period of time is work we’ve done since the first time that the Blaster Worm was discovered in the wild. As I said, let me start by talking about the history prior to MS It really did begin for us in the initial phase when we were informed by the Security Research Group, Last Stages of Delirium, on the first of July. They reported to us a vulnerability in the RPCD comp component of Windows. RPC is a core communications protocol used by many Microsoft platforms and by many third party products. RPC stands for Remote Procedure Call. The MS vulnerability is exposed through a mechanism called remote activation, where DCOM is accessed across the RPC protocol. The vulnerability affects Windows NT 4.0, Windows 2000, Windows XP and Windows Server As I mentioned, it was reported to us by a group called the Last Stages of Delirium, also known as LSD. The key thing we did at that time was to immediately activate our highest level of emergency response, one by the Microsoft Security Response Center. We literally had a team working around the clock from the moment we had the report from the security researchers to both build, test and release a patch, and we did that within two weeks because we understood the importance of both having a patch available quickly, but also we wanted to balance the need to have a very high quality patch in a timely period. During the process, though, I also want to make you aware of the fact we did work very closely with the security researcher to follow through on what we call Responsible Disclosure Guideline, and really the goal here is to make sure that they share information with us in a confidential way and then work with us to both develop and test the patch, and together, we go off and share information about the vulnerability and the patch. The real goal here is to make sure that while we get the fix out there and help them to get credit for their great work, we minimize the risk we put customers at. I want to thank the Last Stages of Delirium team for their responsible behavior through this process. Stage two really began for us on July 16 when we delivered the security bulletin, MS03-026, to customers. It included patches for all affected supported Windows versions. That includes Windows NT Server 4.0 Service Pack 6A. Windows NT 4.0 Workstation is not a generally supported platform anymore, but we subsequently added support for that platform because of the fact that so many customers were in that situation. The patch included support for Windows 2000, Service Pack 3 and Service Pack 4, and again, we later announced support of the patch for Windows 2000 Service Pack 2, again, based on customer feedback. The patch is effective for Windows XP, both in the gold or no service pack version, along with Windows XP Service Pack 1, and the patch also was effective against Windows Server I do want to note that Windows ME is not affected and Win 9X versions, which are not supported, are also not affected. On July 17, we updated the bulletin as we learned more information about the potential exploits. I want to be clear that that update to the bulletin included more details about mitigation, new steps that we thought were important for customers to follow to protect themselves in addition to the ones that were in the original bulletin. Of course, the patch itself was the most effective way of protecting the customer, but I want to be clear that the re-release of the bulletin on July 17 was simply more guidance. The patch was unchanged, and the initial patch that we shipped on July 16 was effective even in light of the information and continues to be effective against Blaster and its variants. At the time, we also notified customers through our Security Notification Service. We have customers signed up for that service. We sent from Microsoft account teams to their contact at the customer. We also had our support teams and technical account managers in particular working directly with technical staff and security staff within our customer organizations to make sure things were well understood. We also spent a lot of time using the press to do outreach both to enterprise customers and to consumers, to make sure there was a high level of awareness of both the existence of the patch and vulnerability, but also the importance of this patch in particular getting high priority to protect customers of all sizes. Some of the examples of that outreach – and I’m sure a lot of you saw this through channels that you participate in – analyst briefings with Gartner, IDC, Meta and Forrester, a lot of proactive press with both the technical press and the business press. A PSS alert was sent out to all of our platform support customers. We used our Microsoft Valued Professionals or MVP’s in online communities, our support team, our Microsoft Consulting Services staff, essentially everyone at Microsoft who has any interaction with IT professionals. We made updates to all Microsoft properties; Microsoft.com, MSN, etc., to make sure people were aware of this. We also issued a joint advisory with the Department of Homeland Security to make sure that customers – even ones that don’t necessarily interact with Microsoft directly – had information about the situation. A lot of work was done on the Consumer Web Outreach to make sure that on the homepage of both Microsoft.com, MSN and other properties, that we had the right tools to help customers to be successful. A lot of work was done with security researchers, in particular Russ Cooper, Mark McFray and other security research firms. We spent time with them. I want to thank them for the great work they’ve done to reach out to customers to get people the best information about both the vulnerability and the exploits. One of the things that we got feedback from many of you coming out of the Slammer situation in January, was there was a need for us to have a much more efficient clearinghouse of information between Microsoft and Security Partners. So we’ve created something that we announced in April called the Virus Information Alliance, or VIA, and the key thing we did with the VIA partners was to contact them to let them know about the bulletin and also share with them information we had about the vulnerability, things including network traces, etc., so they could actually change their security platforms and train thoseplatforms to help you protect yourselves from the potential of exploits that we envisioned back on July 16. Quite a bit of outreach down here with … with Internet Security Systems and Foundstone, all who did a great job of helping to get the word out, but also to provide even more in depth information about how to be secure. On July 25 an important milestone was reached when Exploit Code was first publicly released, really initiating the third phase of this process for us. X-Focus, which is a full disclosure security research group, published an exploit tool that would allow an attacker to run any software of their choice, any software of the attacker’s choosing, on an unpatched system. Another full disclosure researcher published an updated version of the exploit tool and said that their real goal here was to make the tool more reliable. Very clearly there was a lot of community feedback that indicated the real possibility of this exploit code being used as the basis for a potential worm. As a result, we took the obvious step, which is part of our process, to heighten our efforts to get information out to customers. We repeated all the proactive communication around MS03-026, while at the same time upgrading our level of concern given there was now exploit code that was available publicly. We proactively did updated conversations with both analysts and press to make sure they understood the importance of this situation and worked with the Department of Homeland Security to release an updated advisory. At the same time, the Computer Emergency Response Team, or CERT, issued an advisory on July 31, very much in response to the existence of this exploit code. We worked with our security partners and vendors, and again, members of the Virus Information Alliance, to insure that not only had they gotten the guidance from us on how to make their platforms have the capabilities to detect and protect against these attacks, but to actually make sure that that protection was available. We also sent to all of our enterprise and smaller customers to make sure that they had information that they needed. All of our account managers were instructed to personally make contact with their customers to convey the increased level of risk, but also to explain that that level of risk had be increased because of the existence of exploit code. Of course, the Web sites were updated to make sure that they had a more forceful, more urgent advisory around the importance of installing MS Last week on August 11, very clearly we began the final phase. The Blaster Worm was first seen in the wild, and the Blaster Worm really attempts to do two things. The first, it replicates itself using the vulnerability that was corrected in MS At the same time, it was also designed to, at a specific time, to get attacking windowsupdate.com and in particular, to make sure that it really was focusing on trying to take down the site that we use to protect customers. I really want to emphasize, first of all, that there was never a successful attack against windowsupdate.com because we had the opportunity to take the steps to protect the system. The result of the Blaster Worm is, in some cases, systems, when they’re infected, crashing during the attempt to infect the system. Also in the process of the infection, it can generate a lot of network traffic, in particular on customer networks, but also on Internet backbones as a whole. We also began to notice variants of the Blaster Worm within a few days. Again, the Microsoft Emergency Response Team continues to work 24x7 to, first of all, investigate the worm and its variants, to make sure we understood them, had protection, but also worked very closely with our partners to make sure that they had the information they need to make sure their tools were effective at protecting our customers. To provide even more security guidance for customers, to make sure you had the information you needed to protect yourselves, we used the Virus Information Alliance again to make sure that they could help not only build tools, but also to provide guidance, help and support to make sure customers were safe. Blaster shows the complex interplay between security researchers, software companies, and hackers © 2002 Microsoft Corporation. All rights reserved. This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.
12 Decreasing Time To Patch (Sasser)April 13 April 24-29 April 30 Bulletin & patch available No exploit Exploit code in public Worm in the wild Bulletin MS delivered to customers (7/16/03) Continued outreach to analysts, press, community, partners, government agencies Exploit Reverse shell code posted to various web sites Worm Sasser worm discovered. Multiple variants hit simultaneously Here is a recap of the events around Microsoft Security Bulletin, MS03-026, commonly known as “O-twenty six” Note: Points made are business centric. RPC / DCOM explanation if needed is below: RPC: Remote Procedure Calls Used to communicate API calls between machines or across processes within a machine When intermachine, runs over port 135 DCOM: Distributed COM Used as the activation mechanism for RPC servers Runs on NT4, Win2K, WinXP, WS03 These technologies are broadly used in the OS, hence the degree of vulnerability due to the bug MS03-026: a new class of loop-based buffer overflow not caught by prior tools Mike’s transcript on the events: The four phases that we’re looking at are the time between when we were first notified of the vulnerability by the Security Research Group, Last Stages of Delirium. That was on July 1st. That period ended July 16 when the patch was first available. The second period of time is the time immediately after releasing the patch before the time that there was a public release of the exploit code. That’s the period of time between July 16 and July 25 of last month. The third period of time is the time between the public release of the exploit code on July 25 and the first discovery of the Blaster Worm on August 11. Lastly, the fourth period of time is work we’ve done since the first time that the Blaster Worm was discovered in the wild. As I said, let me start by talking about the history prior to MS It really did begin for us in the initial phase when we were informed by the Security Research Group, Last Stages of Delirium, on the first of July. They reported to us a vulnerability in the RPCD comp component of Windows. RPC is a core communications protocol used by many Microsoft platforms and by many third party products. RPC stands for Remote Procedure Call. The MS vulnerability is exposed through a mechanism called remote activation, where DCOM is accessed across the RPC protocol. The vulnerability affects Windows NT 4.0, Windows 2000, Windows XP and Windows Server As I mentioned, it was reported to us by a group called the Last Stages of Delirium, also known as LSD. The key thing we did at that time was to immediately activate our highest level of emergency response, one by the Microsoft Security Response Center. We literally had a team working around the clock from the moment we had the report from the security researchers to both build, test and release a patch, and we did that within two weeks because we understood the importance of both having a patch available quickly, but also we wanted to balance the need to have a very high quality patch in a timely period. During the process, though, I also want to make you aware of the fact we did work very closely with the security researcher to follow through on what we call Responsible Disclosure Guideline, and really the goal here is to make sure that they share information with us in a confidential way and then work with us to both develop and test the patch, and together, we go off and share information about the vulnerability and the patch. The real goal here is to make sure that while we get the fix out there and help them to get credit for their great work, we minimize the risk we put customers at. I want to thank the Last Stages of Delirium team for their responsible behavior through this process. Stage two really began for us on July 16 when we delivered the security bulletin, MS03-026, to customers. It included patches for all affected supported Windows versions. That includes Windows NT Server 4.0 Service Pack 6A. Windows NT 4.0 Workstation is not a generally supported platform anymore, but we subsequently added support for that platform because of the fact that so many customers were in that situation. The patch included support for Windows 2000, Service Pack 3 and Service Pack 4, and again, we later announced support of the patch for Windows 2000 Service Pack 2, again, based on customer feedback. The patch is effective for Windows XP, both in the gold or no service pack version, along with Windows XP Service Pack 1, and the patch also was effective against Windows Server I do want to note that Windows ME is not affected and Win 9X versions, which are not supported, are also not affected. On July 17, we updated the bulletin as we learned more information about the potential exploits. I want to be clear that that update to the bulletin included more details about mitigation, new steps that we thought were important for customers to follow to protect themselves in addition to the ones that were in the original bulletin. Of course, the patch itself was the most effective way of protecting the customer, but I want to be clear that the re-release of the bulletin on July 17 was simply more guidance. The patch was unchanged, and the initial patch that we shipped on July 16 was effective even in light of the information and continues to be effective against Blaster and its variants. At the time, we also notified customers through our Security Notification Service. We have customers signed up for that service. We sent from Microsoft account teams to their contact at the customer. We also had our support teams and technical account managers in particular working directly with technical staff and security staff within our customer organizations to make sure things were well understood. We also spent a lot of time using the press to do outreach both to enterprise customers and to consumers, to make sure there was a high level of awareness of both the existence of the patch and vulnerability, but also the importance of this patch in particular getting high priority to protect customers of all sizes. Some of the examples of that outreach – and I’m sure a lot of you saw this through channels that you participate in – analyst briefings with Gartner, IDC, Meta and Forrester, a lot of proactive press with both the technical press and the business press. A PSS alert was sent out to all of our platform support customers. We used our Microsoft Valued Professionals or MVP’s in online communities, our support team, our Microsoft Consulting Services staff, essentially everyone at Microsoft who has any interaction with IT professionals. We made updates to all Microsoft properties; Microsoft.com, MSN, etc., to make sure people were aware of this. We also issued a joint advisory with the Department of Homeland Security to make sure that customers – even ones that don’t necessarily interact with Microsoft directly – had information about the situation. A lot of work was done on the Consumer Web Outreach to make sure that on the homepage of both Microsoft.com, MSN and other properties, that we had the right tools to help customers to be successful. A lot of work was done with security researchers, in particular Russ Cooper, Mark McFray and other security research firms. We spent time with them. I want to thank them for the great work they’ve done to reach out to customers to get people the best information about both the vulnerability and the exploits. One of the things that we got feedback from many of you coming out of the Slammer situation in January, was there was a need for us to have a much more efficient clearinghouse of information between Microsoft and Security Partners. So we’ve created something that we announced in April called the Virus Information Alliance, or VIA, and the key thing we did with the VIA partners was to contact them to let them know about the bulletin and also share with them information we had about the vulnerability, things including network traces, etc., so they could actually change their security platforms and train thoseplatforms to help you protect yourselves from the potential of exploits that we envisioned back on July 16. Quite a bit of outreach down here with … with Internet Security Systems and Foundstone, all who did a great job of helping to get the word out, but also to provide even more in depth information about how to be secure. On July 25 an important milestone was reached when Exploit Code was first publicly released, really initiating the third phase of this process for us. X-Focus, which is a full disclosure security research group, published an exploit tool that would allow an attacker to run any software of their choice, any software of the attacker’s choosing, on an unpatched system. Another full disclosure researcher published an updated version of the exploit tool and said that their real goal here was to make the tool more reliable. Very clearly there was a lot of community feedback that indicated the real possibility of this exploit code being used as the basis for a potential worm. As a result, we took the obvious step, which is part of our process, to heighten our efforts to get information out to customers. We repeated all the proactive communication around MS03-026, while at the same time upgrading our level of concern given there was now exploit code that was available publicly. We proactively did updated conversations with both analysts and press to make sure they understood the importance of this situation and worked with the Department of Homeland Security to release an updated advisory. At the same time, the Computer Emergency Response Team, or CERT, issued an advisory on July 31, very much in response to the existence of this exploit code. We worked with our security partners and vendors, and again, members of the Virus Information Alliance, to insure that not only had they gotten the guidance from us on how to make their platforms have the capabilities to detect and protect against these attacks, but to actually make sure that that protection was available. We also sent to all of our enterprise and smaller customers to make sure that they had information that they needed. All of our account managers were instructed to personally make contact with their customers to convey the increased level of risk, but also to explain that that level of risk had be increased because of the existence of exploit code. Of course, the Web sites were updated to make sure that they had a more forceful, more urgent advisory around the importance of installing MS Last week on August 11, very clearly we began the final phase. The Blaster Worm was first seen in the wild, and the Blaster Worm really attempts to do two things. The first, it replicates itself using the vulnerability that was corrected in MS At the same time, it was also designed to, at a specific time, to get attacking windowsupdate.com and in particular, to make sure that it really was focusing on trying to take down the site that we use to protect customers. I really want to emphasize, first of all, that there was never a successful attack against windowsupdate.com because we had the opportunity to take the steps to protect the system. The result of the Blaster Worm is, in some cases, systems, when they’re infected, crashing during the attempt to infect the system. Also in the process of the infection, it can generate a lot of network traffic, in particular on customer networks, but also on Internet backbones as a whole. We also began to notice variants of the Blaster Worm within a few days. Again, the Microsoft Emergency Response Team continues to work 24x7 to, first of all, investigate the worm and its variants, to make sure we understood them, had protection, but also worked very closely with our partners to make sure that they had the information they need to make sure their tools were effective at protecting our customers. To provide even more security guidance for customers, to make sure you had the information you needed to protect yourselves, we used the Virus Information Alliance again to make sure that they could help not only build tools, but also to provide guidance, help and support to make sure customers were safe. Sasser shows the continually shrinking window between the time a patch is released, exploit code is generally available and a worm is written to exploit it. © 2002 Microsoft Corporation. All rights reserved. This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.
13 Solution Components Now let’s discuss the guidance, online services, and tools Microsoft provides today to enable patch management in your environment. © 2002 Microsoft Corporation. All rights reserved. This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.
14 Solution Components Prescriptive Guidance Analysis ToolsMicrosoft Guide to Security Patch Management Patch Management Using SUS Patch Management Using SMS Analysis Tools Microsoft Baseline Security Analyzer (MBSA) Office Inventory Tool* Online Update Services Windows Update Office Update Content Repositories Windows Update Catalog Office Download Catalog Microsoft Download Center Management Tools Automatic Updates (AU) feature in Windows Software Update Services (SUS) Systems Management Server (SMS) This table categorizes the various components. The prescriptive guidance provides process & best practice recommendations on patch management using the tools described here. The analysis tools enable detection of missing security patches, the online update services allow automated detection and installation of missing patches, the content repositories allow the ability to manually or selectively download the relevant updates, the management tools enable patch management within the enterprise. Details on each of these components are in the appendix section. I will only discuss the key components – MBSA, Windows Update, Automatic Updates, SUS, SMS, and the prescriptive guidance. Now, let’s take a look at these components. *Office Inventory Tool is no longer needed – MBSA 1.2 (released in January 2004) includes Office scanning functionality © 2002 Microsoft Corporation. All rights reserved. This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.
15 Update Management GuidanceImplementing a consistent, high quality update management process is the key to successful update management Microsoft delivers best practices prescriptive guidance for effective update management Uses Microsoft Operations Framework (MOF) Based on ITIL* (defacto standard for IT best practices) Details requirements for effective update management: Technical & operational pre-requisites Operational processes & how technology supports them Daily, weekly, monthly & as-needed tasks to be performed Testing options Three update management guidance offerings Microsoft Guide to Security Patch Management** Patch Management using Software Update Services*** Patch Management using Systems Management Server*** Assess Identify Evaluate & Plan Deploy One of the goals in the area of update management is to provide best practice process and solution usage guidance along with the delivery of Microsoft’s update management technologies. This guidance is based on the Microsoft Operations Framework which is in turn, based on the best practices codified in the IT Information Library (ITIL). The guidance covers technical and operational pre-requisites, operational processes, best practices for apply the technology in various real-world scenarios, and testing. Three update management guidance offerings are currently available. The first provides guidance on the update management process specifically in the context of security patch management. The other two apply the update management process guidance to SUS and SMS respectively. The titles of these offering are hyper-linked to the locations on Microsoft.com from where the guide can be downloaded. *Information Technology Infrastructure Library **Emphasizes security patching & overall security management ***Comprehensive coverage of patch management using the specified technology © 2002 Microsoft Corporation. All rights reserved. This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.
16 MBSA Helps identify vulnerable Windows systemsScans for missing security patches and common security mis-configurations Scans various versions of Windows and other Microsoft applications Scans local or multiple remote systems via GUI or command line invocation Generates XML scan reports on each scanned system Runs on Windows Server 2003, Windows 2000 and Windows XP Integrates with SUS & SMS New Update Assess MBSA is an assessment tool that serves two functions: it identifies missing security patches for a variety of Microsoft products and it also identifies common security mis-configurations for Microsoft products. It’s a standalone tool that can be invoked from the command line, via a GUI interface or programmatically by another product (e.g. SMS uses it to identify missing security patches). It scans the local machine or multiple remote machines and generates an XML report of missing security patches and security mis-configurations for each system. MBSA works for Windows 2000 and newer versions of the operating system. Note the ovals on the bottom left of the slide. These represent the four steps of the patch management process as discussed earlier and a highlighted oval indicates that the component we’re discussing provides functionality for that step in the process. The ‘New Update’ oval has been added to the list to indicate whether the component enables delivery of new or existing updates. Identify Evaluate & Plan Deploy © 2002 Microsoft Corporation. All rights reserved. This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.
17 Microsoft Download CenterMBSA: How It Works* MSSecure.xml contains Security Bulletin names Product specific updates Version and checksum info Registry keys changed KB article numbers Etc. Run MBSA on Admin system, specify targets Microsoft Download Center Downloads CAB file with MSSecure.xml & verifies digital signature MSSecure.xml Scans target systems for OS, OS components, & applications Parses MSSecure to see if updates available Checks if required updates are missing Now let’s do a walk through of how MBSA works. MBSA Computer Generates time stamped report of missing updates SUS Server *Only covers security patch scanning capabilities, not security configuration detection issues © 2002 Microsoft Corporation. All rights reserved. This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.
18 Windows Update (WU) Microsoft online update service (windowsupdate.microsoft.com): Identifies missing Windows OS* patches / updates on accessing computer Generates targeted list of missing updates Installs user selected missing updates Provides update installation history WU content can be automatically downloaded via Automatic Updates Supplemented by Windows Update Catalog site which provides: Comprehensive repository for all Windows and ‘Designed for Windows’ logo device driver updates Search – to find desired update Manual download of desired updates Download history for accessing computer New Update Assess Now let’s take a look at the online services Microsoft offers to allow consumers and enterprise users to automatically detects and install patches and updates. The first of these is Windows Update. Information on the other online service (Office Update) is in the appendix. As you can see from the highlighted ovals, Windows Update enables identification of missing updates, delivery of the missing updates, and installation (deployment) of the missing updates. Users access the service by selecting Windows Update from the Start Button Menu or the Help and Support Center Menu or by directly typing the Windows Update URL in the browser input area. Windows update provides the use with a list of missing updates, allows the user to select the updates for installation, and automatically installs the selected updates. The Automatic Update feature in Windows can be used to automatically download and install new updates available on Windows Update. The Windows Update Catalog supplements the Windows Update service and allows manual access and download of updates available via Windows Update. Identify Evaluate & Plan Deploy *Windows 98 and later versions. Note: also updates 64-bit editions of Windows Server © 2002 Microsoft Corporation. All rights reserved. This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.
19 Windows Update: How It Works Scenario 1: User Initiated Access Scenario 2: Access via Automatic Updates (AU) User points browser to WU site & selects ‘Scan for updates’ or AU automatically checks for new updates (every hours) Windows Update Client side code (CC) in browser (or AU) validates WU server & gets download catalog metadata CC (or AU) uses metadata to identify missing updates WU (or AU -- if so configured) lists missing updates and user selects updates to download Let’s take a look at how the two scenario for accessing Windows Update work. Before we do that, I want to emphasize that Windows Update does not collect any personally identifiable information from the systems it services. For additional details on exactly what information is collected, please see the privacy policy available at the URL on the bottom of this slide. User types URL (windowsupdate.microsoft.com) or selects WU option in Start Menu or ‘Help & Support’ GUI. Client-side code (CC) in browser detects language, OS, & browser information. User selects ‘Scan for updates’ option on Windows Update page. Alternatively, AU automatically checks for new updates every hours CC (or AU) checks digitally signed WU site server identification information and searches WU server to get metadata (securely over SSL) for list of applicable updates. Note: Metadata exchange over SSL also includes hash for validating patch integrity CC (or AU) uses metadata info to determine how to find out if each update is installed on the client machine CC displays list of missing updates & allows user to select updates for installation and downloads user-selected updates, or AU notifies user that new updates are available and allows user to select updates to download and install CC (or AU) verifies integrity of downloaded updates & installs them CC (or AU) updates the installation history file the client machine Download & installation statistics sent to WU site server. Note: No personally identifiable information is collected. For details, please see CC (or AU) downloads, validates, & installs updates. AU downloads using BITS, and can be configured to allow user to select updates to install CC (or AU) updates history & statistics information* *Note: No personally identifiable information is collected See © 2002 Microsoft Corporation. All rights reserved. This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.
20 SUS 1.0 Deploys Windows security patches, security rollups, critical updates, and service packs only Deploys above content for Windows 2000, Windows Server 2003 and Windows XP only Provides patch download, deployment, and installation configuration options Bandwidth optimized content deployment Provides central administrative control over which patches can be installed from Windows Update Provides basic patch installation status logging New Update Assess Now let’s discuss the corporate update management products: SUS and SMS. Let’s start with SUS. The currently available version of SUS is SUS 1.0 with Service Pack 1 Like Windows Update, SUS provides functionality in the areas of patch & update delivery, identification of missing patches, and deployment / installation of patches. It is more limited than Windows Update in terms of the operating system versions it supports – it does not support NT 4 or Windows 98 – and in the content it delivers – it does not deliver non-critical update and drivers. However, SUS is a behind-the-firewall patch management product that IT organizations can use to automatically download new patches and updates, and perform the evaluation & planning steps of the process before approving patches & updates for deployment in their environment. SUS is based on a ‘pull’ mechanism where the SUS server maintains the list of approved updates and the SUS client, which is the Automated Updates feature in Windows, periodically checks the SUS server for newly approved updates and downloads and installs them on the individual systems. In conjunction with the client component and Windows Group Policy capabilities, it can be used to enable administrator control over the application of patches in the environment by preventing users from going directly to Windows Update and installing non-approved updates on their systems. SUS also provide basic logging, bandwidth optimization, and administrative control options. Identify Evaluate & Plan Deploy © 2002 Microsoft Corporation. All rights reserved. This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.
21 Windows Update Service Windows Update ServiceSUS 1.0: How It Works Windows Update Service Windows Update Service Firewall SUS Server check for updates every 24 hours* Bandwidth Throttling Administrator reviews, evaluates, and approves updates Bandwidth Throttling Approvals & updates synced with child SUS servers** Child SUS Server Parent SUS Server AU (the SUS client) gets approved updates list from SUS server Now, let’s take a look at how SUS works. Parent SUS server downloads metadata about new updates as well as the updates themselves from Windows Update site Administrator reviews new updates and approves the appropriate ones after completing any required testing. SUS maintains download and approval logs on statistics (IIS) server Approved updates information including information for determining if the update is installed is distributed to child SUS servers AU contacts it’s SUS server to determine if there are any newly approved updates. If so, it gets the metadata information for the update and checks to see if it is installed on the system If not, depending on AU configuration, AU either automatically downloads the missing updates or notifies the user that there are missing updates and requests the user’s permission to download them. For target systems with AU configured to pull updates from a SUS server, AU downloads approved updates from specified SUS server. For target systems with AU configured to pull updates from Windows Update, AU downloads the appropriate updates from Windows Update. AU checks digital signatures on downloaded updates to verify authenticity & integrity Depending on AU configuration, it either automatically installs the updates or notifies & allows the user to choose which updates to install and when to install them. In latter case user reviews & selects updates to be installed AU logs the success / failure history for update installs on the target machines Bandwidth Throttling AU downloads approved updates from SUS server or Windows Update Child SUS Server AU either notifies user or auto-installs updates AU records install history *Configurable 1/day or 1/week **SUS maintains approval logs & download, sync, & install statistics © 2002 Microsoft Corporation. All rights reserved. This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.
22 SUS Client Component: Automatic UpdatesCentrally configurable to get updates either from corporate SUS server or Windows Update service Can auto-download and install patches under admin control Consolidates multiple reboots to a single reboot when installing multiple patches Included in Windows 2000 SP3, Windows XP SP1, and Windows Server 2003 Localized in 24 languages As noted earlier, the client component of SUS is the Automated Updates feature in Windows. The appropriate version of Automated Updates is included in Windows SP3, Windows XP SP1, and Windows Server 2003 and may be downloaded from the Microsoft Download Center for installation on systems that have the pre-SP1 versions of Windows XP. Automated Updates is centrally configurable via Group Policy settings to talk to the appropriate SUS server and downloads and installs updates listed in the approved updates list on the SUS server it points to. It has the ability to consolidate multiple reboots into a single reboot when installing multiple patches, and is localized in 24 languages. © 2002 Microsoft Corporation. All rights reserved. This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.
23 SUS Server Component: SUS ServerDownloads updates from Windows Update Web based administration GUI Specify server & update process configuration options View downloaded updates Approve updates & view approved updates Security by design and default Requires NTFS; Installs IIS Lockdown and URL scanner* Supports secure administration over SSL Digital signatures on downloaded content validate authenticity Uses HTTP for content synchronization – only port 80 needs to be open Server side XML based logging on Web server Patch deployment & installation statistics Supports geographically distributed or scale-out deployments with centralized management for content synchronization & approvals Localized** in English & Japanese The SUS server allows automated synchronization of new updates and provides an administrative GUI to allow control of various setup, download, and update approval options. All content downloaded by SUS as well as the other solution components covered in this presentation verify the source and content of the downloaded updates by checking their digital signatures. All SUS communications take place over HTTP so only port 80 needs to be open and administrations can be done securely via SSL. As indicated earlier, while SUS does not provide reporting capabilities, it logs update download and installation status in XML format to a Web server and the logs can be manually reviewed or consumed by a reporting utility to enable easier perusal of the installation results. A utility that provides this capability is available at Microsoft has not evaluated this utility and does not endorse, support, or recommend its use. However, we are making customers aware of it so they can evaluate it for themselves if they so desire. *If not already installed **Note: Delivers updates for all 24 supported client languages © 2002 Microsoft Corporation. All rights reserved. This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.
24 SMS 2003 Identifies & deploys missing Windows and Office security patches on target systems Can deploy any patch, update, or application in Windows environments Inventory management & inventory based targeting of software installs Install verification and detailed reporting Flexible scheduling of content sync & installs Central, full administrative control over installs Bandwidth optimized content distribution Software metering and remote control capabilities New Update Assess Now let’s discuss SMS. SMS is a for-fee (license fees apply) product from Microsoft. It is a full software distribution solution for Windows environments that also includes patch management capabilities with SMS 2.0 version with the Software Update Services Feature Pack and with the SMS 2003 version where patch management functionality has been merged into the core release. In this presentation we will focus on the patch management capabilities provided with SMS The appendix section details the enhancements to the patch management capabilities over those provided with SMS 2.0 with Software Update Service Feature Pack. Another point to note is that though the patch management feature pack for SMS 2.0 is call the SUS Feature Pack, it unrelated to the SUS product. As you can see from the highlighted ovals, SMS delivers functionality in all the areas of patch management. While it is optimized for deployment of Windows and Office security patches, it can deploy any patch, update, or application because it’s a full software distribution solution and includes the ability to inventory the hardware and software assets, provides a high level of administrative control over software distribution and patch management, and provides detailed reporting capabilities. Identify Evaluate & Plan Deploy © 2002 Microsoft Corporation. All rights reserved. This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.
25 SMS 2003 Patch Management: How It WorksMicrosoft Download Center Setup: Download Security Update Inventory and Office Inventory Tools; run inventory tool installer Firewall Scan components replicate to SMS clients Clients scanned; scan results merged into SMS hardware inventory data SMS Distribution Point SMS Site Server Administrator uses Distribute Software Updates Wizard to authorize updates SMS Clients Let’s take a look at how SMS 2003 works. It should be noted that in contrast to SUS which uses a ‘pull’ model, SMS uses a ‘push’ model to distribute patches to the target systems and then relies on the SMS clients on these systems to complete the installation. The SMS administrator, in a one-time event, downloads the Security Update Inventory Tool and/or the Microsoft Office Inventory Tool from the Download Center Web site. The administrator, in a one-time event, runs the inventory tool installer program on the SMS site server, which creates the necessary packages, collections, and advertisements for distributing the software update scan tools to the clients. Simultaneously, the installer program creates the program for the synchronization component on the synchronization host. The software update scan component packages replicate to distribution points in the SMS site, and from there to the target client computers. The scan component analyzes the installed and applicable software updates on the client computer. The information is converted to SMS hardware inventory data, and it propagates up the hierarchy along with the rest of the hardware inventory data. Note that the time it takes for the information to reach the site server depends on the scan component configuration, hardware inventory agent schedule settings, and site server load. The SMS administrator runs the Distribute Software Updates Wizard to view, evaluate, and authorize applicable software updates from the software update inventory data. The Distribute Software Updates Wizard downloads the source files for the specified software update from the Microsoft Download Center Web site, stores the source file in the specified package source shared folder, and creates or updates the necessary packages, programs, and advertisements for distributing the software updates to SMS clients. To every package the Wizard creates or updates, it appends an SMS program that contains commands to run the Software Updates Installation Agent. The software update packages replicate to distribution points in the site, and the programs are advertised to the clients. The Software Update Installation Agent runs on the clients and deploys the software updates. The agent runs the scan component to ensure that it installs only the needed software updates. Periodically (weekly by default,) the synchronization component checks the Microsoft Download Center Web site for updates to the scan component and software updates catalog. The synchronization component downloads these new updates and updates the packages, programs, and advertisements associated with the scan component. The updated scan component package and advertisement are distributed to the destination SMS client computers. Update files downloaded; packages, programs & advertisements created/updated; packages replicated & programs advertised to SMS clients SMS Distribution Point Software Update Installation Agent on clients deploy updates SMS Clients Periodically: Sync component checks for new updates; scans clients; and deploys necessary updates SMS Clients © 2002 Microsoft Corporation. All rights reserved. This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.
26 SMS 2003 Patch Management: FunctionalitySystem scanning & patch content download Content from Microsoft Download Center MBSA & Office Inventory plug-ins scan for missing patches Supports updating of remote & mobile devices Updates various versions of Windows, Office, SQL, Exchange, and Windows Media Player without need for update packaging / scripting Administrator control Update targeting based on AD, non-AD groups, WMI properties; additional options via scripting Patches content is downloaded from a central SMS repository only when the deployment process is initiated by the SMS administrator Specific start and end times (change windows); multiple change windows Easily move patches from testing into production Reference system patch configurations can be used as a template to verify or enforce compliance of systems that must mimic reference system configuration These two slides provide additional detail on the patch management capabilities of SMS 2003. In the area of administrative control, SMS can target deployment of updates based on Active Directory groups, non-Active Directory groups (enumerated lists of systems), or Windows Management Instrumentation information for the target systems. WMI provides detailed information on the hardware and software components of Windows systems and these can be used to define targeting criteria. Additionally, SMS allows administrators to specify multiple time windows for patch installation, e.g., between 5 and 7 am, between noon and 1 pm, etc. Deployment of a patch to production systems after it has been evaluated in a test environment is as easy as changing the targeting of the patch. SMS also makes it easy to use a reference systems configuration to evaluate the compliance and when necessary, enforce compliance with the reference systems, for all the systems in the specified target group. © 2002 Microsoft Corporation. All rights reserved. This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.
27 SMS 2003 Patch Management: Functionality (2)Patch download & installation Delta replication (site-site, server-server) of patches Uses BITS* for mobile / remote client-server Uses SMB* for LAN / priority situations Reminders and rescheduling of install / reboot & enforcement dates Optimized graceful reboots, but forced when enforcement date arrives Per-patch reboot-needed detection to reduce reboots Status & Compliance Reporting Deployment status as patches are attempted Standard and customized reports through read-only SQL queries Determine actual baselines in the environment before changing the environment SLA measurement and rate-of-spread SMS also provides these capabilities in the areas of download, installation, bandwidth optimization, and reporting. Administrators can specify that deployment status be reported back to the central SMS database up in real time so that they can evaluate the rate of installation and be notified in a timely manner of any issues with the deployment. In addition to the set of standard reports provided in SMS, administrators can created customized reports via SQL queries *Requires SMS Advanced Client © 2002 Microsoft Corporation. All rights reserved. This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.
28 Adopt the solution that best meets the needs of your organizationChoosing A Patch Management Solution Needs-Based Selection Adopt the solution that best meets the needs of your organization Capability Windows Update SUS 1.0 SMS 2003 Supported Platforms for Content NT 4.0, Win2K, WS2003, WinXP, WinME, Win98 Win2K, WS2003, WinXP NT 4.0, Win2K, WS2003, WinXP, Win98* Supported Content Types All patches, updates (including drivers), & service packs (SPs) for the above Only security & security rollup patches, critical updates, & SPs for the above All patches, SPs & updates for the above; supports patch, update, & app installs for MS & other apps Granularity of Control Targeting Content to Systems No Yes Network Bandwidth Optimization Yes (for patch deployment) Yes (for patch deployment & server sync) Patch Distribution Control Basic Advanced Patch Installation & Scheduling Flexibility Manual, end user controlled Admin (auto) or user (manual) controlled Administrator control with granular scheduling capabilities Patch Installation Status Reporting Assessing computer history only Limited (client install history & server based install logs) Comprehensive (install status, result, and compliance details) Additional Software Distribution Capabilities Deployment Planning N/A Inventory Management Compliance Checking Core Patch Management Capabilities Microsoft recommends that you use the technology that best meets your needs, i.e., compare your needs to the capabilities and limitations of each technology. This table provides a high level comparison for some for the key criteria that differentiate the update management technologies from Microsoft. In addition to supported operating systems and the types of content delivered, the key criteria relate to the level of control – in terms of targeting of content, bandwidth optimization, patch distribution, patch installation and scheduling flexibility, and reporting. Organization looking for a single solution for software deployment and patch management will also want to evaluate the solution in light of it’s software distribution capabilities. This table shows how Windows Update, SUS 1.0 and SMS 2003 stack up along these criteria. Microsoft recommends that customers adopt the solution that best meets the needs of their organizations. *MBSA does not support scanning Win98 – Win98 can be updated using SMS2003 inventory management and software distribution capabilities © 2002 Microsoft Corporation. All rights reserved. This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.
29 Choosing A Patch Management Solution Typical Customer DecisionsCustomer Type Scenario Customer Chooses Large or Medium Enterprise Want single flexible patch management solution with extended level of control to patch & update (+ distribute) all software SMS Want patch management solution with basic level of control that updates Windows 2000 and newer versions* of Windows** SUS Small Business Have at least 1 Windows server and 1 IT administrator** All other scenarios Windows Update Consumer All scenarios This slide show how Microsoft believes customers will choose between the solutions based on the size of the organization they represent. We believe that large and medium enterprise customers will most likely choose between SMS and SUS. Those that want a full software distribution that includes patch management functionality will likely choose SMS while those looking for simple but somewhat limited patch management solution will choose SUS. For small business customers, the logical options are SUS and Windows Update. Customers in this size segment who have at least one Windows Server and one skilled administrator will choose SUS while the rest will use Windows Update. For consumers, the only logical choice is Windows Update. *Windows 2000, Windows XP, Windows Server 2003 **Customer uses Windows Update or manual process for other OS versions & applications software © 2002 Microsoft Corporation. All rights reserved. This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.
30 What could be better than patching?Not having to patch Introducing Slipstreaming!
31 Slipstreaming “Slipstreaming” – Integrating a patch into a product installation directory Windows, Internet Explorer, and Office support “Slipstreaming” It’s so simple! An example . . . Copy Windows 2000 CD to network share “Slipstream” Service Pack 4 into the share “Slipstream” all post-SP4 critical security updates into the share Perform network / RIS installation of Windows 2000 from that share Fully patched after setup completes!
32 Slipstreaming For instructions on “slipstreaming” service packs – consult the deployment guide for the service pack you are deploying For instructions on “slipstreaming” hotfixes and udpates – consult the hotfix deployment guide
33 Finding critical security updates to slipstreamSubscribe to the Security Alert Notification Service We’ll tell you when critical updates are available! Visit the Security Bulletin Search site to view security bulletins for all products Under Product/Technology choose the product you are interested in finding updates for Under Service Pack choose the SP level you are using Check “Show only bulletins that have not been superseded” and press ‘Go’
34 Roadmap We’ve talked about how Microsoft is addressing the patch management situation, the progress made to date in this area, the currently available offerings, and how to choose between the various offerings. Now let’s discuss the roadmap for the various areas of patch management. © 2002 Microsoft Corporation. All rights reserved. This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.
35 Informed & Prepared CustomersNew Security & Patch Management workshops Regular web casts on security patch management* Updated roadmap, whitepapers, and guidance Q1 ‘03 Q2 ‘03 Q3 ‘03 Q4 ‘03 Q1 ‘04 Q2 ‘04 H2 ‘04 H1 ‘05 Updated Patch Management Guidance for SMS 2003 SP1 Patch Management Guides Bulletin Search Page Improved KB Articles Patch Management Workshops Security Bulletin Teleconferences GTM Partnership Deliverables Revised Patch Management Guides Informed and Prepared Customers Security Readiness Kit (Guides, Tools, Best Practices) Clearer Severity Rating Levels Patch Management Guides In the area of customer communications, guidance, and training, you can expect to see a set of new security and patch management workshops, training sessions, and web casts that will be delivered on an ongoing basis. We will also continue to provide additional guidance on patch management process and share additional details on the testing process for patches. Two important deliverables in this are the updated patch management guidance for SMS2003 SP1 and the new patch management guidance for the next version of Windows Update Services, the next version of SUS. Patch Management White Paper Sustaining Engineering Practices White Paper Patch Management Roadmap Patch Management Guidance for Windows Update Services Security Guidance Kit *See for upcoming web casts © 2002 Microsoft Corporation. All rights reserved. This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.
36 Consistent & Superior Update ExperienceQ1 ‘03 Q2 ‘03 Q3 ‘03 Q4 ‘03 Q1 ‘04 Q2 ‘04 Q3 ‘04 Q4 ‘04 Add/Remove Program improvements in XP SP2 Standard Detection Manifest Standard installer switches defined Standard terminology for documentation defined Naming & signing standard defined 2 Installers: MSI, Update.exe MSI 3.0 Consistent & Superior Update Experience Patches & Security Bulletins released once a month Standard Titles* defined Standard Registry Entries defined Product teams compliant with SE Baseline standards These are the milestones for the roadmap for improving the patching experience. In addition to the reduction in frequency of patch releases from once a week to once per month, except in the rare case where an patch needs to be released on an emergency basis because exploit code for the vulnerability addressed by the patch is in the public domain, the major milestones on this roadmap are the standardization of the installation of patches & updates; the standardization of the patch & update detection manifest; and the release of the next version of Windows Installer (MSI 3.0) which will enable rollback of patches as well as improved network bandwidth optimization and upto 90% reduction in patch sizes due to binary delta patching technologies; and the convergence to two installers – one for the OS and OS components (update.exe), and the other for applications (MSI) MSI 3.0 supports uninstall, binary delta patching, etc. Converge to two installers -- end of 2004 Consistency standards implemented in all new updates -- end of 2004 *For Add/Remove Programs, Windows Update, and Download Center © 2002 Microsoft Corporation. All rights reserved. This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.
37 Superior Patch QualityUp to 75% reduction in patch size* 10% reduction in patch reboots Patch test process extended to include customers Q1 ‘03 Q2 ‘03 Q3 ‘03 Q4 ‘03 Q1 ‘04 Q2 ‘04 H2 ‘04 H1 ‘05 Installer restarts services when possible 75% Reduction in Patch Size* 90% Reduction in Patch Size 25% Reduction in Patch Size Superior Patch Quality The key milestones on the patch quality roadmap are further reduction in patch sizes to 90% and extension of this 90% reduction to all types of patches delivered through Windows Update / Microsoft Update, SUS, and SMS. In addition to the 10% reduction in reboots for updates delivered via Windows Update and the ability for Windows Update, SUS, SMS, and the Hotfix installer to consolidate multiple reboots to a single reboot when installing multiple patches, further reductions of at least 30% in the number of operating system patches requiring reboots are expected to be delivered via new HotPatching (in-memory patching) technology to be delivered first in Windows Server 2003 Service Pack 1, with plans to extend this capability to additional OS versions beyond the Windows Server 2003 SP1 time frame. 10% Reduction in Patch Reboots Patch test process includes participating customers 30+% Reduction in Patch Reboots** *For Windows Update installs, more than 25% reduction for other patches **For Windows Server 2003 patches using HotPatching (in-memory patching) technology, delivered in SP1 © 2002 Microsoft Corporation. All rights reserved. This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.
38 MBSA Update Scanning FuturesOverall direction Microsoft will have a single scanning engine for detecting missing updates The scanning engine will be part of the Windows Update Services / Automatic Updates client MBSA and other product that need to detect or report on missing updates will request this information from the Windows Update Services / Automatic Updates client MBSA becomes Windows vulnerability assessment & mitigation engine Near-term plans MBSA 2.0 (H1 2005) Initial integration with Windows Update Services / Automatic Update client for update scanning Further deprecation of native MBSA scanning occurs on an ongoing basis as Microsoft Update continues to add support for updating additional Microsoft software over time Here’s the overall direction and short-term plans for MBSA. Essentially, the missing update scanning functionality currently in MBSA will be migrated to WUS / Automated Updates and MBSA will focus on providing enhanced functionality to detect security mis-configurations and when possible mitigate these security issues. Starting with the 2.0 version, MBSA will use WUS / AU to report on missing updates for applications supported by WUS/AU. © 2002 Microsoft Corporation. All rights reserved. This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.
39 WU and XPSP2 AU ImprovementsNew release of Windows Update (v 5) Improved homepage design and navigation Implements download throttling for dial-up and low bandwidth connections Will not recommend updates that have already been installed Download regulation feature reduces amount of data transmitted per update Improved ability to update systems with latest critical updates Customer offered choice during Windows XP SP2 install to have AU automatically download and install critical updates New version of Automatic Update client Uses BITS 2.0 to enable restart of interrupted download and improved bandwidth throttling Ability to delay reboot to next system shutdown
40 Microsoft Hosted Update Services Microsoft UpdateDownload Center Microsoft Update H1 2005 Today Office Update Windows Update Windows Update Microsoft Update Online service and update repository for updating all Microsoft software Microsoft Update: superset of Windows Update Initially supports Windows XP, Windows 2000, Windows Server 2003, Office XP, Office 2000, SQL Server 2000, MSDE 2000, and Exchange Support for additional Microsoft products will be added on an on-going basis Built on Windows Update Services (formerly SUS 2.0) infrastructure Includes automated scanning, update install, and reporting capabilities Windows Update maintained for legacy reasons WUS SMS Today we have two online update services – Windows Update and Office Update. In H Microsoft will deliver the Microsoft Update services which will support not just Windows and Office updates, but also updates for other Microsoft products. Support for additional Microsoft products will be phased in over time. Microsoft will continue to maintain an updated Windows Update service for legacy reasons © 2002 Microsoft Corporation. All rights reserved. This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.
41 Patch Management Products Future DirectionNear-term milestones Windows Update Services (H1 2005) SMS 2003 / WUS Phase 1 Integration (H1 2005) Leverages Windows Update Services for update scanning Longer-term (Longhorn time frame) Windows Update Services (WUS) becomes core update management component of Windows Server WUS updates all Microsoft corporate software SMS / WUS Phase 2 integration – SMS builds on WUS infrastructure to deliver advanced patch management WUS infrastructure can be used to build patch management solutions for 3rd party and in-house built software The strategy for the enterprise patch management products is to expand the functionality provided in future versions of SUS (i.e., WUS) to include core patch management and software distribution capabilities and integrate this into Windows. WUS will thus be the core infrastructure delivered in Windows that will be leveraged by future versions of SMS, as well as 3rd party patch management solutions to provide value added software distribution and patch management capabilities. This will eliminate the need for customers using Windows systems to implement and manage multiple infrastructures and provide them the freedom to start with an implementation of basic capabilities and easily add on products that build on these basic capabilities to deliver increasingly sophisticated functionality. This is the plan for the Longhorn timeframe. In the near-term, Windows Update Services, which is scheduled for release in H will deliver the first installation of the core Windows patch management infrastructure, extend support to additional Microsoft products, and provide significantly enhanced patch management capabilities. In conjunction with the release of WUS, SMS will be updated to leverage the new functionality provided via WUS, initially using it to do update scanning. © 2002 Microsoft Corporation. All rights reserved. This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.
42 Windows Update Services*The update management component of Windows Server that enables IT administrators to more easily assess, control and automate the deployment of Microsoft software updates Update management solution for all Microsoft products Initially supports Windows XP Pro, Windows 2000 Pro, Windows 2000 Server, Windows Server 2003, Office XP, Office 2003, SQL Server 2000, MSDE 2000, Exchange 2003, + additional products over time** Support for additional update types – security, critical and non-critical updates, update rollups, service packs, feature packs, and critical driver updates Core update management infrastructure in Windows Data Model - supersedence, update dependency & bundle relationships Built-in update scanning engine to detect missing updates Server APIs (.NET) and remoteable Client APIs (COM) Enhanced bandwidth optimization Uses BITS for client-server and server-server communication ‘Binary delta compression’ technologies dramatically reduce data transfer needs Configurable update subscriptions -- specify subset of content to be downloaded This slide and the next one provide additional detail on the functionality planned for delivery in the Windows Update Services. See the Windows Update Services datasheet at for additional information on the benefits and features of Windows Update Services In addition to providing the ability to patch and update various version of Windows, WUS will initially provide support for updating Office 2003, SQL Server 2000, and Exchange Microsoft will continue to add support for additional versions of Office, SQL Server, and Exchange, as well as for other Microsoft products over time. It should be noted that support of these additional version and products will be delivered without the need to upgrade or redeploy WUS A major area of focus for WUS is delivery of the initial set of core patch management infrastructure. In addition to providing WUS server and client APIs, WUS will include a data model for patch management that will enable specification of bundle relationships, dependencies, and supersedence between patches / updates. BITS technology provides ability to only use unused network bandwidth, restart download from point of network loss, and specify max bandwidth usage *WUS is currently in beta. Microsoft does not guarantee that all capabilities listed will be in the released version. Datasheet and sign up for the Open Evaluation Program at: **Without the need to upgrade or redeploy WUS © 2002 Microsoft Corporation. All rights reserved. This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.
43 Windows Update Services (2)Expanded administrative control Scanning: Pre-deployment scan for missing updates Download & approval: Specify only metadata be downloaded, rules for auto-approving updates, etc. Targeting: Install or uninstall to systems grouped via enumerated lists or Group Policy Scheduling: Set new update detection frequency*, specify install deadline**, etc. Implementation: Options to use specified communication port, work with Internet proxy, deploy in hierarchical replica or independently managed server topologies, support update management for networks not connected to the Internet, etc. End-user experience: Options to notify users of new updates, reboot, etc. Status reporting Deployment status aggregation per machine/per update/per group Download / install success, failure, and error info Logs statistics to SQL Server or MSDE Improved ease of administration New, intuitive Web administration console simplifies ongoing administration and provides detailed information on new updates Command line utilities and scriptability to enable scalable, efficient administration Microsoft will have a single update scanning engine that will be built-in to the Windows Update Services client (AU client). *Max. frequency 1/hour. Can use command line option or script to trigger new update checks on demand **Deadlines also enable enforcement of update installs (re-installation of required updates removed from the system at a later date) © 2002 Microsoft Corporation. All rights reserved. This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.
44 Windows Update ServicesComparing Microsoft Update, Windows Update Services, and SMS 2003 Adopt the solution that best meets the needs of your organization Capability Microsoft Update Windows Update Services SMS 2003 Supported Software and Content Supported Software for Content Same as Windows Update Services + WinXP Home Win2K, WS2003, WinXP Pro, Office 2003, Office XP, Exchange 2003, SQL Server 2000, MSDE Same as Windows Update Services + NT 4.0 & Win98* + can update any other Windows based software Supported Content Types for Supported Software All software updates, critical driver updates, service packs (SPs), and feature packs (FPs) All software updates, critical driver updates, SPs, & FPs All updates, SPs, & FPs + supports update & app installs for any Windows based software Update Management Capabilities Targeting Content to Systems N/A Simple Advanced Network Bandwidth Optimization Yes Patch Distribution Control Patch Installation & Scheduling Flexibility Manual & end user controlled Patch Installation Status Reporting Install errors reported to user. Lists missing updates for accessing computer Deployment Planning Inventory Management No Compliance Checking No – status reporting only Now let’s compare the capabilities of Microsoft Update, Windows Update Services, and SMS As you can see, Windows Update Services provides significantly more functionality than SUS 1.0, but the enhanced capabilities still provide less control than the more advanced capabilities available in SMS 2003, since, as indicated previously, SMS 2003 is a full software distribution solution that includes inventory / asset management and provides more flexibility in administrative control. The notes below provide additional context for capabilities that need additional explanation. Targeting content to systems: Windows Update Services allows targeting based on server side target groups containing enumerated lists of systems as well as based on client side definitions (centrally configurable via group policy or scripting) of the target group a client belongs to. SMS 2003 allows additional flexibility over enumerated list and AD based group definitions by supporting WMI property based and customized criteria (via scripting) based group definitions. Patch distribution control: Windows Update Services uses a ‘pull’ mechanism where client systems contact the server for approved updates. Administrators can configure the frequency with which clients contact the server or a check for new approved updates can be triggered using a command line utility or script. SMS 2003 uses a push mechanism and allows additional control for when updates should be distributed to client systems, how the distributions should be sequenced, etc. Patch installation & scheduling flexibility: Windows Update Services allows specification of a deadline by when an update should be installed on the system. In addition to install deadline specification, SMS 2003 allows specification of rolling install windows (e.g., install between 1 am and 5 am and if that window is missed, between 12 noon and 1 pm, etc.) Patch installation status reporting: Windows Update Services provides pre-defined standard reports on install status. SMS 2003 provides pre-defined reports as well as the ability to define custom reports. Deployment planning: Windows Update Services allow administrators to find out which systems need specific updates so the admin can quickly assess the need for a patch across systems talking to the WUS server. SMS 2003 provide richer functionality in this area by maintaining an inventory of install status for all client systems and allowing definition of baseline system configurations (i.e., which patches should be installed on a system or category of systems) and delivering built-in functionality to plan the sequence of rolling out updates. Compliance checking: refers to the ability to report on which systems don’t have the full set of updates (as dictated by the organization’s compliance policy) as well as the missing updates that lead to non-compliance for each of the systems. Windows Update Services allows reporting of which updates are missing (relative to the full set of updates relevant for that system) but does not address compliance reporting for systems that are offline when the report is generated. In addition to maintaining and reporting statistics on install status for currently connected and offline client systems, SMS 2003 allows on-demand compliance scanning and reporting to verify currently installed updates and show gaps between reference systems (in-compliance systems) and scanned systems. *MBSA does not support scanning Win98 – Win98 can be updated using SMS2003 inventory management and software distribution capabilities © 2002 Microsoft Corporation. All rights reserved. This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.
45 Windows Update Services*Choosing A Patch Management Solution Typical Customer Decisions Customer Type Scenario Customer Chooses Large or Medium Enterprise Want single flexible update management solution with extended level of control to update (+ distribute) ALL Windows OSes and Applications, as well as an integrated asset management solution SMS 2003 Want update management-only solution that provides simple updating for Microsoft software and initially supports Windows (Win2K & later versions), Office (2003 & XP), Exchange 2003, SQL Server 2000, and MSDE 2000 Windows Update Services* Small Business Have at least 1 Windows server and 1 IT administrator All other scenarios Microsoft Update* Consumer All scenarios We believe that large and medium enterprise customers will most likely choose between SMS and Windows Update Services. Those that want a full software distribution that includes patch management and asset management functionality will likely choose SMS while those looking for an update management only solution will likely choose SUS. For small business customers, the logical options are Windows Update Services and Microsoft Update. Customers in this size segment who have at least one Windows Server and one skilled administrator will choose Windows Update Services while the rest will use Microsoft Update. For consumers, the only logical choice is Microsoft Update. *Customer uses Windows Update, another update tool, or manual update process for OS versions & applications not supported by Windows Update Services or Microsoft Update © 2002 Microsoft Corporation. All rights reserved. This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.
46 Consolidated Solutions RoadmapLonghorn Time frame Q4/2003 H1/2005 Update Content Repositories and Online Services Download Center Download Center Windows Update Windows Update Windows Update Office Update Microsoft Update Microsoft Update 3rd party apps update repository Standalone Update Scanning Tools Office Inventory Tool Office Inventory Tool In-house developed apps update repository MBSA 1.2 (includes OIT) MBSA 2.0 MBSA 1.1.1 MBSA 1.1.1 This slide provide a visual representation of Microsoft’s overall roadmap for update management solutions – notice how the various tools get integrated and consolidated over time, a single update management infrastructure is delivered in Windows, and this same infrastructure can be leveraged by add-on solutions to update Microsoft, packaged software, as well as in-house built software. SMS 2.0 with Feature Pack SMS 2003/ WUS phase 1 integration SMS v4 3rd Party / In-house Tools WUS N.0 Windows Server Longhorn SMS 2003 SUS 1.0 WUS Server WUS Client Update Management Products Manual / Script Based Updating © 2002 Microsoft Corporation. All rights reserved. This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.
47 Adopt a Patch Management SolutionAt Microsoft, our #1 concern is the security and availability of your IT environment If none of the Microsoft patch management solutions meet your needs consider implementing a solution from another vendor Partial list* of available products: Company Name Product Name Company URL Altiris, Inc. Altiris Patch Management BigFix, Inc. BigFix Patch Manager Configuresoft, Inc. Security Update Manager Ecora, Inc. Ecora Patch Manager GFI Software, Ltd. GFI LANguard Network Security Scanner Gravity Storm Software, LLC Service Pack Manager 2000 LANDesk Software, Ltd LANDesk Patch Manager Novadigm, Inc. Radia Patch Manager PatchLink Corp. PatchLink Update Shavlik Technologies HFNetChk Pro St. Bernard Software UpdateExpert Finally, I’d like to stress that Microsoft’s #1 priority is the security and availability of your IT environment. Consequently if you feel that none of the Microsoft offerings meets your needs, we strongly encourage you to evaluate a patch management solution from another vendor. Here’s a listing of some of the companies that provide patch management products along with the names of their products and company URL. This is not an exhaustive list of vendors providing products in this space and is only meant to provide a sampling of available products. Patch Management capabilities are also included in Enterprise Systems Management products from IBM, Computer Associates, HP, and others. Microsoft does not endorse, recommend, or support any of these products but encourages customers to evaluate non-Microsoft options if to determine if they better meet your needs. *Microsoft does not endorse or recommend a specific patch management product or company Note: Enterprise Systems Management products such as IBM Tivoli, CA Unicenter, BMC Patrol, and HP OpenView may also provide patch management functionality © 2002 Microsoft Corporation. All rights reserved. This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.
48 Summary Addressing the patch management issue is a top priorityTaking a comprehensive, tactical & strategic approach Made progress, but much more work to be done Microsoft focused on: Reducing the number of vulnerabilities & associated patches Improving customer preparedness, training & communication Simplifying & standardizing the patching experience Improving patch quality Unifying and strengthening patch management offerings Key Recommendations: Implement a good patch management process – it’s the key to success Adopt a patch management solution that best fits your needs Make use of the resources referenced in these slides To summarize… …please review the slides in the appendix for additional information. © 2002 Microsoft Corporation. All rights reserved. This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.
49 Security Incident ResponseWe’ve talked about how Microsoft is addressing the patch management situation, the progress made to date in this area, the currently available offerings, and how to choose between the various offerings. Now let’s discuss the roadmap for the various areas of patch management. © 2002 Microsoft Corporation. All rights reserved. This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.
50 Trends – 2003 CSI / FBI SurveyOf 532 respondents, 92% detected attacks Only 251 organizations were able to quantify losses 25% of respondents suffered attacks on WWW servers Only 50% of intrusions were reported to law enforcement for complete results According to a recent CNet article, every week there are over 4,000 DDoS attacks that occur on the Internet that are performed using, by some estimates over 100,000 compromised hosts (http://news.com.com/ html?tag=nl) The hosts that participate in these DDoS attacks, sometimes referred to as ‘Zombies’ are accumulated by self-propagating worms. A financial institution may be the victim of far more targeted attacks than say a small business connected to the Internet via a DSL line which may be the target of far more WORM attacks and may never or only occasionally be targeted individually. To get a “feel” for what the most attacked / ports / protocols are simply consult any number of “Internet Health” sites such as: For information about the DNS root servers performance: © 2002 Microsoft Corporation. All rights reserved. This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.
51 Case Study – Edge ServerSymptoms Admin shares deleted repeatedly New service / security patch installed Server reboots unexpectedly Bandwidth consumption / server sluggish Low disk space Findings Malware “hidden” (+H) in subdir of system32 Malware “hidden” (+H) in c:\recycler Malware really hidden in “c:\System Volume Information” directory FTP / Backdoor Server installed to run as SYSTEM service
52 Case Study – Intranet DoSSymptoms High CPU utilization on affected systems (DC’s may have high CPU in LSASS) Account lockouts Increased TCP 139/445 network traffic RPC / LSASS crashing, machines rebooting AV stops working on some machines Can’t access AV web sites on some machines Findings You’ve got bot like Gaobot.AFW or Agobot.JF, Phatbot, SDBot, Randex Agobot.JF – Trend Micro This worm is the first known AGOBOT variant to exploit the Windows LSASS Vulnerability (MS04-11), which is a buffer overrun vulnerability that allows remote code execution and enables an attacker to gain full control of the affected system. For more information on this vulnerability, please refer to the following Microsoft page: Microsoft Security Bulletin MS04-011 It also attempts to log on to systems using a list of user names and passwords. It drops a copy of itself into accessible machines. This worm has backdoor capabilities. It executes commands sent in via Internet Relay Chat (IRC). It terminates certain antivirus processes and other security-related programs. It also modifies the Windows HOSTS file so that any access to specific antivirus Web sites is redirected to the local machine. This UPX-compressed worm runs on Windows NT, 2000, and XP. Gaobot.AFW – Symantec W32.Gaobot.AFW is a worm that spreads through open network shares and several Windows vulnerabilities including: The DCOM RPC Vulnerability (described in Microsoft Security Bulletin MS03-026) using TCP port 135. The WebDav Vulnerability (described in Microsoft Security Bulletin MS03-007) using TCP port 80. The Workstation Service Buffer Overrun Vulnerability (described in Microsoft Security Bulletin MS03-049) using TCP port 445. Windows XP users are protected against this vulnerability if Microsoft Security Bulletin MS has been applied. Windows 2000 users must apply MS The UPnP NOTIFY Buffer Overflow Vulnerability (described in Microsoft Security Bulletin MS01-059). The vulnerabilities in the Microsoft SQL Server 2000 or MSDE 2000 audit (described in Microsoft Security Bulletin MS02-061) using UDP port 1434. Exploits the Microsoft Windows Local Security Authority Service Remote Buffer Overflow (described in Microsoft Security Bulletin MS04-011). The worm also spreads through backdoors that the Beagle and Mydoom worms and the Optix family of backdoors install. W32.Gaobot.AFW can act as a backdoor server program and attack other systems. It attempts to kill the processes of many antivirus and security programs. © 2002 Microsoft Corporation. All rights reserved. This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.
53 There is no spoon . . . In the last century, organizations relied upon firewalls / perimeter defense as the basis for protecting the Intranet This has created a hard crunchy shell with a soft chewy center for most organizations In the 21st century with blended threats, firewalls alone do not effectively stop worms Did your firewall stop Slammer or Blaster? Will it stop bots like Gaobot / Phatbot / Agobot? VPN connections from home machines blur the ‘perimeter’ and increase the threat of automated attacks The Perimeter: Throughout much of the 80’s and 90’s organizations security strategies revolved around the perimeter with many organizations believing that port filtering and active content inspection was all that was needed to defend internal hosts from hostile attacks. Throughout the 90’s we learned that this was simply not enough, and that good security best practices & policies must extend all the way to the host to mitigate emerging security threats. A good question to ask is: If my firewall were compromised or misconfigured, how long would the hosts behind it last before being compromised? “Hard crunchy shell, with a soft chewy center” is attributed to Rob Thomas (www.cymru.com) from whom I’ve borrowed the phrase with permission. “UFTP – Universal Firewall Traversal Protocol” is attributed to Jesper Johannson & Steve Riley. © 2002 Microsoft Corporation. All rights reserved. This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.
54 Threats – Modus OperandiFact: Most intrusions are not accomplished via awe-inspiring skill. Fact: It is much harder to secure than it is to hack. Most intrusions involve: Weak administrator passwords!!! Un-patched security vulnerabilities in underlying software products (OS and applications) Weak out of box security settings that were never hardened Lack of secure coding in custom applications Intruders taking advantages of poor security posture usually rely on an organizations lack of an incident response plan and ability to recognize that an intrusion has occurred rather than super-sophisticated techniques and skill. Once it is known that a security incident has occurred, in the vast majority of cases it is usually fairly trivial for an incident response specialist to gather information about a system and determine whether it has been attacked or compromised (but as with everything, it’s all about having the right tools for the job). Many times the attack tools are created and automated by an elite few and then used and distributed by the relatively clueless masses (example: Jillwin32.exe, Kaht.exe for NTDLL.DLL BO). Other times the attack tools are created and distributed by relatively clueless individuals but are still very effective (example most SMB worms / botnets which combine batch files and malicious .exe’s to gain unauthorized access via weak admin passwords to setup a machine for participation in DDoS attacks) in low numbers. Something that is effective in low numbers (i.e. 1% of the time) can be made VERY effective if automated and made ‘wormable’ and unleashed on the Internet. © 2002 Microsoft Corporation. All rights reserved. This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.
55 Recommendations Normal operations staff trained to recognize symptoms of security incidents Escalate cases to security incident response team to: Determine time / date intrusion occurred Determine how the intrusion occurred Develop ‘signature’ for the intrusion Scan nearby machines for ‘signature’ Make changes to security posture to prevent future incidents
56 Preparing a Security Incident Response PlanProcesses should be put in place before an incident has occurred that will facilitate: Detection Determining whether an incident has occurred Investigation Determining how an incident has occurred Containment Isolating affected hosts Resolution Restoring service / lessons learned This is where business continuity planning comes into play, security policies, escalation procedures, contact lists etc. Process plays arguably the biggest part of creating an effective security incident response team. Organizations should start by creating a configuration management database to catalog all of the servers and applications running in an environment. This database should contain at a minimum: Asset tag or serial number information for each system. MAC addresses for each NIC in each system An owner for each system Authorized applications / role of each system Processes should be put in place to allow suspicious activity / events to be escalated to the security incident response team and a game plan should be developed for responding to each type of incident. Threat modeling should be used along with risk assessment to determine likely attack vectors and to assign value to assets. © 2002 Microsoft Corporation. All rights reserved. This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.
57 Escalating the IncidentDefine symptoms or behaviors that become triggers that will kick off an investigation Ensure admins and helpdesk staff understand and can recognize them! Security Incident Response team should Compare current ‘state’ to previous ‘state’ Look for new processes, files, folders, network connections, listening ports, services Not possible if you don’t know what the previous state was Baseline and catalog your servers! Run a live response IR toolkit to collect data Have trained IR specialist analyze output How do you know what ‘abnormal’ looks like if you don’t know what ‘normal’ looks like? Many intrusions go un-noticed for months (sometimes even years) because admins can’t distinguish ‘abnormal’ behavior form ‘normal’ Baseline your servers using Performance Monitor / System Monitor Baseline your networks by collecting and analyzing Netflow data An incident response plan may be activated under the following scenarios: An IDS sensor alerts you to suspicious network activity (i.e. a vulnerability scan is being performed against a device / server) A periodic port scan of the network reveals new / unauthorized servers listening on an IP address A periodic port scan of the network reveals a new application listening on an unauthorized port based on the role of the server Netflow data from your perimeter networking equipment indicates that more than 2x the normal limit of outbound traffic has been reached or that a new un-identified port is suddenly accounting for XX percentage of network bandwidth. A system administrator or user reports unexplained activity (i.e. mysterious reboot, suspicious event log entries, sluggishness, processes crashing etc.) Incident Reporting is often one of the biggest challenges. Often an incident is noticed fairly early on in the response processes, however notification of the *proper* individuals can takes hours / days if it happens at all! © 2002 Microsoft Corporation. All rights reserved. This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.
58 Suspicious Symptoms, BehaviorsSuspicious event log data Suspicious server reboot (no admins remember rebooting) Admin shares disappearing Security patches installed mysteriously News processes / services / files / folders Abnormal process termination (i.e. IIS crashes) A blue-screen occurs Sluggish system performance Suspicious network traffic to/from an IP address These become triggers or entry points into the investigation © 2002 Microsoft Corporation. All rights reserved. This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.
59 Things You Need To Know Why you need an Incident Response team within your organization Because it’s not a matter of ‘if’ but ‘when’ Auditing is everything Sufficient auditing is not usually enabled by default! Proper business continuity planning facilitates successful incident response If business isn’t down – more likely to have time to do a proper investigation Most organizations realize they should be auditing data, and most organizations backup critical information, but many organizations do NOT realize the importance of backing up critical audit / logging data. Scenario: Your incident response team discovers that a web server has been compromised. One of the first questions asked by management was ‘how long have the intruders had unauthorized access to this server’. To answer this question the incident response team will likely need access to known-good copies of the event logs and or the IIS logs for the last XX months. The nightly tape backups are a possible solution here and the backup data from 6 months ago is restored to an empty partition only to discover that the event logs have never been properly backed up (because at the time of the backups they are always open by the Event Log service and thus are not backed up properly). In addition the IIS logs have never been part of the backup set because they are too big / cumbersome and would cause the backup process to take too long. The IIS logs on the disk have been deleted for the last 6 months by the attacker (eliminating possible clues as to how they gained unauthorized access). To recover the deleted IIS logs you may have to resort to specialized software like EnCase. Event log and application log data should be backed up at an interval that matches the importance of the role of the server. It may be wise to at a minimum make daily / nightly backups of the event logs and application logs or you may wish to perform this task hourly!! It may also be interesting to store these backups on some form of read only media (such as CDR or DVD+R etc.) to ensure they are not tampered with. The “Microsoft Audit Collection System” currently in beta will offer secure remote logging of event log data (encrypted, signed, transactional etc.). Chapter 12 of the Windows Security Resource Kit provides excellent information on tools to use to archive event log data and which events to watch out for! © 2002 Microsoft Corporation. All rights reserved. This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.
60 Building the Security Incident Response Team
61 Overview Training – Staying Current Tracking Security IncidentsLive Response vs. Offline Response Assembling a Live Response Toolkit Microsoft PSS Security Incident Response Toolkit
62 Training Know your adversary Learn Incident ResponseStrongly recommend reading security and hacking related books Attend security conferences (Blackhat, RSA etc.) Subscribe to managed security service (ISS, TruSecure, LUHRQ etc.) Learn Incident Response Read books Attend specialized incident response training Managed Security Services © 2002 Microsoft Corporation. All rights reserved. This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.
63 Training Recommended resources Hacking KnowledgeHacking Exposed series of books Security Warrior Stay abreast of security vulnerabilities and exploits as they are released by subscribing to managed security services and monitoring Full-Disclosure mailing list Exploit web sites Incident Response Knowledge Windows Security Resource Kit: Foundstone: Ultimate Hacking Incident Response / Forensics Incident Response & Computer Forensics 2nd Ed. SANS: Track 4 – Incident Handling CERT Incident Response Handbook: Full Disclosure: © 2002 Microsoft Corporation. All rights reserved. This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.
64 Tracking Incidents Tracking incidents is extremely important OptionsHistorical data can be used to spot trends Central repository for keeping case notes during an investigation (encrypted?) Can be used for reporting progress to upper level management as incidents are resolved Options Literally Hundreds of Help Desk software solutions Request Tracker IR (Best Practical) Request Tracking software specifically for CERT teams Track-IT! (Intuit) CRM / CIM Solutions – Not always a great fit here Home grown solution may be best? A good list of resources can be found here: © 2002 Microsoft Corporation. All rights reserved. This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.
65 Live Response vs. Offline ResponseTwo different approaches to IR Offline response involves imaging disks and using specialized software to look for clues and evidence ProDiscover IR EnCase NOT mutually exclusive Create disk image first for use with ProDiscover / EnCase if necessary Then perform live response using automated IR toolkit
66 Live Response: Risks RootkitsIntroduced for Windows, publicly, circa 1997 They modify operating system behavior to hide files, folders, processes, registry entries, and network connections to avoid detection by live response tools Kernel mode drivers, usermode processes By observing the system, you alter its state Sort of like Schroedinger’s cat theorem. Placing output on target system overwrites free space / slack space etc. Altering time stamps and files may invalidate collected evidence if pursuing litigation
67 Assembling a Live Response ToolkitPurpose Offline forensic analysis not always possible, needed or timely Technical barriers, unacceptable downtime etc. Not always able to respond, in person to remote locations Live response toolkit facilitates consistent data collection from remote systems for offline analysis by an IR specialist Can be used as a first response tool to triage and investigate reported security incidents Systems can remain online during investigation Very important when an intrusion has not been confirmed positively Offline analysis has specific applications. Offline analysis is good for investigations where someone has hidden something on a disk and you are trying to find it to use as evidence in a court of law. For example pornography cases often involve an offline disk image. In an enterprise where you are responding to security threats (vs. acceptable use policy / misuse cases) offline response isn’t always a good fit. In addition it may not be technically possible. How do you image a SAN or a NAS device? How do you image RAID arrays? © 2002 Microsoft Corporation. All rights reserved. This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.
68 Microsoft Incident Response ToolkitDesign Goals Trustworthiness (anticipate that a rootkit is installed) Run in automated fashion on NT4 or later Collect volatile data from a live system Compress collected data into a .CAB file for submission to an IR specialist Not designed to Create or preserve evidence for use by law enforcement for use in legal proceedings Image a drive for offline analysis and response The incident response toolkit can be burned to read-only media to avoid having to overwrite free space and slack space on a hard drive While it is not designed to collect evidence for law enforcement the toolkit can be used to generate MD5 and SHA-1 values for all executable content on the disk as well as for the files produced by the IR toolkit. Offline response © 2002 Microsoft Corporation. All rights reserved. This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.
69 Microsoft Incident Response ToolkitTwo tools Data collection agent (The “IR toolkit”) Batch file that automates dozens of .EXE’s zipped up in a zip file with a readme.txt Data analysis tool (The “IR Viewer”) C# application, runs on examiners workstation Utilizes custom-built tools designed for incident response Utilizes free 3rd party tools Had to work with legal team and get written permission from authors to redistribute their tools! Be aware of EULA’s and licensing fee’s associated with ‘free’ tools when used in a business environment
70 Microsoft Incident Response ToolkitRandomized filenames Gets local system / Internet time kernel profiler Netstat / arp / ipconfig / routing table DIR commands (hidden, modified, accessed, created) Rootkit detection Dumps registry as text Saves event logs as TSV Enumerate NULL session information Get patch status Scan for ADS’s Enumerate running processes Get file versions of all loaded modules / key directories Get audit policy Dump security policy information (policy, users, rights, etc.) Map processes Ports Enumerate installed services several ways Enumerate ACL’s (if specified) Generate hashes for executables (if specified) Run ‘net’ commands Dump scheduled tasks Copies all .log, .bat, .cmd, .vbs, .js files from system32 DIR commands run with elevated privileges to be able to list contents of all directories (SeBackupPrivilege) Rootkit detection – numerous tools © 2002 Microsoft Corporation. All rights reserved. This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.
71 Microsoft Incident Response ToolkitTakes anywhere from 10 to 20 minutes to run Can be used to identify signs of an intrusion (some rootkits, suspicious processes, services, files, folders, registry entries, event log entries, suspicious accounts in the administrator group, missing security patches etc. Areas for improvement Better approach to rootkit detection (in progress) Run file system commands as SYSTEM (in progress) Registry last write times (in progress) Rootkit detection will always be a huge risk when performing live response. New ways of hiding information (processes, files, folders, drivers) from the operating system and IR tools are constantly being developed. Microsoft is working hard on this problem – the Next Generation Secure Computing Base will be a combined hardware + software solution designed to guarantee the authenticity and integrity of certain parts of the operating system. © 2002 Microsoft Corporation. All rights reserved. This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.
72 Security Incident Response Team Objectives
73 Incident Response ObjectivesConfirm whether an intrusion has actually occurred By analyzing the contents of the IR toolkit output for a specific server(s) Determine when the intrusion occurred Based on a lead like an event ID or a suspicious files or folders creation date Determine how the intrusion occurred Based on implicit or explicit evidence (absence of a critical security update at the time the intrusion occurred etc.) Identifies weakness in security posture and leads to corrective action being taken If new malware identified – submit samples to the antivirus partners PSS Security team in partnership with most leading antivirus vendors Virus Information Alliance: The Microsoft Product Support Services (PSS) Security Team is pleased to announce our participation in a virus information-sharing program with industry leading anti-virus software vendors. Microsoft has joined forces with several of our anti-virus partners to provide our customers with detailed information on significant viruses that are affecting Microsoft products and our customers. Members of this alliance will exchange valuable technical information on newly discovered viruses so that we can more quickly communicate to customers their targets, impact and methods of remediation. We believe this alliance will allow us to provide authoritative and timely information to customers on newly discovered viruses in the wild, while at the same time referring customers to their preferred participating anti-virus vendor for additional details. Microsoft Product Support Services will continue to offer additional support to customers affected by viruses free of charge and will work with our customer’s anti-virus vendor on the necessary steps for resolution. The PSS Security Team will also post updated information on this website regarding new and potentially damaging viruses that have been discovered in the wild. It is important to note that PSS Security will only provide information on viruses that affect Microsoft products and that meet the criteria for an alert as defined by our severity matrix. The PSS Security Virus Severity Matrix and its related definitions can be found here: Customers can find virus alerts issued by the PSS Security team at this link: The PSS Security Team looks forward to working with members of the Virus Information Alliance to help us deliver on our commitment to Trustworthy Computing by increasing the availability of accurate and timely information regarding virus outbreaks affecting our customers. Moving forward, PSS Security will also update this site will with additional information on virus trends, and, updated ‘how-to’ articles for all of our customer segments and additional information. Members of the Virus Information Alliance currently includes: Computer Associates F-secure Global Hauri Network Associates Norman Panda Sophos Sybari Symantec Trend Micro PSS Security Team © 2002 Microsoft Corporation. All rights reserved. This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.
74 To rebuild or not, that is the question!Microsoft’s stance It’s a risk assessment really We provide evidence (or lack thereof) of an intrusion. Sometimes we find no evidence of a compromise Most of the time it’s pretty straightforward We provide case notes for malware we’ve identified Submit to the AV partners so they can update signatures Customer usually cleans manually or waits for new sigs Other times, when a rootkit is known to be installed and hiding software, who knows what else is on the machine We recommend formatting and rebuilding the machine to a known good state Most of the time customers were hacked by an automated worm, not a targeted attack. Automated threats don’t usually employ active hiding techniques and are usually well studied by the AV vendors and can easily be removed by hand by deleting files, folders, and registry entries. Targeted attacks on the other hand can be trickier and require more caution. In general try to isolate the date / time the malware was dropped on the machine and then use that date / time (plus or minus a few days) to find other malware which may have been dropped on the machine. © 2002 Microsoft Corporation. All rights reserved. This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.
75 Facilitating Effective Incident ResponseHow to avoid common mistakes . . .
76 Common Mistakes Companies MakeWhen helping organizations investigate security incidents we see the same mistakes being made over and over again. The following slides detail the most common mistakes that are usually made and give guidance on how to avoid making these mistakes. Having helped numerous Microsoft customers by performing both security assessments and incident response engagements, there are some elements of security planning and incident response preparation that most organizations don’t have in place, which in various ways inhibit the incident response process. © 2002 Microsoft Corporation. All rights reserved. This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.
77 Common Mistakes Companies MakeNo formal, documented policies Server security hardening policy Acceptable Use policy Auditing policy Password complexity requirements Secure operating system builds Security patch deployment policy No formal change management process Many systems are shared between groups with many user accounts in the administrators group No process for tracking changes to the system back to a group or person No documentation about what should be installed on a system vs. what actually is installed on a system Most organizations have existing ‘security’ teams but the goals / mission statements are usually geared towards Incident PREVENTION and the goals of this team, with respect to incident RESPONSE may not be clearly defined (because by definition these teams are usually for Incident PREVENTION). Detection and/or Response sadly are usually roles that are overlooked because most organizations mistakenly think that if the security team ‘does their job’ they won’t ever have to worry about detection or response! The output of threat modeling / STRIDE can be used to justify the creation of an IR team to protect resources when a business plan / justification needs to be presented for approval before creating a dedicated IR team. Threat modeling (i.e. STRIDE, or threat tree’s, or both) can be used to identify potential threats to your network and then it is possible to rank these threats using DREAD. STRIDE S – Spoofing (identity etc.) T – Tampering (data etc.) R – Repudiation (denial) I – Information Disclosure D – Denial of Service E – Elevation of Privilege DREAD D – Damage Potential (how bad can it be?) R – Reproducibility (Is the attack reliable?) E – Exploitability (How easy is it to mount the attack?) A – Affected Users D – Discoverability This information should help you identify your potential threats and rank them in order of importance. For more information please read Chapter 4 of “Writing Secure Code, 2nd Edition” The team should have members from all aspects of the company Techies Network teams Server / Admin teams Application / Development teams Managers Once the security incident response team has identified that an incident has occurred they will need to summarize and report the incident to management so that they can make the important business decisions Public Relations Messaging may need to be communicated Trust me, you DON’T always want the techies talking to the press. © 2002 Microsoft Corporation. All rights reserved. This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.
78 Common Mistakes Companies MakeNo baseline data If you don’t know what ‘normal’ looks like – how can you spot abnormal behavior Perform software inventory updates Perform period port-scans of the network Know the normal operating thresholds for your servers Know the normal traffic patterns for your network Inability to ‘scale out’ during an investigation Suppose after the initial response you confirm that a group of servers were successfully attacked? How do you scale out the investigation to the neighboring servers / networks? Lack of proper base-lining: It’s hard to know what’s ABNORMAL when you don’t know what normal looks like. Determining what ‘normal’ looks like is also called ‘base-lining’ and plays a critical part early on in the Incident Response process. Sometimes the only clue that ‘something is wrong’ is when a system starts operating outside of normal baselines but if you don’t know what these are you may have to rely on someone else to report suspicious behavior to you (see example above): IR team should determine what ‘normal’ looks like (i.e. base-lining the network). Network base-lining: Perform periodic port scans of networked systems Goals: Catalog existing / new servers / systems that may be unauthorized Catalog existing / new applications running on those systems, listening on a port that may be unauthorized and increase the attack surface Analyze and monitor netflow data from network equipment that supports this Cisco usually does (please refer to the appendix for more information on Netflow technology) Answers questions like ‘how much of what protocol is flowing to what hosts at date / time X:Y:Z Server base-lining: Use Performance monitor to establish performance profiles for all critical servers within your organization. Most servers tend to quickly reach equilibrium and operate within a known, easily described profile (with some rare exceptions). Some organizations simply put up with servers that experience periodic / prolonged hangs, crashes or 100% CPU issues because they lack the technical expertise to resolve these issues or because they are infrequent enough, incurring the cost needed to resolve these issues isn’t a justifiable business expense and it’s easier to bounce the servers or restart the applications than it is to investigate and fix them! This is NOT a security-friendly business decision because these symptoms could ALSO be symptoms of attempted intrusion and / or compromise that you are ignoring!! If you have periodic but persistent outages due to these symptoms, you don’t have a ‘stable’ application and it becomes much harder to know what ‘normal’ looks like under these circumstances. If your security teams and support staff are used to frequent 100% CPU, hangs, and crashes they are much less likely to investigate such issues as security incidents and ignore them (like the boy who cried wolf too many times!) Example of system base-lining data: Our IIS web farm servers typically have the following 15 processes running in memory and operate at 50% CPU (on average). The Inetinfo.exe process typically consumes between 200 and 256MB of private bytes. Disk utilization due to caching is usually less than 10% write time and 5% read time. The server typically listens on the following ports and has between concurrent incoming connections from our load balancers IP address. Baseline the event logs Since you know you should already have a good audit policy in place, it is usually not wise to only review the event log data for the first time during an Incident Response investigation. Your security team should have a good understanding of what data is being audited and what it looks like ahead of time so that it is easier to spot anomalous activity in the event logs during an investigation. Chapter 12 of the Windows Security Resource Kit provides great tips and techniques for automating the analysis of event logs as well as providing guidance about which events are important to watch for and which events can be safely ignored (with respect to incident response). Inability to scale-out during an investigation During a security incident response investigation, in any sufficiently complex environment, it becomes necessary to ask ‘how many systems were impacted’. Once the symptoms of a security incident has become apparent (i.e. suspicious process listening on port XYZ, suspicious file in directory XYZ, suspicious process running in memory) it becomes necessary to determine how many systems may have similar symptoms. There is no easy solution to this problem and forcing the techies to solve this problem DURING the investigation is not the time to deal with this. Once an IR specialist has identified a suspicious ‘signature’ (for lack of a better word), having that IR specialist manually logon to all affected systems looking for signs of this ‘signature’ not only scales poorly, but may even alert the intruder that the game is up! The Security IR team should anticipate the need to be able to, at some point in time, rapidly scan their network for the presence of a suspicious file, or process, or registry setting in a scalable / automated fashion! Possible solutions: Research whether your existing software inventory management system can be used for this (i.e. Tivoli, SMS etc.) purpose Consider writing standard scripts that make use of WMI / automation to achieve this goal? Perl, VBScript. .NET? May have to write custom compiled applications? Microsoft developers wrote custom tools to scan networks for servers vulnerable to SQL Slammer and the IIS / NTDLL BO. For slammer we had to scan the file system for the presence of a file of a specific version For IIS / NTDLL BO we had to remotely inspect the registry and modify a setting. Automated toolkits help here. © 2002 Microsoft Corporation. All rights reserved. This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.
79 Common Mistakes Companies MakeNo formal security incident response team Why? Usually lack of budget and planning? Use some form of risk assessment and threat modeling to make a business case for a team! (STRIDE / DREAD) Incident Response team is old-school So you have an IR team but they aren’t up to date? Do they know about rootkits? Do they know about the latest worms and bots? Consider performing a penetration test of the environment to see how they do. Play with malware and study it in undoable isolated virtual machines! STRIDE is a method for categorizing potential threats to a system STRIDE – Spoofing Tampering Repudiation Information disclosure Denial of service Elevation of privilege DREAD is a method for ranking identified threats to a system DREAD – Damage potential, Reproducibility, Exploitability, Affected Users, Discoverability These methods can be applied to software development or an entire network of systems to asses risk. IR team has never been tested The key to a successful incident response team is experience. It is not enough to read books and play with the tools. The team needs to actually perform several ‘real world’ investigations (either simulated or real) and iteratively improve on the process. Although there is a certain element of ‘learn as you go’ in most investigations (no two IR investigations I’ve done have been the same) the plans and procedures and tools used during an investigation should all be well understood and well known before commencing. The security IR team should be aware of the security threats facing their environment and when / where possible the exploits AND IR tools should be downloaded and evaluated TOGETHER in a lab environment to determine ‘what they look like’ on the network, in the audit logs, etc. The biggest questions / cause for uncertainty seem to stem from events in one of the 3 event logs (what do they mean? Why are they there? Are they normal?). Proper base-lining and testing can tell you what is ‘normal’ in an event log and what is not before your first IR engagement which will save you precious time. At Microsoft we are big fans of emulation. I personally use Microsoft Virtual PC to do a lot of testing / repro’s with malicious / suspicious programs. VMWare and Virtual PC allow you to make a VM ‘un-doable’ or non-persistent so that when you power off the VM, all changes made are undone (think of it as a read-only OS image). © 2002 Microsoft Corporation. All rights reserved. This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.
80 Common Mistakes Companies MakeLack of a business continuity plan Some security incidents can be investigated while the systems are on-line, others require off-line analysis How long can you afford to be down? Lack of a trusted IR toolkit An automated toolkit should be created to facilitate the process of gathering information off of live systems The output of the toolkit should be known and well understood! Lack of proper BCP (business continuity planning) Security incidents can result in significant down time for a company while an investigation is on-going. It is critical for the Security IR team to have contingency plans in place to allow for maximum up time while the investigation and containment of the problem is on-going. Part of this plan may involve Server provisioning: Use of standard known-good secure OS images and an automated deployment process to quickly get these images onto servers These images should be ‘slipstreamed’ with the latest windows service pack and necessary security rollups Network reconfiguration: Firewalls and edge / perimeter devices may need to be re-configured to filter ingress or egress traffic to cut off the source of an attack NOTE: IPSec policies can be quickly deployed in some situations as well to block / filter malicious / suspicious traffic without having to reconfigure hardware devices. Lack of proper tools There are many aspects of a system that need to be investigated when looking for signs of unusual or suspicious activity. Logged on users Suspicious network connections Suspicious processes / services Are they listening on ports? Hidden files / directories Open files / handles Suspicious users / groups / membership Suspicious shares Files embedded within alternate data streams Suspicious registry entries Local security settings Currently installed products & their patches Scheduled tasks Etc. It should be clear that there is no magic tool that will provide all of this information. You will have to assemble multiple / disparate tools to analyze and collect information about your systems. Organizations can either develop or use existing IR toolkits. More importantly, your IR staff should understand how to analyze and interpret the output of these tools! Using a tool for the first time during an investigation is NOT the time to learn how to use a tool or interpret its output © 2002 Microsoft Corporation. All rights reserved. This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.
81 Tips for Responding To Security IncidentsAdvice from the front line . . .
82 Incident Response TipsDecide as quickly as possible whether or not to involve law enforcement They have their own evidence collection process and procedures Anything you do before law enforcement is involved potentially hinders the investigation and collection of evidence Interview the person reporting the incident thoroughly What’s the behavior being reported, how are things different? What day / time did you first notice something was wrong? Write everything down and keep accurate time / date stamps
83 Identify Symptoms of a RootkitIf a rootkit is installed, the output of the IR toolkit should be considered trustworthy It is imperative to identify whether a rootkit is possibly installed right away Consider using rootkit detection tools like VICE
84 Identify Symptoms of a RootkitPort scan the server remotely from a known good machine (all TCP and UDP ports) Look for any ports that show up on the network but not in local netstat, portqry or fport output Sure sign that a rootkit is hiding a backdoor listening on a port Boot the system into safe mode and examine installed services Look for services that show up in safe mode but not normal mode (rootkit may not load in safe mode) Locally list the files in the %windir% directory and all subdirectories and then do it again from a mapped network drive Look for files that don’t show up locally but that do remotely (again, rootkit)
85 Identify Symptoms of a RootkitConfigure Device Manager to show ‘hidden’ devices and view them Look for suspicious device drivers under ‘Non-Plug and Play Drivers’
86 IR Toolkit Data AnalysisDetermining a Date / Time gives you something to search on Look for leads that will yield a date or a time Suspicious processes, services, event log entries or files created on or around the date / time of the reported incident Once you have a ‘lead’ (i.e. a suspicious process or service) get the creation date of the file on the file system Perform a search for other files created on or around that time
87 Build a Time-Line of EventsOnce you have found some ‘leads’ build a chain of events that paint the picture Example leads from the System Event log System mysteriously rebooted on 4/20/2004 at 2:41am Just before that a Microsoft Security update was installed by the ‘SYSTEM’ account Could be a remote-shell, attackers often install the security patch they used to compromise a system to prevent others from stealing it Look for files created on that date / time
88 Build a Time-Line of EventsExample Suspicious service identified in Services snap-in That’s your ‘lead’ Identify the process backing that service (double click the service) Find the creation date of that file Look for other files created on that date Look for account logons on that date at around that time Determine when security patches were installed relative to that date time (before or after?)
89 Look In The Right PlacesMiscreants often hide their malware in the c:\recycler\
90 What you don’t know can hurt you . . .Laws and Legal Issues What you don’t know can hurt you . . .
91 Laws and Legal Issues Decide early on whether you might want to prosecute or not There are usually laws surrounding the collection of evidence and surveillance In litigious investigations you will be much more successful if you involve law enforcement immediately The decision to prosecute is an extremely important one. If the goal of the incident response process is the containment of the problem and quick restoration of service without desire for prosecution, the incident response process can probably be handled internally by a dedicated security incident response team. However, if the goal of the incident response process is to gather evidence to be used against someone in a liturgical fashion, law enforcement agencies should be contacted immediately. FBI has regional CART (Computer Analysis & Response Team) teams © 2002 Microsoft Corporation. All rights reserved. This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.
92 Laws and Legal Issues Most companies have a lack of knowledge about “Cyber crime” laws Acceptable Use Policies Search and Seizure Laws Reasonable Expectation of Privacy Is it lawful to monitor an employees / network traffic / or search their hard drive? Due Diligence Laws Can you be held liable for personally identifiable information that was stolen? Always involve proper legal counsel at the onset of a security related incident response investigation! Legal There are lots of laws governing computer crime and investigations, especially concerning privacy of ‘personally identifiable information’ or ‘PII’. It is unlawful to “practice law without a license” so steer well clear of ‘playing lawyer’ and giving legal advice. The best legal advice to give is “always consult proper legal counsel for questions you may have about cyber crime laws”. Legal counsel should be involved at each step of the investigation to make sure the actions taken by the techies are legal! © 2002 Microsoft Corporation. All rights reserved. This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.
93 Laws and Legal Issues Australian LawsList of Worldwide Cyber Crime Law Links U.S. Laws European Laws Australian Laws More great links © 2002 Microsoft Corporation. All rights reserved. This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.
94 © 2004 Microsoft Corporation. All rights reserved.This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary. © 2002 Microsoft Corporation. All rights reserved. This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.