1 Software Testing & QA (III)Kerry Zhu
2 Part III: Applying Your Testing Skills 8. Configuration Testing. 9. Compatibility Testing. 10. Foreign-Language Testing. 11. Usability Testing. 12. Testing the Documentation. 13. Testing for Security 14. Web Site Testing
3 Applying Your Testing Skills Configuration TestingChapter 8 Applying Your Testing Skills Configuration Testing
4 Hardware
5 More hardware A3D Abit升技 Add Addonics花王 Adsp ALi扬智 Amax中宇 AMD Analog Devices Aopen建基 APAC ASIO Toshiba东芝 Trident Triplex启亨 VIA威盛 Videologic VORTEX ASUS华硕 ATrend中凌 AudioExcel Audiotrak Aureal傲锐 Aztech爱捷特 CirrusLogic Cmedia骅讯 Creative创新 Crystal Semiconductors CSUN世讯 DataExpert联讯 Voyetra Turtle Beach Wta Xitel Yamaha雅马哈 Yuan小影霸 Zoltrix速捷时 Diamond帝盟 DIYEDEN乐之邦 Eagle金鹰 Echo Emu ESI Gamtec和跃 Genius T&W同维 Taiyanfa太阳花 Terratec德国坦克 TM Togotech岛谷科技 Topstar顶星 Guillemot&Hercules大力神 HiTeC Hotonhitech恒邦高新 IBM InnoVISION映众 ITE联阳 Jazz Labway ESS ForteMedia Rta S3(VIA) SigmaTel SiS矽统 Soyo梅捷 Superpower上普 Leadtek丽台 Magic3D M-AUDIO Mediatek瑞丽 Megastar皇朝 Microsoft微软 nVIDIA Octek海洋 OOAOO傲王 OPTi PcChips明致 Philips飞利浦 Pine松景 PreSonus Realtek瑞昱 RealWorld RME Rongfeng融丰 FPM (Fast Page Mode) EDO (Extended Data Out) SGRAM (Synchronous Graphics RAM) SDRAM (Synchronous DRAM) Video RAM (VRAM) WRAM Window RAM Network card, Display card, Sound card, Printer, Scanner …
6 Contents Overview of configuration testing. Approaching Task Overview of configuration testing. Approaching Task Obtaining the Hardware Identifying Hardware Standards Configuration Testing Other Hardware
7 Highlights of this chapter include Why configuration testing is necessary. Why configuration testing can be a huge job. A basic approach to configuration testing. How to find the hardware you need to test with. What to do if you are not testing software for a desktop computer
8 What must be done ? Normally we do dynamic, white box testing for this. Don’t assume hardware standards, protocols, or specs are correct. First identify what is really going to affect the running of the software-No graphics, try standard video cards or none. Major Problem: If you suspect the fault is the hardware manufacturer, who is responsible?
9 How to handle this? Tries to answer the question, “ Does the software work correctly on a variety of hardware or, if the design called for proprietary hardware, does it run correctly on it?” As with earlier types of testing, it is impossible to check all possibilities: PC, components, peripherals, interfaces, options such as memory size, device drivers,... Book speculates 336 display cards, 210 sound cards, 1500 modems, 1200 printers which gives a total 12 billion configurations. Ultimately, the software developers are responsible for making things work, even if the hardware is faulty.
10 You can’t test all so you ... Equivalence partition! What hardware is really vital? Game – sound and video cards important Word processor- handles printing As before, might want to test Most popular hardware Only recent hardware Hardware that uses unique features of the software.
11 Avoiding the cost of buying all that hardware ... If you are clearly a software developer with some reputation, you can often have hardware donated or loaned for testing purposes. Some companies ask their employees to bring in units from home to run on the tests (with some incentive, of course). Another possibility- outsource. Note the locations of hardware specs. Most are on the web.
12 a. Overview of configuration testing 1. Configuration testing definition: Configuration testing is the process of checking the operation of the software you are testing with all these various types of hardware. 2. Different configuration possibilities for a standard PC used in Homes and businesses: PC; Components; Peripherals; Interfaces; Options and memory; Device Drivers.
13 a.1: Isolating Configuration Bugs (1) 1. Who should fix the bug? You discover a problem when you testing your software on a unique configuration. Your team or hardware manufacturer ? 2. How to identify Configuration bugs? The sure way to tell if a bug is a configuration problem and not just an ordinary bug is to perform the exact same operation that caused the problem, step by step, on another computer with a completely different configuration. If the bug does not occur, it is very likely a configuration problem. If the bug happens on more than one configuration, it is probably just a regular bug
14 a.1: Isolating Configuration Bugs (2) 3. All kinds of Bugs: 1.) Software may have a bug that appears under a broad class of configurations. 2.) Software may have a bug specific only to one particular configuration. 3.) The Hardware device or its device may have a bug that only your software reveals. 4.) The hardware device or its device drives may have a bug that can be seen with lots of other software.
15 a.2: Sizing Up the Job The job of configuration testing can be a huge undertaking. We need to figure out a way to reduce the huge set of possible configurations to the ones that matter the most. For example:
16 b. Approaching the Task (1) The following general process that you should use when planning your configuration testing. 1.) Decide the types of hardware you will need. Look closely a your software feature set to make sure that you cover everything. 2.) Decide what hardware brands, models, and device drivers are available. Work with your sales and marketing people to create a list of hardware to test with. If they can not or won’t help, grab some recent editions and back issues of some information to get an idea of what hardware is available and what is popular. Do some research to see if some of devices are clones of each other and therefore equivalent-falling under the same equivalence partition.
17 b. Approaching the Task (2) 3) Decide which hardware features, modes, and options are possible. Every device has options, and your software may not need to support all of them. A good example of this is computer games. 4) Pare down the identified hardware configurations to a manageable set. Given that you do not have the time or budget to test everything, you need to reduce the thousands of potential configurations into the ones that matter—the ones you are going to test.
18 b. Approaching the Task (3) Ultimately, the decision-marking process that you use to equivalence partition the configurations into smaller sets is up to you and your team. There is no right formula. Every software project is different and will have different selection criteria. Just make sure that everyone on the project team, especially your project manager, is aware of what configurations are being tested and what variables went into selecting them. For example:
19 b. Approaching the Task (4) 5.) Identify your software unique features that work with the hardware configurations. The key word here is unique. You do not want to, nor do you need to, completely test your software on each configuration. You need to test only those features that are different from each other that interact with the hardware. For example:
20 b. Approaching the Task (5) 6.) Design the test cases to run on each configuration. Select and set up the next test configuration from the list. Start the software. Load in the file test.doc Confirm that the displayed file is correct. Print the document. Confirm that there are no error messages and that the printed document matches the standard. Log any discrepancies as a bug. In reality, the steps would be much more involved, including more detail and specifics on exactly what to do.
21 b. Approaching the Task (6) 7.)Execute the tests on each configuration. You need to run the test cases and carefully log and report your results to your team, and to the hardware manufacturers if necessary. 8.) Rerun the tests until the results satisfy your team. It is not uncommon for configuration testing to run the entire course of a project. Initially a few configurations might be tried, then a full test pass, then smaller and smaller sets to confirm bug fixes. Eventually you will get to a point where there are no known bugs or to where the bugs that still exist are in uncommon or unlikely test configurations. At that point, you can call your configuration testing complete.
22 c. Obtaining the Hardware How to obtaining the all kinds of hardware unless buy? Buy only the configurations that you can or will use most often. Contact the hardware manufacturers and ask if they will lend or even give you the hardware. Send a memo or in your company asking what hardware they have in their office or even at home-and if they would allow you to run a few tests on it. If you have the budget, work with your project manager to contract out your test work to a professional configuration and compatibility test lab.
23 d. Identifying Hardware Standards Knowing some details of the hardware specifications can help you make more informed equivalence partition decisions. We can find the detail hardware specifications from internet. A easy way is search engine
24 e. Configuration Testing Other Hardware It does not matter what the hardware and software is and what it connects to; if it connects to anything else, configuration issues need to be tested. Create equivalence partitions of the HW based on input from the people who work with the equipment, your project manager, or your sales people. Develop test cases Collect the selected hardware Run the tests
25 Q & A
26 Exercise P140: 2., 3.
27 Applying Your Testing Skills Compatibility TestingChapter 9 Applying Your Testing Skills Compatibility Testing
28 Content Overview of Compatibility testing Overview of Compatibility testing Platform and Application Version Standards and Guidelines Data Sharing Compatibility System compatibility Data compatibility
29 Highlights of this chapter include What it means for software to be compatible How standards define compatible. What platforms are and what they mean for compatibility. Why being able to transfer data among software applications is the key to compatibility.
30 Overview of Compatibility testing 1. Software compatibility testing definition: Checking that your software interacts with and shares information correctly with other software. 2. Software compatibility testing on a new piece of software, we will need to get the answer to a following few questions: 1) What other platforms and application software is your software designed to be compatible with? 2) What compatibility standards or guidelines should be followed that define how your software should interact with other software? 3) What types of data will your software use to interact and share information with other platforms and software.
31 Examples [email protected]Cutting test from a web page and pasting it into a document opened in your word editor Saving accounting data from one spreadsheet program and then loading it into a completely different spreadsheet program Having photograph touchup software work correctly on different versions of the same operating system Having your word editor load in the names and addresses from your contact program and print out personalized invitations and envelopes Upgrading to a new database program and having all your existing database load in and work just as they did with the old program
32 Network Import/ExportFigure 9.1 Word Editor From Company U Running on Operating System W Company C Operating System L Spreadsheet Company L Operating System N File Import/Export Network Import/Export File Load/Save Cut, Copy, Paste Backup
33 b. Platform and Application Version Selecting the target platform or the compatible applications is really a program management or a marketing task. They will also identify the version or versions that the software need s to be compatible with. 1. Backward and Forward Compatibility. Backward/Forward Compatibility: If something is backward/forward compatible, it will work with previous/future versions of the software. 2. The Impact of Testing Multiple Versions. We can not test all the thousands of software programs on your operation system, so we need to decide which ones are the most important to test. The key word is important.
34 Backward & forward compatible Data.doc Word 98 running on Win98 Word 2003 running on Windows XP Forward Compatibility Backward Compatibility XXX running on OS 2005 Word 2000 running on Windows 2K Word running on Win NT Word 95/97 running on Win95/97 Data.RTF Data.txt Tools Load old data Import old data Exchange new data Must do Had better design
35 Impact of testing Multiple versions New Computing Platform 2005 Painting and Drawing Programs Database Word Processors Spreadsheet Programs Games Educational Programs Compatibility test Popularity: Top 100 or 1000 based on sales dada Age: <3-year old Type: break into types: DB, Word, Graph, … Manufacturer Criteria to Choose programs
36 New Example Between Powerpoint 2000/XP and Powerpoint 2003 Between Powerpoint 2000/XP and Powerpoint 2003 A .ppt file is created by Powerpoint 2000/XP, and edited by Powerpoint Then it cannot be opened by Powerpoint 2003, but can be opened by 2000. MS has to release a patch to fix it.
37 c. Standards and Guidelines 1. High-Level Standards and Guidelines High-Level standards are the ones that guide your product’s general compliance, it looks and feel, its supported features, and so on. 2. Low-Level Standards and Guidelines Low-level standards are the nitty-gritty details, such as the file formats and the network communications protocols. We should treat low-level compatibility standards as an extension of the software’s specification.
38 d. Data Sharing Compatibility A well-written program that supports and adheres to published standards and allows users to easily transfer data to and from other software is a great compatible product. File save and file load File export and file import Cut, Copy and paste DDE ( Dynamic Data Exchange ) OLE ( Object Linking and Embedding) Application #1 System Clipboard Application #2 Multiple Data Formats Single Data Temporary hold place
39 Exercise P151: 3.
40 Applying Your Testing Skills Foreign-Language TestingChapter 10 Applying Your Testing Skills Foreign-Language Testing
41 Highlights General Concepts Make the Words and Picture Make Sense General Concepts Make the Words and Picture Make Sense Translation Issue Localization Issue Configuration and Compatibility Issue How Much Should You Test?
42 General Concepts [email protected]Translation – the language piece only. Internationalization - Designing and engineering a product so that it can be easily “localized.” Localization - Adapting an entire product for a specific “locale” Globalization How to make product to basic i18n support
43 i18N (internationalization) is the process of designing an application so that it can be adapted to various languages and regions without engineering changes. Internationalization is the task of software developers. An internationalized program has the following characteristics: With the addition of localized data, the same executable can run worldwide. Textual elements, such as status messages and the GUI component labels, are not hardcoded in the program. Instead they are stored outside the source code and retrieved dynamically. Support for new languages does not require recompilation. Culturally-dependent data, such as dates and currencies, appear in formats that conform to the end user's region and language. It can be localized quickly.
44 L10N (localization) is the process of translating and adapting software to a particular language and culture for an already internationalized software. Localization need implementation translation of text, the change of UI, sounds and images, product testing。 Roughly, i18N is considered an engineering process while L10N is considered a translation process. Which part need consider Localization for a product? menu, dialog box, Hint, Alt text，Error message, static text etc. Local & Time Zone &Date&Number&Currency&Measurements Shortcut key and Hotkey, Bitmap & Icon Installation Wizard Message Template, On-line help and samples Documentations (User's Manual, Tutorial, Installation Guide etc) Package outlook
45 G11N (Globalization) Is a general term which is used to cover two different processes, internationalization and localization.
46 How to make product to basic i18n support For web page: make sure all UI strings that need to be localized are enclosed within web tags. Example: A PHP code segment without tag:
47 Translation Issue Text Expansion ASCII, DBCS, and Unicode Text Expansion ASCII, DBCS, and Unicode Hot Keys and Shortcuts Extended Characters Computations on Characters Reading Left to Right and Right to Left Text in Graphics Keep the Text of the Code
48 Translation Issue - Text Expansion
49 Translation Issue-DBCS Set correct maximum length of text field on page and in database schema, to avoid any database access error. 1.Change the MAXLENGTH of Text to the half of the original e.g: Original: should be: 2. Check the inputted char whether exceed the length before submit page
50 I18N Issues in web page Layout and UI issue Layout and UI issue Disorder code or garbled text Char Index or Sort Issues Full name difference between European and Asian
51 Localization Issue Content Data Format Content The Content is all the other “stuff” besides the code that goes into the product. We should consider all the parts which make up a software product. Data Format Different locales use different formats for data units. So we need to become very familiar with the units of measure when we’re testing localized software.
52 Localization Issue-Data Format
53 Example - Localization product Region —The selected region determines the formats of numbers, currency, dates, and time on the Web pages. Language —The default language for the Web pages, messages, and client software.
54 Configuration and Compatibility Issue Localized OS – use to interact with Localized names of built-in elements OS Environment of your market East Asian System locale Non-Unicode data path assumes single-byte text European System locales OEM vs. Windows “ANSI”
55 How Much Should You Test? Internationalization Testing Localization Testing Linguistic/Translation Testing Cosmetic/UI Testing Functionality Testing More on Functionality Testing Delivery Testing
56 INTERNATIONALIZATION TESTINGI18n Testing Internationalization testing is done in order to determine how well internationalization has been done. For instance, will the product be easy to localize? Have all the localizable resources been separated from the source code? Does the software support Unicode? INTERNATIONALIZATION TESTING INTERNATIONAL SUPPORT LOCALIZABILITY TEST Pseudo-translation (Catalyst) is an Important part of i18n testing. localizable resources externalized? any regional settings hard-coded? any concatenated strings? do the RC files contain non-localizables (over-externalizing)? text expansion allowed for? text on non-layered graphics? how many other components (icons, graphics, etc.) need to be adapted? MBC characters and scripts? MBC input and display? MBC folder, file, data handled? Regional settings? Collation/sorting? Run on localized OS? Keyboard support? MBC- Multiple Byte character
57 internationalization localization functionalityL10n QA L10n QA Assurance Procedures Testing prevention detection internationalization localization functionality It is important to note that the software the localization vendor receives is already tested and working. Localization is all about preventing working code from being broken during the localization process. Localization is therefore dependent on rigorous testing as part of the Localization QA process.
58 L10n Testing LOCALIZATION TESTING LINGUISTIC FUNCTIONAL COSMETIC LOCALIZATION TESTING LINGUISTIC FUNCTIONAL COSMETIC In-country testing Translation verification testing (TVT)
59 Linguistic Testing all text been translated? all text been translated? accented chars handled properly? punctuation rules of target? target word wrap, hyphenation, sorting? no truncations in dialogs? consistency in terminology/usage? all icons, graphics or sounds need to be adjusted? are concatenated strings displayed properly? have leading and trailing spaces been deleted, causing errors when strings are concatenated? are strings with variables displaying properly? are hot key and control key assignments consistent with OS standards? Focuses on all the language elements of an application. Done (ideally) in running localized application. Uses test scripts ideally.
60 Cosmetic (UI) Testing all of the menus, options and commands of the original? dialog boxes all properly resized? all characters display properly? did you test to see that all popup boxes, tool tips, balloons, status messages and dialogs fit on the screen at all resolutions? when expanding and resizing, has alignment and size consistency been maintained? hot keys unique? has the tab order (if any) of the original been changed? some controls (combo boxes, menus) have drop down elements, do they display properly? do all the dialogs display the correct regional settings? Focuses on all the visual elements of the UI. Done (ideally) in running localized application.
61 Functionality Testing did localization introduce any problems? All functions and features present? can the localized and original versions save and open the same files? do international keyboards and layouts work with the hot keys and control keys? is the proper regional setting, keyboard, language the default? does the clipboard preserve MBC and accented characters? Can they be cut and pasted to other applications? does the application work on both the localized and original version of the OS? do links to web and on-line help point to target language help? do target language spell checkers, style checkers, dictionaries work? does the localized version work on the hardware platform, with peripherals and accessories, drivers? Does the localized version interact with browsers and other programs properly? Focuses on whether the application still works after localization. A well internationalized product will likely not fail functionality testing. Not a standard task, but done at vendor request with test scripts.
62 More on Functionality Testing Seldom done as part of localization process. Cannot overcome lack of thorough internationalization testing! Duplication of source-language test scripts and routines required Special attention paid to locale-specific components and issues Need to create complete testing environments, client/server applications and hardware Client proprietary tools Test scripts integration testing: testing two localized products or components together to see how they work performance testing: how does the product (web site?) work under heavy load A localization vendor that has the capability of doing a comprehensive functionality test Suite canprdouce a so-called “gold master” of the localized software. This product can be manufactured and distributed as is.
63 TVT and other testing Translation verification testing (TVT) Translation verification testing (TVT) TVT ensures that the completed translation is contextually accurate, grammatically correct, and culturally appropriate. Certified linguists perform TVT In-country testing Verifies "last mile" connectivity (local ISP, wireless, and DSL services) and functionality (localized disk images) through VeriTest's network of regionally based test resources in Europe and Asia Release engineering VeriTest's Release Engineering services can dramatically reduce time-to-market. VeriTest software engineers can identify and correct many defects during the globalization testing process, and deliver gold master builds for release-to-market.
64 Delivery Testing Are the folders and files correct in number and location? Are all files in the specified formats and on the specified media? Are there the same number of files in the original and in the target? Has the installer and uninstaller been localized and tested? Are all the versions of files and components the correct and latest versions? Is the distribution media folder structure identical to the source? Have all files been virus checked? Focuses on ensuring that all the client required deliverables were provided according to project specifications.
65 Q & A
66 Exercise P168: 4., 6.
67 Applying Your Testing Skills Usability TestingChapter 11 Applying Your Testing Skills Usability Testing
68 Highlights Why need do Usability Testing Why need do Usability Testing User Interface Testing and Usability Testing What Makes a Good UI Testing for the Disabled: Accessibility Testing
69 What decide customer to choose a product? What decide customer to choose a product?
70 Which is important? Price Brand Quality Quality Price Brand All are right! But the importance is what the customer like.
71 What is usability? Easy to discover Easy to learn Easy to use Easy to discover Easy to learn Easy to use Availability
72 How to get good usability? 易见 Visibility 映射 Mapping 反馈 Feedback
73 Usability test in Microsoft180 Usability test engineers 25 Usability test Lab
74 UI /Usability Testing UI is User Interface - Obtain user input ; UI is User Interface - Obtain user input ; - Display the results; Usability is how appropriate, functional, and effective that interaction is. A good UI determine the usability of product; GUI’s need USABILITY testing The main content of usability testing is UI testing.
75 What Makes a Good UI Follows Standards or Guidelines Intuitive Correct Follows Standards or Guidelines Intuitive Correct Consistent Flexible Comfortable Useful Simple
76 Follows Standards or Guidelines Follows existing standards and guidelines – or has a really good reason not to. Have accounted for a great deal of formal testing, experience, and trial and error to devise rules that work well for their users You may create the usability standards for your software.
77 Intuitive [email protected]Is the user interface clean, unobtrusive, not busy? Is the UI organized and laid out well? Does it allow you to easily get from one function to another? Is there excessive functionality? If all else fails, does the help system really help you? 客户：喂，你们的产品我不会用。 技术支持：你可以看说明书啊！ 客户：可是你们的说明书有300多页啊，而且我也看不太懂。 技术支持：这样啊，我们建议你首先自学计算机专业本科的课程。 客户：…，算了我还是不要用了！！！
78 Different experience from different users % of Responses
79 Correct Language and Spelling Bad media Marketing Differences Language and Spelling Bad media WYSIWYG ( what you see is what you get)
80 Consistent Terminology and naming, “find” vs “Search” Audience Shortcut keys and menu selection, example, F1 –help Terminology and naming, “find” vs “Search” Audience Placement and keyboard equivalents for buttons Color, Shape, Text, Operation, …
81 Flexible State jumping State termination and skipping State jumping State termination and skipping Data input and output - type, paste, load file, insert object, drag, …
82 Comfortable Appropriateness: not too garish, not too plain, … Appropriateness: not too garish, not too plain, … Error handling Performance
83 Simple 1-click Next Next Next …
84 Testing for the disabled: - Accessibility Testing Visual impairments: Color blindness, extreme near and far sightedness, tunnel vision, dim vision, blurry vision Hearing impairments: someone may be partially or completely deaf, have problems hearing certain frequencies, … Voice or sound that accompany an onscreen video, audible help or system alerts. Motion impairments: It may be difficult or impossible for some people to properly use a keyboard or a mouse, … Cognitive and language: Dyslexia and memory problem may make it difficult for someone to use complex user interfaces It’s the Law – help the disabled
85 Example
86 Q & A
87 Exercise P182: 3., 4.
88 Applying Your Testing Skills Testing the DocumentationChapter 12 Applying Your Testing Skills Testing the Documentation
89 Highlight Distinguish between system & user documentation Distinguish between system & user documentation Types of Software Documentation The Importance of Documentation Testing What to Look for when Reviewing Documentation The Realities of Documentation Testing
90 Components of a Software
91 Distinguish between system & user documentation Two audiences for documentation The information systems personnel who will maintain the system throughout its productive life The people who will use the system as part of their daily lives System Documentation Detailed information about a system’s design specs, its internal workings, and its functionality User Documentation Written or visual information about an application system, how it works, and how to use it.
92 Documentation Types Types of Documentation Types of Documentation - Considering the Audience - User’s Manuals - Operator’s Manuals - General System Guide - Tutorials and Automated System Overviews - Other system Documentation User help and Troubleshooting Failure Message Reference Guide Online Help Quick Reference Guides
93 More Types of Documentation Package text and graphics Marketing material, ads and other inserts Warranty/Registration(sign up) EULA ( End User License Agreement) Labels and stickers Samples, example and template
94 User Manuals System Summary: Manual functional description: System Summary: - The system’s purpose or objectives - The system’s capabilities and functions - The system’s features, characteristics, and advantages, including a clear picture of what the system accomplishes Manual functional description: - A map of the major functions and how they relate to one another Each function in terms of the screens the user can expect to see, the purpose of each, and the result of each menu choice or function key selection All input expected by each function All output that can be created by each function The special features that can be invoked by each function
95 Failure Message Reference Guide - The name of the code component executing when the failure occurred - Source code line number in the component that was executing - Failure severity and its impact on the system - Contents of any relevant system memory or data pointers, such as registers or stack pointers - Nature of the failure, or a failure message number
96 Registration Return
97 EULA Return End User License Agreement
98 Installation and setup instructions
99 Importance of Documentation Testing Good documentation contributes to the product’s overall quality in three ways: It improves usability It improves reliability It improves support costs To uncover documentation fault, documentation checking is needed.
100 What to Look for when Reviewing DocumentationWhat Makes Good documentation? Documentation testing Criteria A Documentation testing Checklist
101 What Makes Good documentation Good user documentation includes: A glossary for unusual terminology Error messages, troubleshooting, and recovery information Index of key topics a detailed table of contents Good to outline the documentation first, and make sure key functions are accounted for Contains task-based documentation “How to’s…” Frequently Asked Questions Messages & their meanings Samples and examples Documentation should use short, simple paragraphs and sentences
102 Documentation testing Criteria Documentation Testing is conducted to ensure Correctness, Completeness, Understandability It is to justify that all document is up-to-date with respect to model logic specification.
103 Documentation testing Checklist General Areas Audience Terminology Content and Subject matter Correctness Just the facts Step by Step Figures and screen captures Samples and examples Spelling and grammar Microsoft is Microsoft
104 Realities of Documentation Testing What make documentation development and testing a bit different from software development: Documentation often gets the least attention, budget and resource; Documentation developers are not experts in software; Printed documentation takes time to produce;
105 Q & A
106 Exercise P197: 2., 4.
107 Applying Your Testing Skills Testing for SecurityChapter 13 Applying Your Testing Skills Testing for Security
108 Highlight Why someone would want to break into a computer Why someone would want to break into a computer What types of break-ins are common How to work with your design team to identify security issues Why software security problems are nothing more than software bugs How can find security vulnerabilities （弱点） How the new filed of computer forensics is related to software security testing
109 A example – WarGame, 1983
110 More examples
111 Understand the motivationChallenge Curiosity Use/Leverage Vandalize: 3-D ( defacing, destruction, denial of service – DoS ) Steal
112 Threat modeling Assemble the threat modeling team Identify the assetsCreate an architecture overview Decompose the application Identify, rank and document the threats Rank: damage potential, reproducibility, exploitability, affected users, discoverability
113 5 Principles Needing to Test Authentication: Identity - Validity Login, timeout, failures, pw changes, mins/maxs, stored encrypted, bypass captured URL, handling deletion of outdated, expirations, 2-factor:atm Unix:Access.conf, .htaccess, .nsconfig Windows: challenge/response; SSO; Passport Integrity: protection from tampering/spoofing (篡改/欺骗) Privacy: protection from eavesdropping (偷听) Non-Repudiation: accountability Availability: RAID,clusters,cold standbys 5 Principles of Security Test: A process! Integrity: dig sigs Availability: Prevent denial of service by disabling unneeded services, hiding internals, filtering (allow/deny) traffic, and load balancing
114 Some concepts Certificates LDAP Cryptography Encryption Certificates LDAP Cryptography Symmetric: Kerberos, Blowfish, DES Asymmetric: RSA, MD5, SHA-1 Encryption Dig sigs are a bit length hash constant (cipher) combined with a private key (variable). It should be tested that they are not modified or altered. Symmetric Keys use a reverse process using the SAME key. DES (by IBM) in the 70’s was considered high standard using a 56 bit key 2 to the 56 power (or 72 quadrillion keys). Blowfish was developed by Bruce Sheier, author of excellent security book, “Secrets and Lies”.
115 SERVERS: web, app, database server OS’s: NT, UNIX, LINUX Somarsoft’s DumpSec Reports Configuration: shares, services, registry, user enumeration, Access/Object Privileges/Views/Stored Procs Preventing DoS Preventing Buffer Overflows (example in P.202) Log Files: keep separate – less traffic Patches Compilers/Interpreters- don’t keep in cgi-bin Denial of Service attacks a server with bad data- Check the length of all buffers and inputs before storing them. Do load testing/load balancing. Not of sheer volume, but verifying the number of packets sent. Ex) PCANY 9/10 – random characters inserted may cause many open connections/SYN attack creates too many zombie processes Buffer Overflows: Not as possible in .Net framework since ‘verified code’ checks the assembly BEFORE executing. Use a boundary router to isolate the network- don’t wait until null. Ex) UPnP (universal plug and play in XP) will echo endless download of discovery or “Notify Directives” of devices on system. It will flood so hacker can run their own code. Disable SSDP Discovery Service (NOT UPnP Device Host).
116 CLIENT: browser, other apps, components Browser settings: Zones Macros – Shift OLE Trojan Horses Floppy Boot in BIOS Cookies never accepted in Restricted Zone
117 Cookies AcceptingCookies: Cannot be used as a virus or plug-in AcceptingCookies: Cannot be used as a virus or plug-in text only Max 4k Windows: Cookies.txt Unix: can be read into PERL using $ENV{‘HTTP_COOKIE’} When deleting- close browser first! NS limit = 300 total / 20 per domain IE limit = 2% default Encrypted in .Net “Super Cookies” – intruder uses jscript to grab GUID in Windows Media Player (GUID)- grab your web sites from temporary internet folder.
118 Open Systems Interconnect The Firewall Architecture Communication Stack. Layer 1- Hardware/Media 2- Ethernet/LAN – Mac addresses 3- WAN – IP 4- TCP Ports 5- SSL These network layers are just 1 example of how we can build test cases to break out the security testing by architectural layer. The tip of the iceberg! Add on services include DHCP and VPN.
119 Protocols SSL, TLS, PCT – session layer 2 sided (both c and s must be configured) S-HTTP – application layer IPSec – network or IP layer (implemented in routers/switches) Microsoft owns PCT (Private Communications Technology) SSL on 443 needs client AND server authentication on the IP Stack
120 NETWORK Firewalls – catch all rule: everything not previously allowed is explicitly denied Router based (Packet filtering) at IP level Headers inspected based on port, protocols, and destination/source IP addresses Proxy based (gateways) More secure: software on the perimeter Proxy server interacts with internet and extensively logs traffic Can be used in combo if a proxy fails May be a performance cost XP comes built in with it’s own firewall (installed by default but needs to be manually turned on). Internet Connection Firewall (ICF). A bastion Host is a hardened firewall put on your external perimeter like a boundary router Proxy agents like DNS, FTP, LDAP etc require AUTHENTICATION (extra protection since network layer can be spoofed).
121 Router Tools: Lancope StealthWatch Watch abnormal traffic patterns Monitor bandwidth spikes Routers should encrypt data & authenticate one another for traffic exchange Test the Routers Built-in Filters that set limits on which IP’s can be used on other ISP networks
122 Network Scanning Tools NAI’s Cybercop 5.5 : Network Discovery: Ping scans, OS identification, TCP and UDP port scan, password guessing, SNMP data capture, limited app banner grabbing, limited packet sniffing, limited remote control software, no modem testing For UNIX: tests Trusted Host, TFTP, FTP/Anonymous FTP,Finger,NFS,NIS, Xwindows,Sendmail For Windows: ,Anonymous Null access (IPC$), unprotoected Registry Elements, Windows SMB File shares, Limited NT Service Pack level detection, no Netware or Vax vulnerabilities Web Security: Http server vulnerabilities, web browser vulnerabilities, firewall/router, router product, limited firewall product, DOS warnings and vulnerabilities Product Admistration Analysis and Fix Guidance, Scripting to add new scans,selectable tests, no scheduled scanning like CISCO secure scanner,customizable reports, product update, unlimited IP address ranges (ISS has a limit and CISCO is limited by # of hosts).
123 Example – CyberCop Scanner
124 DMZ Small network/host between private and outside public networkSeparated by another packet filter Does not initiate any inward connections- no access to hosts within private network Open subnet -> router -> proxy -> router -> internal network (good for web-commerce with SSL) Testing should be done outside the network perimeter as well as inside Most common firewall environment – network connecting 2 firewalls
125 DMZ- a example
126 VPN Remote users dial into local Point of Presence to connectProvides private encrypted tunnel through public internet space -app IPSec, PPTP, L2TP Your firewall encrypts the data then forwards it to remote VPN gateways. Decrypts it and passes it on. Your intranet has no external connectivity while your Extranet is more like a B2B intranet using a VPN.
127 Cerebus Internet Scanner(NT/2000-free tool Test points of failure, screen architecture, backdoors, holes Modem scan in commercial version
128 www.whois.net Social Engineering: phone numbers/contactsDMZ Network Address targets Backdoors Even internal network address disclosures DNS Server targets Ebay just recently had a redirect scam using FormMail. An was sent to you to tell you your order was cancelled and to validate it at a redirected URL.
129 WEB Vulnerabilities HTML – run as nobody – fork from root (binds to 80) JAVA – signed applets Jscript/VBScript – not in a sandbox Active X – signed script policy CGI, ASP, PHP, SSI Ebay just recently had a redirect scam using FormMail. An was sent to you to tell you your order was cancelled and to validate it at a redirected URL.
130 Host/Network IdentificationIpconfig /all Nslookup Nbtstat Net use Netstat –s 5 (intervals stats every 5 seconds) oracle.com Unbreakable? LANGUARD: DNS Lookup, Enumerate, Traceroute, New Scan
131 Viruses and Worms Worms: self-propagatingTransport mechanism for other apps Viruses: infect another program by replicating itself onto the host : Testing Anti-Virus Hoaxes: or
132 Password Cracking Dictionary & Brute Force attacksDon’t leave passwords in memory- empty arrays may be visible in core dumps Disable emulators (telnet) that could show passwords in clear text : sqlplus Limit the lifetime Brute Force uses Guessing Algorithms. Dicationary uses any type of dictionary (pop culture for example to crack). LC3 uses password auditing and is automated How long it can take to crack: L0phtCrack can try over 450,000 guesses per second (NT). A 4 digit numeric (pin for example) can be cracked in .02 seconds. Use mixed case, alphas, special characters with variable length – can take 430 years
133 Valid Remote Apps vs RogueCarbon Copy, iCloseup, CoSession, ControlIT, Laplink, PCAnywhere, Reachout, Timbuktu, VNC VS. Back Orifice,Girlfriend,NetBus,PhaseZero, Sockets de Troi,Stacheldracht,SubSever,Trin00 DDoS Agent PORT OF CALL…….next ->
134 Port List Echo 19 chargen 20 FTP data 21 FTP Control22 SSHD secure shell 23 Telnet 25 SMTP service listens on 37 TIME (tcp/udp) 45,46,47 Page II 53 DNS Zone Transfers (tcp/udp) 66 SQL*NET 67,68 DHCP/bootstrap protocol server 69 Trivial file transfer 70 Gopher 79 fingerd httpd Web servers 98 LinuxConf Port List Note if the port is using TCP or UDP UDP – user datagram uses connectionless state for lookups - faster TCP – transmission control – domain name server for zone transfers- connections
135 POP2/POP3 111/2049 RPC tcp/udp portmap & rpcbind 119 NNTP for newsgroups 123 NTP NBT/NetBIOS in NT tcp/udp 139 NetBIOS Session Service tcp 143/220 IMAP SNMP 161/UDP 179 BGP (tcp) 194/529 IRC 389 LDAP 443 SSL 445 Microsoft CIFS (TCP/UDP) ; Windows2000 uses for NetBIOS /TCP Berkley r commands: login,rexec,rsh 514/UPD Syslog Unix: LDP (local print daemon) - can have a buffer overflow- turn off /etc/inetd.conf MIT Kerberos SWAT – Samba admin
136 ports above 1024 do not have to run as root for DNS:1080/tcp SOCKS 1352 Notes Remote Protocol NRPC 1521 /etc/services: {oracle listener-name} NFS 2301 Compaq Insight Manager 4045 lockd 5190 AIM X Windows 7777 Apache web server HTTP 8888 Netscape default Admin Server RCP Loopback ports - Unix; remote procedure call vulnerable for buffer overflows IIOP
137 More Tools…. AW Security Port Scanner Network File SharesSoftware Banner Grabbing : telnet qasecure.com Trace Routes/Hops Packet Sniffers Check out for templates, articles, and test tools
138 Other Technologies Biometrics Wireless/ 802.11b Smart Cards TokensGlobal Positioning
139 Policy Tying it together with cross-team buy-inYour company’s security team (NOT the software testing team alone) determines policy on user access, time outs, content availability, database viewing, system protection, security tools etc. As a team we need to document and model our structures, flows, dependencies, and protocols. The role of the test group is test the existing system to look for errors in security implementation, primarily at the application level. Gather configuration issues for the tech support knowledge base. IT is generally responsible for network security, firewall testing, packet counting, traffic monitoring, virus protection, and server-break in testing. They would install IP address screening policies. Bridging the testing Principles together: security is not ad hoc/plug and play. After the fact patching is better than not at all but much more difficult than test being involved in the design of the system. Policy should take into account liability. Make sure you do a risk analysis: determine the IMPACT vs. the COUNTERMEASURES you already have in place. Tying it together with cross-team buy-in
140 Test Objective In Web ApplicationForm Element Input/Output Control XSS Path Travelling URL Redirection SQL Injection Java script in Source Code Pre-fill Information Leakage Implement Strategy AA Bypass Account Brute Crack Encrypting Not Enforced Credential Theft
141 XSS Testing Cross-Site Scripting harm Cross-Site Scripting Normal XSS Cross-Site Scripting Normal XSS Inject scripting via HTTP Post Inject scripting via HTTP Get Stored XSS
142 Security test tool Paros - security tool for web application vulnerability assessment Klocwork delivers the most comprehensive source code analysis solution. HttpWatch is an HTTP viewer and debugger that integrates with IE and Firefox to provide seamless HTTP and HTTPS monitoring without leaving the browser
143 Applying Your Testing Skills Web Site TestingChapter 14 Applying Your Testing Skills Web Site Testing
144 Highlight Web Page Fundamentals Web site Testing knowledge Web Page Fundamentals Web site Testing knowledge Black-Box Testing Gray-Box Testing White-Box Testing Configuration and Compatibility Testing Usability Testing Introducing Automation
145 What is a Web Page? A web page comprises: A web page comprises: A layout A framework of instructions Links to related files Content (words, graphics, sounds, media) In other words: A lot of files! The main file is the .htm file (default.asp, .php, …) – this is the framework which holds the codes, links, and content Each graphic (both nice pictures or a simple line) is a file called up by the .htm file Each link connects to another file
146 Example
147 Web Page Organization Standards
148 Web Page Fundamentals-Example Input data field Yahoo.WebEx.Com Button
149 Web Site Testing Web page content - Content text, and spelling Web page content - Content text, and spelling - Different sizes, fonts, colors, … - Graphics, photos, UI Functionality - Links, Buttons, Navigate bar, … - Log in/ Log out, Cookie , Session, … - Logic, …, Various operation Usability Security Performance
150 Technologies in Web page HTML/DHTML/XML JavaScript ,Java VBScript ActiveX, Plug-in Perl, CGI PHP/ASP/JSP
151 Black-Box Testing Text, ATL text ( Alternate ) Hyperlinks Graphics Text, ATL text ( Alternate ) Hyperlinks Graphics Forms Objects and other Simple Miscellaneous Functionality
152 Gray-Box Testing Gray-Box Testing (or translucent-box), it is the effective combination of black box (external) and white box (internal or unit) testing. Test the software as a black-box, but you supplement the work by taking a peek( not a full look, as in white-box testing) at what makes the software work. var lmonth=months[time.getMonth() + 1];
153 White-Box Testing Precondition: Then: [email protected]Need have some knowledge of the Web site’s system structure; Need have some knowledge of programming Then: Dynamic Content Database Driven Web pages Programmatically Created Web Pages Server performance and Loading Security
154 Compatibility Testing Is performed to ensure that a site performs as intended across multiple operating system and browser configurations. Ensure certain coding such as Active X, Java, Javascript and CGI, function properly across multiple configurations.
155 Configuration and Compatibility Testing - Hardware platform - Multiple Operating Systems - Multiple Browser Compatibility: IE 4.0, IE 5.0, IE 6.0, NS 4.5, NS 4.7, NS 6.0 - Connection Speed: Dial 64K, ADSL, T1 or LAN - Browser Options: Disable ActiveX in IE, plug-Ins - Java Machine Versions in Mac OS 9/OS X - SSL w/t proxy, SSL w/o proxy - Monitor Resolution, Scrolling, Text size, Compatibility Testing - Forward Compatibility - Backward Compatibility
156 Network Testing Tests need to be run to ensure that a product is networkable The product should also be tested to make sure it can be run on several network OS environments. Also stress tests is needed to ensure that a minimum number of users can use an application at one time
157 Usability Testing Ease-of-Use vs. Visual Virtuosity Ease-of-Use vs Visual Virtuosity Buttons are buttons Menus are menus Breadcrumb trails (线性回溯 ) Clear compatible colours Stick to Web-Norms Splash Page Matt-pastel colours (Flash) graphics, (Shockwave/Real) sounds, (Quicktime) movies Unusual designs
158 Top Ten mistakes in Web Design Gratuitous Use of Bleeding-Edge Technology Scrolling Text, marquees and Constantly Running Animations Long Scrolling Pages Non-Standard Link Colors Outdated Information Overly long Download Times Lack of navigation Support Orphan Pages Complex Web Site Addresses ( URLs) Using Frames
159 Practice in Web Design Page Width Colors Typeface Page Width Colors Typeface Page Length /Scrolling Navigation (bars, buttons, image maps) Page Names (description)
160 Introducing Automation Organization of Web Test Tools Listing Load and Performance Test Tools Java Test Tools Link Checkers HTML Validators Free On-the-Web HTML Validators and Link Checkers PERL and C Programs for Validating and Checking Web Functional/Regression Test Tools Web Site Security Test Tools External Site Monitoring Services Web Site Management Tools Log Analysis Tools Other Web Test Tools See file:///E:/Trainging%20Package/Specilization/Tool/Web%20Test%20Tools.htm#LOAD
161 Rational Site Check Internet and intranet website testing Internet and intranet website testing Checks for broken links, structure etc Can deploy website to target destination using Site check
162 Example
163 GUI Record and Playback Capture the keystrokes, input, and responses as tests being run, and compare expected with actual outcome; Generate script records when capturing Microsoft: Visual Test Suite Rational: Visual Test and TeamTest Mercury Interactive: XRunner, WinRunner Compuware Corporation : QARun WinRunner SilkTest Robot
164 Script example [email protected]web_browser_invoke(IE, "http:// /mc2/"); wait(2); # menu set_window("menu",124); web_image_click("hostameeting2", 32, 7); web_image_click("ScheduleMeeting2", 30, 4); # main set_window("main",5); edit_set("userName","test"); password_edit_set("password","9c28757eca2b3741"); button_press("Log In"); web_sync(8); edit_set("ConfName","WebEx Test"); button_press("Start Meeting"); …… while(win_exists("WebEx Meeting Manager - WebEx Test", 20) != E_OK){ wait(2); } #set_window ("WebEx Meeting Manager - WebEx Test", 30);
165 Load Test The purpose is to simulate thousands of users concurrently visiting and interacting with web site There are Tools such as WebLoad that allows tester to provide different scripts to simulates the stress/load testing. Indicates where problems are occurring. Indicates how many concurrent users your site can handle before response times become unacceptable
166 WebLoad 3.0.1
167 WebLoad Default Report
168 WAS ( Web Application Stress)
169 Test Tools - JMeter Why JMeter (http://jakarta.apache.org/)?1. It is free tool. 2. It is open source. 3. Its basic function is powerful. 4. It is easy to use and program. 5. 3-rd part support from BadBoy (Recorder) tool.
170 Test Tools – JMeter Basic Elements: Test Plan Thread GroupLoop Controller Logic Controller Sampler Timer Listener Pre Processors Config Element Post Processors
171 Test Tools – JMeter
172 Test Tools Market Share (Newport Group, Inc)
173 Load Testing Tools (From Abraham Jacob)Microsoft's web application stress tool Cyrona's OpenSTA 1 Quest Software's benchmark factory 4 Empirix's E-Test Suite 6.0 6 RadView's WebLoad 5.0 7 IBM Rational Robot HP Mercury LoadRunner 8 Compuware's QA Load 4.7 10 Segue Software's SilkPerformer 5.0 10+ Best
174 Test Tools – ComparisonAdvantages Disadvantages WebLoad 1. easy to learn 2. can generate proper report 3. can record the test script automatically 1. License is not free 2. could not capture program error effectively Robot VU 1. can generate sophisticated report 2. can use C or Java to write the test script 1. license is not free 2. very hard to analysis the test result 3. the longer learning curve JMeter 1. License free/open source 2. Can use third-party software to record the test script. 3. can check the every request and response pair more in detail 1. Could not generate sophisticated report. 2. some bugs on itself
175 Uptime Monitoring ToolUptime requests a page from your server every 15 minutes. If the site is unreachable, Uptime sends you . Uptime will continue checking your site. When it becomes reachable again, Uptime will send you one more message
176 Server Uptime Monitoring ToolIt doesn't really matter how good your site is if people can't rely on its being there.
177 Q & A
178 Exercise Page 227: 3., 4., 5.