1 Are you in the right course?March 28, 2017 Are you in the right course? SE 433/333 Software Testing & Quality Assurance March 28, 2017 SE 433: Lecture 1 Lecture 1

2 SE 433/333 Software Testing & Quality AssuranceMarch 28, 2017 SE 433/333 Software Testing & Quality Assurance Dennis Mumaugh, Instructor Office: CDM, Room 428 Office Hours: Tuesday, 4:00 – 5:30 March 28, 2017 SE 433: Lecture 1 Lecture 1

3 Administrivia: IntroductionSE 433 March 28, 2017 Dennis Mumaugh Undergraduate: BSEE – University of California, Berkeley MS Computer Science – University of Maryland Ph.D. Studies – University of Maryland Teaching at DePaul since September 2000 Work Senior Engineer – National Security Agency ARPANet Pioneer, Unix™ Technology Transfer Member of the Technical Staff – Bell Labs/Lucent Technologies Unix Development – Current Engineering IS&R Systems – Knowledge Based Systems Software Tools and OO Technology Interests Operating Systems and System Programming Software Productivity, Compilers and Software Metrics Software Engineering March 28, 2017 SE 433: Lecture 1 Lecture 1

4 Administrivia: Basic InformationSE 433 March 28, 2017 Contact Information: Phone: (10:00 am - 11:00 pm) except just before class (After 3pm) Office Hours Tuesday: 4:00 pm to 5:30 pm, Loop, CDM, Room 428 By arrangement March 28, 2017 SE 433: Lecture 1 Lecture 1

5 Administrivia: Basic InformationSE 433 March 28, 2017 Class home page contains syllabus and schedule, lecture notes, homework, more reading material About the Lecture Notes – look at “notes” section of the slides Also look at the expanded readings page: Desire2Learn: Course materials, assignments, assignment submissions, assignment solutions, examinations and quizzes and grades will be available on the Desire2learn – https://d2l.depaul.edu/ All students are expected to have an address. Please make sure it is valid and make sure Campus Connection has the current address. March 28, 2017 SE 433: Lecture 1 Lecture 1

6 Administrivia: Basic InformationSE 433 March 28, 2017 Course mailing list: To subscribe to the list or unsubscribe from it, go to I’ll bulk subscribe on Saturday. If necessary, update your spam filter to accept messages from the mailing list. Unless your message is personal, send it to the course mailing list! I will respond to questions related to the lectures and assignments. You are also encouraged to respond to other students’ questions. The mailing list is archived. You may subscribe to a digest. Last minute information will go to the mailing list March 28, 2017 SE 433: Lecture 1 Lecture 1

7 Administrivia: reading materialsSE 433 March 28, 2017 Primary Textbook Software Testing and Analysis: Process, Principles and Techniques, by Mauro Pezze and Michal Young References The Art of Software Testing, Second Edition by Glenford J. Myers et al Available in e-book through library Software Engineering: A Practitioner's Approach, 6th ed by Roger S Pressman (Chapters 13 and 14) Lecture notes and supplementary materials Available on D2L March 28, 2017 SE 433: Lecture 1 Lecture 1

8 Administrivia: reading materialsSE 433 March 28, 2017 A note on reading list You are not expected to read all of the material on the reading list. Look at the various articles as you have time. Many say the same thing but with a different perspective. Don't get overwhelmed in reading March 28, 2017 SE 433: Lecture 1 Lecture 1

9 Administrivia: Course structureMarch 28, 2017 Ten classes + Midterm Exam + Final Exam Weekly reading, assignments (10) Class structure: lecture (with short break near the middle). Topics and reading assignments are on the class web page March 28, 2017 SE 433: Lecture 1 Lecture 1

10 Administrivia: required softwareSE 433 March 28, 2017 Access to MS Word and PowerPoint (or equivalent) Java Java development and testing software: Eclipse, JUnit, Ant, Cobertura. March 28, 2017 SE 433: Lecture 1 Lecture 1

11 Administrivia: SupportSE 433 March 28, 2017 Technical questions can be addressed during office hours or by Use the mailing list for all technical questions Provide appropriate support to each other I do not preview homework, but I will answer questions or make suggestions to generic problems If you contact me by Please include your name and the course number in all correspondence March 28, 2017 SE 433: Lecture 1 Lecture 1

12 Administrivia: MiscellanySE 433 March 28, 2017 Communications development: An essential part of this course is communicating your ideas in prose, as well as in technical graphical artifacts. The ability to communicate clearly and effectively is a skill that will pay off both in and out of class. Motivation from a recent NPR business report: Robert Half surveyed their corporate customers concerning resumes they had received. Corporate reviewers spent about seconds deciding whether to examine the resume further. They instantly tossed the resume if they detected any grammatical or spelling errors. Treat your coursework as if it were being reviewed by the manager who does your performance review and sets your salary. March 28, 2017 SE 433: Lecture 1 Lecture 1

13 Administrivia: MiscellanySE 433 March 28, 2017 Intellectual property issues: All material in this course is property of either the instructor or Xiaoping Jia. You are permitted to download and print copies of the material. You are not permitted to redistribute the material in any form. Plagiarism: All individual assignments must represent your own work. It's a great idea to get together in study groups to discuss the problems, but you should then do the assignments individually. Plagiarism is to take and use as one’s own, or copy without acknowledgement, the works of another person. The provider of such material can be ruled equally culpable. If you hand in late homework with prior permission, it must be your own work, not a copy of the solutions presented in class. March 28, 2017 SE 433: Lecture 1 Lecture 1

14 Administrivia: MiscellanySE 433 March 28, 2017 There will be a lot of ambiguity and lack of firm direction in the assignments and the information. That is typical of much of engineering. This requires you to provide your own experience. Or to research and discover your information. Using previous solutions/techniques will not necessarily work. You may have to invent something. Understanding a problem (statement): An essential part of this course is understanding written material, ideas in prose. The ability to understand a document, to "read between the lines", is a skill that will pay off both in and out of class. Areas of concern: Projects are implemented with the wrong features. Students answer the wrong question on a test. People confuse the subsystems of a design. People do not read, really read things. March 28, 2017 SE 433: Lecture 1 Lecture 1

15 Administrivia: Feedback/ParticipationSE 433 March 28, 2017 Feedback/Participation Share your thoughts Ask questions – wave your hand forcefully to get my attention Speak loudly so all can hear Give me verbal and non-verbal feedback Don't just sit there nod, smile, frown, shake your head Make sure your address is correct and works March 28, 2017 SE 433: Lecture 1 Lecture 1

16 Homework logistics SE 433 March 28, 2017 Homework must be submitted via Desire2Learn by 11:59 PM Chicago Time on the assignment due date. Do not rely on the deadline, clocks differ. Be early. Submit MS Word or Adobe PDF files only [I accept either format of MS Word] No extra credit assignments. Late Policy Late assignments will not be accepted except for special circumstances. March 28, 2017 SE 433: Lecture 1 Lecture 1

17 Grading – Attendance & AssignmentsIn-class students: attendance taken during class On-line students: must watch each lecture within 6 days Submit attendance confirmation Assignments and projects (individual) SE333 – 50% SE433 – 45% Including several programming projects March 28, 2017 SE 433: Lecture 1

18 Grading – Exams & Final PaperMid-term exam – 20% Final exam – 20% Final paper – 5% (SE433 only) No late submission will be accepted. March 28, 2017 SE 433: Lecture 1

19 Administrivia: assessmentMarch 28, 2017 Regular assignments (10) Midterm examination Final examination Each of the above will be weighted as follows: Attendance – 10% Assignments and projects – 50% for SE % for SE433 Mid-term exam – 20% Final exam – 20% Final take-home exam, a short paper – 5% SE433 only, Grading will be done on the usual 60/70/80/90 bands but will be adjusted to account for clustering and banding of scores. Bands may be adjusted if there seems to be a systemic bias to the scores. March 28, 2017 SE 433: Lecture 1 Lecture 1

20 Assignments Read the relevant chapters in the textbook and the relevant reference materials. Understand all the materials covered in the lecture notes. Use the mailing list for questions and discussions. March 28, 2017 SE 433: Lecture 1

21 Course Specific InformationMarch 28, 2017 SE 433: Lecture 1

22 Prerequisites Data Structures II (CSC 301, 403, 383, 393), orObject-Oriented Modeling (SE430) Proficient in programming and data structures Assignments will use Java Familiar with object-oriented modeling and principles Familiar with software development processes March 28, 2017 SE 433: Lecture 1

23 Course Objectives Understand how to achieve high quality in software development Study the processes, techniques, and tools for software testing Focus on software testing and analysis basic principles and techniques underlying theory organizational and process issues March 28, 2017 SE 433: Lecture 1

24 Learning Outcomes Applications of techniques and tools at the module and system levels Understanding of the techniques, processes, and tools at the project and organization levels Emphasis practical techniques to achieve an acceptable level of quality at an acceptable cost. realistic strategies for reliable and cost-effective software testing. March 28, 2017 SE 433: Lecture 1

25 Some Topics We will cover a subset of the above.Introduction to software testing Functional testing Structural testing Test case selection Inspection Static analysis Unit testing Test automation and tools Integration and system testing Regression testing Performance testing Web application testing Security testing Graphical user interface (GUI) testing Usability testing Fault-based testing Planning and monitoring the software quality process Testing of object-oriented software We will cover a subset of the above. March 28, 2017 SE 433: Lecture 1

26 Engineering MethodologySE 433 March 28, 2017 What is engineering and how does it differ from science? Engineering is the application of science to the needs of humanity. This is accomplished through the application of knowledge, mathematics, and practical experience to the design of useful objects or processes in a cost effective way. One of the objectives of this course is to teach some of the methodology of engineering Understanding the problem How to approach a problem Developing and following requirements Writing and reading specifications Analyzing a problem Designing a solution Verifying each step Testing the design against the original problem statement March 28, 2017 SE 433: Lecture 1 Lecture 1

27 Surviving SE433 SE 433 March 28, 2017 Make sure you read things, sometimes more than once. People do not seem to read assignments and web pages (or do not follow instructions). The pace is quick. The intent of the course is to learn by doing, not by reading; but there is a lot of information to process. Read the assignments carefully. Note special requirements, such as formats and use of predefined templates! Start your assignments right after they are handed out (assigned). They will take some time and starting on the night before it is due is not a good strategy. Reading list: Is it required? No. Is it useful? Yes, especially if you are serious about a career in software development. The articles are usually short but informative. Most are supplemental – useful for understanding but the notes cover the major points. Reading should be done in parallel with the lectures. Pace yourself. Remember: “This too shall pass.” March 28, 2017 SE 433: Lecture 1 Lecture 1

28 SE 433 – Class 1 Topics: Introduction and Overview: ReadingMarch 28, 2017 Topics: Introduction and Overview: Class Logistics and Administrivia;  Introduction to Software Testing Reading Pezze and Young: Chapters 1-4 March 28, 2017 SE 433: Lecture 1 Lecture 1

29 SE 433 Software Testing takes at least twice as long as estimated.March 28, 2017 Thought for the Day Software Testing takes at least twice as long as estimated. March 28, 2017 SE 433: Lecture 1 Lecture 1

30 On Testing For the Snark’s a peculiar creature, that won’tSE 433 March 28, 2017 On Testing For the Snark’s a peculiar creature, that won’t Be caught in a commonplace way. Do all that you know, and try all that you don’t: Not a chance must be wasted to-day! — Lewis Carroll The Hunting of the Snark, an Agony in Eight Fits (1876) March 28, 2017 SE 433: Lecture 1 Lecture 1

31 Why Programs fail In Federalist 51, James Madison wrote: “If men were angels, no government would be necessary.” If he lived today, Madison might have written: “If software developers were angels, debugging would be unnecessary.” March 28, 2017 SE 433: Lecture 1

32 Why Programs fail Congratulations!Your code is complete. It compiles. It runs … Your program fails. How can this be? There is a defect in the code. When the code is executed, the defect causes bad behavior, which later becomes visible as a failure. Before a program can be debugged, we must set it up such that it can be tested — that is, executed with the intent to make it fail. The first step in debugging is to reproduce the problem in question — that is, to create a test case that causes the program to fail in the specified way. The first reason is to bring it under control, such that it can be observed. The second reason is to verify the success of the fix. Now you will have the fun of testing and debugging! March 28, 2017 SE 433: Lecture 1

33 Myth Busters Software TestingMarch 28, 2017 SE 433: Lecture 1

34 Myth #1 in Software TestingQ: What is the objective of software testing? A: Testing is to show that there are no errors/bugs/defects in the software. March 28, 2017 SE 433: Lecture 1

35 Myth #1 in Software TestingQ: What is the objective of software testing? A: Testing is to show that there are no errors/bugs/defects in the software. Fact: No!! The main objective of testing is to discover defects. Testing is a destructive activity. March 28, 2017 SE 433: Lecture 1

36 Myth #2 in Software TestingQ: What is the objective of software testing? A: Testing is to ensure that the software does what it is supposed to do. March 28, 2017 SE 433: Lecture 1

37 Myth #2 in Software TestingQ: What is the objective of software testing? A: Testing is to ensure that the software does what it is supposed to do. Fact: Only partly true. Testing is also to ensure the software does not do what it is not supposed to do. March 28, 2017 SE 433: Lecture 1

38 Myth #3 in Software TestingQ: How challenging is software testing? A: Testing is easier than design and implementation. Though it’s simple to prepare straightforward test cases, at times testing can be a real challenging task. Any production issues will, in many cases, backfire first to the testing teams. Why was this scenario not covered in the test plan?? Therefore, a Tester should develop the capability to look or think beyond the requirements mentioned in the test plan or specifications. This is very important in case of System Testers who are responsible for ensuring that the software product works appropriately from "end-to-end ". March 28, 2017 SE 433: Lecture 1

39 Myth #3 in Software TestingQ: How challenging is software testing? A: Testing is easier than design and implementation. Fact: Must consider all possible scenarios. Implied and unstated requirements and threats. Must be imaginative and creative. Though it’s simple to prepare straightforward test cases, at times testing can be a real challenging task. Any production issues will, in many cases, backfire first to the testing teams. Why was this scenario not covered in the test plan?? Therefore, a Tester should develop the capability to look or think beyond the requirements mentioned in the test plan or specifications. This is very important in case of System Testers who are responsible for ensuring that the software product works appropriately from "end-to-end ". March 28, 2017 SE 433: Lecture 1

40 Myth #4 in Software TestingQ: How challenging is software testing? A: Testing is an extremely creative and intellectually challenging task. March 28, 2017 SE 433: Lecture 1

41 Myth #4 in Software TestingQ: How challenging is software testing? A: Testing is an extremely creative and intellectually challenging task. March 28, 2017 SE 433: Lecture 1

42 What is Software Testing ?Software testing is the process of executing a program with the intent of finding bugs. March 28, 2017 SE 433: Lecture 1

43 Cost of Software ErrorsEstimated 50% of each software project spent on testing (spans from 30% to 80%) Very rough approximation money spent on testing + cost of remaining errors = 66% of size of software industry Remark: At Ericsson: 35% of code is test cases! March 28, 2017 SE 433: Lecture 1

44 Failure and SpecificationSome failures are obvious obviously wrong output/behavior non-termination Crash freeze ...but most are not! In general, what constitutes a failure, is defined by a specification! Correctness is a relative notion — B. Meyer, 1997 March 28, 2017 SE 433: Lecture 1

45 Why Does Software Testing Matter?March 28, 2017 SE 433: Lecture 1

46 Economic Impact NIST, “The Economic Impacts of Inadequate Infrastructure for Software Testing” Inadequate software testing costs the US alone around $59.5 billion annually Better approaches could cut this amount by $22.2 billion We want our programs to be reliable Testing is how, in most cases, we find out if they are March 28, 2017 SE 433: Lecture 1

47 Launch of MS Windows 98 Bill Gates and Blue Screen of DeathMarch 28, 2017 Bill Gates and Blue Screen of Death March 28, 2017 SE 433: Lecture 1

48 Launch of MS Windows 98 Bill Gates and Blue Screen of DeathMarch 28, 2017 Bill Gates and Blue Screen of Death March 28, 2017 SE 433: Lecture 1

49 Why Do We Test? Testing is expensive. What do we gain from that cost?12/18/2017 Testing is expensive. So are failures! What do we gain from that cost? Finding bugs Leading to Fixing bugs Raising the quality of the program or system we are testing March 28, 2017 SE 433: Lecture 1 49

50 Why Do We Test? Kinds of testing “Smoke test.”, aka “Hello World!”12/18/2017 Kinds of testing “Smoke test.”, aka “Hello World!” Check handling of inputs Illegal inputs – text instead of integers Impossible inputs – floating vs. integer Outrageous values – infinities, max and min values Not in domain – negative numbers, values outside specifications Input errors – wrong input, e.g. mis-spellings Stress test multiple inputs without restart run program for long periods of time March 28, 2017 SE 433: Lecture 1 50

51 A Case Study: Ariane 5 Launch VehicleEuropean expendable launch system Double the payload capacity of its predecessor Ariane 4 113 successful launches, 3 failures Flight 501, maiden launch, June 4, 1996,12:34pm The Guiana Space Centre or, more commonly, Centre spatial guyanais (CSG) is a French and European spaceport near Kourou in French Guiana. March 28, 2017 SE 433: Lecture 1

52 Case Study  Ariane 5: What Happened?T0 - T0 + 36s : normal Within SRI 2: BH (Bias Horizontal) > 215 convert_double_to_int(BH) fails! exception SRI -> crash SRI 2 & 1 OBC disoriented angle > 20°, huge aerodynamics constraints boosters separating... T0 + 39s: self destruction cost: € 500M March 28, 2017 SE 433: Lecture 1

53 Case Study  Ariane 5: Why Did It Happen?Not a programming error unprotected conversion = design decision (~1980) Not a design error Justified against Ariane 4 trajectory & RT constraints Problem with integration testing Theoretically detectable. But huge test space vs. limited resources Furthermore, SRI useless at this stage of the flight! Reuse of a component with a hidden constraint Precondition : abs(BH) < Valid for Ariane 4, but no longer for Ariane 5 More powerful rocket March 28, 2017 SE 433: Lecture 1

54 Case Study  Ariane 5: Lessons Learned in Software Eng.Test! Test! Test! Test! Even when the code is reused. When reuse, ensure the assumptions are still valid. When write reusable code, document the assumptions. Write fail-safe code. Do not propagate errors. March 28, 2017 SE 433: Lecture 1

55 Trade-Offs of Cost and FailuresTotal Cost of Quality (CoQ) = Cost of Conformance (CoC) + Cost of Non-Conformance (CoNC) Cost of Conformance Prevention: quality planning, investment in tools, quality training Appraisal: testing, inspection Cost of Non-Conformance Internal failures: rework External failure: liability, loss of properties, loss of lives March 28, 2017 SE 433: Lecture 1

56 Trade-Offs of Cost and FailuresTesting adds to Cost of Conformance It must directly reduce Cost of Non-Conformance March 28, 2017 SE 433: Lecture 1

57 A Self Assessment Test March 28, 2017 SE 433: Lecture 1

58 Test the Following ProgramThe program reads in 3 integer values that represent the lengths of the sides of a triangle. The program prints a message that states whether the triangle is Equilateral (all 3 sides are equal) Isosceles (exactly 2 of the 3 sides are equal) Scalene (all 3 sides are of a different length) Write a set of test cases that you feel would adequately test this program. March 28, 2017 SE 433: Lecture 1

59 How Did You Do?

60 Test the Following ProgramWrite a set of test cases that you feel would adequately test this program. How many cases are needed (look back at slide 50)? How do you plan to test the program. Run it many times with different inputs. Once and manually enter the values. A file with a set of lines with test values. March 28, 2017 SE 433: Lecture 1

61 Assignment Assignment 1 Part I due April 4, 2017SE 433 March 28, 2017 Assignment 1 Part I due April 4, 2017 Part I: Write a Java program to determine types of triangles. The program reads 3 integer numbers from the standard input. The numbers represent the lengths of the sides of a triangle. The program prints a message to the standard output that indicates whether the triangle represented by the input is an equilateral (all 3 sides are equal), or an isosceles (exactly 2 of the 3 sides are equal), or a scalene (all 3 sides are of different lengths) The program should print out appropriate messages when an erroneous input or condition is encountered. Submission: The source code (i.e., the .java file) of your program. Hint: Your program must read the numbers from the standard input, or a file fed to the standard input. One way to do this is to use the java.util.Scanner class. Part II: Due date: April 11, 2017, Design a test suite to thoroughly test your program. And, Test it. March 28, 2017 SE 433: Lecture 1 Lecture 1

62 Software Testing TerminologiesMarch 28, 2017 SE 433: Lecture 1

63 The Very First Software BugA moth found trapped between points at Relay # 70, Panel F Mark II Aiken Relay Calculator Harvard University September 9, 1945. On display in Smithsonian March 28, 2017 SE 433: Lecture 1

64 Failures Failures are deviation of the observed behavior of a system from its specification, i.e., its expected behavior. Failures can only be determined with respect to the specifications. Failures are concerned with the observed behavior and outcome of the system. March 28, 2017 SE 433: Lecture 1

65 Defects Defects are flaws in a system that can cause the system to fail to perform its required function e.g. an incorrect condition or statement. Defects are concerned with specific parts or components of the system. Defects are synonymous with faults, bugs. March 28, 2017 SE 433: Lecture 1

66 Errors Errors are human actions that result in a fault or defect in the system. Errors are concerned with the underlying causes of the defects. Errors are synonymous with mistakes. March 28, 2017 SE 433: Lecture 1

67 The Relations among Failures, Defects, and ErrorsA human being makes an error (mistake) can occur in design, coding, requirements, even testing. An error can lead to a defect (fault) can occur in requirements, design, or program code. If a defect in code is executed, a failure may occur. Failures only occur when a defect in the code is executed. Not all defects cause failures all the time. Defects occur because human beings are fallible Failures can be caused by environmental conditions as well. March 28, 2017 SE 433: Lecture 1

68 The Relations among Failures, Defects, and ErrorsThe terms error, failure and defect have different meaning when testing. Especially in using JUnit. In this case: Test Case Verdicts Pass The test case execution was completed The function being tested performed as expected Fail The function being tested did not perform as expected Error The test case execution was not completed, due to an unexpected event, exceptions, or improper set up of the test case, etc.  March 28, 2017 SE 433: Lecture 1

69 Failures vs. Defects: A Simple ExampleFor any integer n, square (n) = n*n. int square (int x) { return x*2; } square (3) = 6 A failure A defect March 28, 2017 SE 433: Lecture 1

70 Failures vs. Defects: A Simple ExampleFor any integer n, square (n) = n*n. int square (int x) { return x*2; } square (2) = 4 A defect Correct result Not a failure March 28, 2017 SE 433: Lecture 1

71 Software Testing Software testing ≠ Debugging.Software testing is the process of executing a program (or parts of a program) with the intention of finding defects The purpose of testing to find defects. to discover every conceivable weakness in a software product. Software testing ≠ Debugging. Software testing ≠ Quality assurance March 28, 2017 SE 433: Lecture 1

72 Software Testing vs. Quality Assurance (QA)Testing is necessary, but not sufficient for quality assurance Testing contributes to improve quality by identifying problems. Quality assurance sets the standards for the team/organization to build better software. March 28, 2017 SE 433: Lecture 1

73 Test Cases and Test SuitesMarch 28, 2017 Test case A test case consists of inputs, steps/actions, and expected results, i.e., pass-fail criterion Test suite A group of test cases (usually organized around some principles) March 28, 2017 SE 433: Lecture 1

74 Test Plan and Testing ProcessMarch 28, 2017 Test plan A document that specifies how a system will be tested, including criteria for success, i.e., the exit criteria. Testing process The testing process involves developing test plans, designing test cases, running the test cases, and evaluating the results March 28, 2017 SE 433: Lecture 1

75 Techniques for Software TestingMarch 28, 2017 SE 433: Lecture 1

76 Testing Testing “Phases” Testing Types Static vs. Dynamic TestingMarch 28, 2017 Testing “Phases” Unit Integration System User Acceptance Testing Testing Types Black-box White-box Static vs. Dynamic Testing Integration: 3 types Top down Bottom up Sandwich Automated Testing Pros and cons Defect tracking March 28, 2017 SE 433: Lecture 1 Lecture 9

77 Ad Hoc Testing Ad hoc testing is not sufficient Ad hoc testingTesting carried out informally No formal test preparation No recognized test design technique is used Expected results not defined Arbitrariness guides the test execution activity Ad hoc testing is not sufficient March 28, 2017 SE 433: Lecture 1

78 Black Box Testing Treats a program or system as a black boxDesign test cases without the knowledge of internal structure and design of the system Design test cases based on the functional requirements of the system a.k.a: functional testing Send the system inputs, observe the outputs, decide if the system passed or failed the test Sometimes you don’t have access to source code March 28, 2017 SE 433: Lecture 1

79 White Box Testing a.k.a: glass box, clear box, or structural testingOpen up the box! a.k.a: glass box, clear box, or structural testing Design test cases by examining the internal design and source code of the program Require detailed knowledge of its structure Introduce the idea of measuring coverage: How adequate, or thorough, is the test suite? March 28, 2017 SE 433: Lecture 1

80 Software Analysis Software analysis isthe process of finding defects in a program (or parts of a program) without executing the program. a.k.a. static analysis Complementary to software testing Same purpose, different techniques Software analysis can be automated, i.e., using tools, static analyzers manual, e.g., inspection, code review March 28, 2017 SE 433: Lecture 1

81 Basic Principles of Software TestingMarch 28, 2017 SE 433: Lecture 1

82 Testing vs. VerificationGoal: find evidence for presence of failures Testing: execute a program with the intent of detecting failure Testing cannot guarantee correctness, i.e., absence of failures Related techniques: code reviews, program inspections VERIFICATION Goal: find evidence for absence of failures Verification guarantees correctness Related techniques: code generation, program synthesis (from spec) March 28, 2017 SE 433: Lecture 1

83 Testing vs. VerificationBoth, testing and verification attempts exhibit new failures Debugging is a systematic process that finds and eliminates the defect that led to an observed failure Programs without known failures may still contain defects:  If they have not been verified  If they have been verified, but the failure is not covered by the specification March 28, 2017 SE 433: Lecture 1

84 Make No Assumptions Do not assume that no new defects will be found.March 28, 2017 SE 433: Lecture 1

85 Define Expected OutputEach test case must include the expected output. The output of each test must be thoroughly inspected. Preferably: automated inspection, assertions Actual output Expected output ? March 28, 2017 SE 433: Lecture 1

86 Test the Invalid and UnexpectedTest input conditions that are valid and expected invalid and unexpected Test a program to see whether it does not do what it is supposed to do perform some but not all the required functions does what it is not supposed to do perform unnecessary functions, undesired side-effects March 28, 2017 SE 433: Lecture 1

87 Test Independently A program should be tested by independent testers.developer Understands the system but, will test "gently" and, is driven by "delivery" independent tester Must learn about the system, but, will attempt to break it and, is driven by quality March 28, 2017 SE 433: Lecture 1

88 Test Prudently You have a 3-day fishing vacationDay 1: net 10 fish in Pond A Day 2: net 20 fish in Pond B Day 3: which pond do you return to? March 28, 2017 SE 433: Lecture 1

89 Test Prudently Defects tend to cluster rather than evenly distribute.The probability of the existence of more defects in a component is proportional to the number of defects already found in the component. A small number of modules usually contains most of the defects discovered during pre-release testing, or is responsible for most of the operational failures. March 28, 2017 SE 433: Lecture 1

90 Tool Support is EssentialSome Reasons for Using Tools Automate repetitive tasks Avoid typos, etc. Cope with large programs Tools Used Test script Automated compilation, build and testing: Ant Automated running of tests: JUnit Debugging: Eclipse debugger Code Coverage: Cobertura March 28, 2017 SE 433: Lecture 1

91 Limitations of Software TestingMarch 28, 2017 SE 433: Lecture 1

92 Exhaustive Testing Can we exhaustively test a program?Let’s consider this simple program There are possible paths! If we execute one test per millisecond, it would take 3,170 years to test this program!! Exhaustive testing is impossible! loop < 20 iterations March 28, 2017 SE 433: Lecture 1

93 Absence of Defects “Program testing can be used to show the presence of bugs, but never their absence.” (Dijkstra, 1969) Testing reduces the probability of undiscovered defects remaining in the software but, even if no defects are found, it is not a proof of absence of defects. Edsger W. Dijkstra,1930 – 2002 March 28, 2017 SE 433: Lecture 1

94 When to Stop Testing? One of the most difficult problems in testing is not knowing when to stop. You can use metrics to make a guess. Keep track of the defect rate. When it goes towards zero you can use it as an indicator. Or, … Even if you do find the last bug, you’ll never know it. March 28, 2017 SE 433: Lecture 1

95 How Much Testing is Enough?Take into account the level of risk, including technical, safety, and business risks, and project constraints such as time and budget. Testing should provide feedback to stakeholders to make informed decisions about the release of the software, for the next development step, or handover to customers. In Lecture 10 we will discuss some techniques for making this decision. March 28, 2017 SE 433: Lecture 1

96 Summary – Key Concepts Objective and principles of testingCost of quality Fundamentals and basic terminologies in testing Failures, defects, faults, errors, mistakes Testing, analysis, black-box and white-box testing Test cases, test suites, test plan, test process Independent testing Limitations of testing, exhaustive testing March 28, 2017 SE 433: Lecture 1

97 Reading Chapters 1-4 of the textbook. March 28, 2017 SE 433: Lecture 1

98 Next Class SE 433 March 28, 2017 Topic: Software Quality Reading: Pezze and Young: Chapters 1-4 Assignment: Assignment 1: Determining Types of Triangles March 28, 2017 SE 433: Lecture 1 Lecture 1