1 [email protected] www.fe.up.pt/~jpfTQS - Teste e Qualidade de Software (Software Testing and Quality) Software Process Quality: Quality Standards, Quality Certification and Quality Management João Pascoal Faria
2 Index Introduction Overview of the ISO 9000 – International Set of Standards for Quality Management Overview of the Capability Maturity Model Integration for Software Engineering (CMMI-SW) Overview of the ISO/IEC International Standard for Software Process Assessment How to write a quality manual How to write a quality plan Quality control - measurements and reviews Product certification, warranties and liabilities, and quality aspects in customer support and maintenance services
3 Principal product quality factorsImprove product quality by acting upon its main factors For large projects with ‘average’ capabilities, the development process determines product quality Budget and Schedule (source: "Software Engineering", I. Sommervile)
4 Product and process Business System(Software Development Organization) goals Demand, Needs Business Process (Software Development Process) requirements engineer Costumer or Market (Software) Product or Service software developer software tester software quality (assurance) engineer Customer = external or internal resources Product = product or service Test = test and review Development = development and maintenance
5 Process quality attributesProcess characteristic Description Understandability To what extent is the process explicitly defined and how easy is it to understand the process definition? Visibility Do the process activities culminate in clear results so that the progress of the process is externally visible? Supportability To what extent can the process activities be supported by CASE tools? Acceptability Is the defined process acceptable to and usable by the engineers responsible for producing the software product? Reliability Is the process designed in such a way that process errors are avoided or trapped before they result in product errors? Robustness Can the process continue in spite of unexpected problems? Maintainability Can the process evolve to reflect changing organisational requirements or identified process improvements? Rapidity How fast can the process of delivering a system from a given specification be completed? (simplistic quality model ...) (source: "Software Engineering", I. Sommervile)
6 Strategies to improve software qualityFault/bug avoidance prevent bugs from ever occurring process quality, quality assurance Fault/bug detection and repair verification & validation tests, technical reviews Fault/bug tolerance redundancy, statistical methods
7 Software quality management (SQM)The goal of SQM (or Software Quality Assurance (SQA)) is to ensure that the required level of quality is achieved in software products, namely, that defined standards and procedures are followed SQM should aim to develop a ‘quality culture’ where quality is seen as everyone’s responsibility SQM should aim to continuously improve processes in order to improve and assure product quality SQA is distinct from V&V The goal of a software tester is to find bugs, find them as early as possible, and make sure that they get fixed A software quality assurance person's main responsibility is to create and enforce standards and methods to improve the development process in order to prevent bugs from ever occurring (source: Ron Patton)
8 Quality management activities(Organization–wide) Quality standards definition Establish organizational procedures and standards for quality Produce a quality manual (Project–wide) Quality planning Select applicable organisational procedures and standards for a particular project and modify and detail these as required. Set out the most significant quality attributes and how these are assessed. Produce a quality plan (Project–wide) Quality control Check the software development process (within a particular project) to ensure that procedures and standards, as defined in the quality plan, are being followed Produce quality review reports Two approaches to quality control Quality reviews - check the adherence of work products and performed processes to standards Quality measurements - the value of defined metrics is automatically measured on selected components of the product or process
9 Quality management and the software development processdeliverables (includes V&V) (source: Ian Sommerville) Quality management should be separate from project management to ensure independence of budget and schedule pressures!
10 Quality management and business process managementA software development process is the main business process in a software development business A business process is an organized set of activities that produces a result with value (product or service) to a costumer (internal or external) Business process management (BPM) comprises: process modeling and definition includes the definition of key performance indicators (KPI's), for process quality and performance evaluation process implementation, execution and operation process measurement and evaluation process improvement and re-engineering Business Process Management (BPM) and Quality Management (QM) are closely related Process improvement is essential to achieve product (and service) improvement
11 Standards for quality managementStandards are the key to effective quality management They may be international, national, organizational or project standards Product standards define characteristics that all components should exhibit e.g. a common programming style Process standards define how the software process should be enacted Both documentation and coding standards are valuable
12 Product and process standards
13 Documentation standardsParticularly important - documents are the tangible manifestation of the software Documentation process standards How documents should be developed, validated and maintained Document standards (product standards) Concerned with document identification, structure, presentation, changes highlighting, etc. Document interchange standards How documents are stored and interchanged between different documentation systems XML is becoming the de facto standard for document interchange
14 Importance of standardsEncapsulation of best practice - avoids repetition of past mistakes Framework for quality assurance process – it involves checking standard compliance Provide continuity - new staff can understand the organisation by understand the standards applied
15 Problems with standardsNot seen as relevant and up-to-date by software engineers Practitioners should be involved in their development. Engineers should understand the rationale underlying a standard Standards and their usage should be reviewed regularly. Standards can quickly become outdated and this reduces their credibility amongst practitioners Involve too much bureaucratic form filling Unsupported by software tools so tedious manual work is involved to maintain standards Detailed standards should have associated tool support. Excessive clerical work is the most significant complaint against standards
16 Development of process standardsCare must be taken not to impose inappropriate process standards Remember the PDCA cycle (see next)!
17 The PDCA cycle as a framework for improvementOriginally conceived by Walter Shewhart in 1930's, and later adopted by W. Edwards Deming Is a model that provides a framework for the improvement of a process, product or system Plan a change or a test, aimed at improvement Analyze what you intend to improve, looking for areas that hold opportunities for change Choose areas that offer the most return for the effort you put in Do - Carry out the change or test (preferably on a small scale) Check the results. What was learned? What went wrong? After you have implemented the change for a short time, you must determine how well it is working. Is it really leading to improvement in the way you had hoped? Decide on several measures with which you can monitor the level of improvement. Act - Adopt the change, abandon it, or run through the cycle again After planning a change, implementing and then monitoring it, you must decide whether it is worth continuing that particular change. If it consumed too much of your time, was difficult to adhere to, or even led to no improvement, you may consider aborting the change and planning a new one. However, if the change led to a desirable improvement or outcome, you may consider expanding the trial to a different area, or slightly increasing your complexity. This sends you back into the Plan phase and can be the beginning of the ramp of improvement. The PDCA cycle may be applied to all sorts of management activities (source: )
18 Index Introduction Overview of the ISO 9000 – International Set of Standards for Quality Management Overview of the Capability Maturity Model Integration for Software Engineering (CMMI-SW) Overview of the ISO/IEC TR International Standard for Software Process Assessment How to write a quality manual How to write a quality plan Quality control - measurements and reviews Product certification, warranties and liabilities, and quality aspects in customer support and maintenance services
19 ISO 9000 International set of standards for quality management (ISO 9000:2000, ISO 9001:2000, ISO 9004:2000, etc.) Applicable to a range of organisations from manufacturing to service industries ISO 9001:2000 specifies requirements for a quality management system for any organization that needs to demonstrate its ability to consistently provide product that meets customer and applicable regulatory requirements and aims to enhance customer satisfaction, in all business sectors Integrates previous standards ISO 9001, ISO 9002 and ISO 9003 ISO 9001 is a generic model that must be instantiated for each organisation ISO 9004:2000 provides guidance for continual improvement of a quality management system to benefit all parties (employees, owners, suppliers, society in general,…) through sustained customer satisfaction. It should be used to extend the benefits obtained from ISO 9001:2000 to all parties that are interested in or affected by the business operations.
20 Key documents in the ISO 9000:2000 family(source: Vivek Nanda, ISO 9001:2000 Achieving Compliance and Continuous Improvement in Software Development Companies)
21 ISO 9000 certification Quality standards and procedures should be documented in an organisational quality manual External body may certify that an organisation’s quality manual conforms to ISO 9000 standards (namely ISO 9001) Customers are, increasingly, demanding that suppliers are ISO 9000 certified
22 Certification and accreditationexample: IPQ example: APCER (source: Vivek Nanda, ISO 9001:2000 Achieving Compliance and Continuous Improvement in Software Development Companies)
23 ISO 9000 and quality management(source: Ian Sommerville)
24 Some key ISO 9000 terms (source: Vivek Nanda, ISO 9001:2000 Achieving Compliance and Continuous Improvement in Software Development Companies)
25 ISO 9001:2000 quality principles (1)Each requirement in ISO 9001:2000 is founded on one of these principles: 1. Customer focused organization. An organization's success is contingent upon it retaining current customers and attracting new ones. Therefore, in order to achieve customer satisfaction, an organization must understand current and future requirements of its customers, and it must continuously strive to exceed customer expectations. 2. Leadership. Organizational success requires a strong Leadership that has a clear vision and is committed to the organization's goals and objectives. 3. Involvement of people. In order to ensure organizational success, it is vital that each employee be involved and contribute to the best of his or her ability. 4. Process approach. Because all work in organizations is executed in processes, improvements are easier to implement by taking a process view of the organization.
26 ISO 9001:2000 quality principles (2)5. System approach to management. All the processes in an organization interact in order to deliver the end product of the organization. Therefore, an organization cannot improve quality by merely looking at individual processes in isolation. It must improve the interaction and handoff across processes because a poor interface between different processes may have a debilitating impact on overall process execution. 6. Continual improvement. In order to be recognized as one of the better organizations amongst its peers, every organization must strive to distinguish itself. This can only happen by establishing continual improvement as a permanent objective. 7. Factual approach to decision making. Informed decisions can only be made based on facts and data. Therefore, every organization must strive to maintain useful, complete, and accurate data for use in its decision-making processes. 8. Mutually beneficial supplier relationships. An organization's ability to deliver quality products is based upon its ability to establish a synergistic and quality-driven relationship with its suppliers.
27 ISO 9001:2000 structure ISO 9001:2000 groups all the standard's requirements under five major clauses: Quality management system (clause 4). This includes general requirements and documentation requirements that are applicable to the entire QMS. Management responsibility (clause 5). This entails providing direction, setting goals and objectives, and management review of the QMS. Resource management (clause 6). This entails determining resource needs, and providing and managing the resources during the course of the project. Product realization (clause 7). This entails transforming the input (customer requirements) into output (product and/or service) by means of processes, such that the output meets customer requirements and expectations. Measurement, analysis, and improvement (clause 8). "Measurement" entails defining and collecting measurements for continual improvement (including customer satisfaction data), and performing internal quality audits. "Analysis" entails examining the collected measurement data and audit results to identify trends and opportunities for improvement. "Improvement" entails management review and use of the audit results and measurement data for the formulation and implementation of improvement plans. The cycle of "management responsibility resource management product realization measurement, analysis, and improvement" repeats itself in successive iterations and thus results in continuous improvement within the organization.
28 Overview of ISO 9001:2000 requirements (1)(source: Vivek Nanda, ISO 9001:2000 Achieving Compliance and Continuous Improvement in Software Development Companies)
29 Overview of ISO 9001:2000 requirements (2)
30 Overview of ISO 9001:2000 requirements (3)
31 Overview of ISO 9001:2000 requirements (4)
32 Overview of ISO 9001:2000 requirements (5)
33 A sample meta-process to cover the full range of ISO 9001:2000 requirements for software development companies (1) (source: Vivek Nanda, ISO 9001:2000 Achieving Compliance and Continuous Improvement in Software Development Companies)
34 A sample meta-process to cover the full range of ISO 9001:2000 requirements for software development companies (2) A sample Software Product Development Process (source: Vivek Nanda, ISO 9001:2000 Achieving Compliance and Continuous Improvement in Software Development Companies)
35 Quality documentation(source: Vivek Nanda, ISO 9001:2000 Achieving Compliance and Continuous Improvement in Software Development Companies)
36 The two-step auditing process(source: Vivek Nanda, ISO 9001:2000 Achieving Compliance and Continuous Improvement in Software Development Companies)
37 ISO 9000 Implementation - Certification Costs(source: Vivek Nanda, ISO 9001:2000 Achieving Compliance and Continuous Improvement in Software Development Companies)
38 ISO 9000 Implementation - Time frame for achieving certification(source: Vivek Nanda, ISO 9001:2000 Achieving Compliance and Continuous Improvement in Software Development Companies)
39 ISO 9000 Implementation - A sample project plan(source: Vivek Nanda, ISO 9001:2000 Achieving Compliance and Continuous Improvement in Software Development Companies)
40 Sample Audit Questions for ISO 9001:2000 certification (1)(source: Vivek Nanda, ISO 9001:2000 Achieving Compliance and Continuous Improvement in Software Development Companies) Clause 4 - Quality Management System Requirements Clause General Requirements Have you identified all the processes in your QMS, their sequence, and interactions? Are any of your QMS processes outsourced to other supplier(s)? If yes, how do you control such processes? Clause 4.2 – Documentation Requirements Subclause General What is your documented quality policy? What are your documented quality objectives? Subclause Quality Manual Does your quality manual clearly specify the scope of your QMS? Does your quality manual describe how the processes in your QMS interact? Does your quality manual contain or reference documented procedures for the QMS?
41 Sample Audit Questions for ISO 9001:2000 certification (2)Clause 4 - Quality Management System Requirements (cont.) Clause 4.2 – Documentation Requirements (cont.) Subclause Control of Documents Are documents approved prior to use? How do you determine who needs to approve changes? Describe how you control document changes. Is this process formally documented? Are obsolete documents appropriately identified? How do you ensure they are not used? Subclause Control of records Do you have a procedure that describes how you control records? Where do you specify the retention time for each type of record?
42 Sample Audit Questions for ISO 9001:2000 certification (3)Clause 5 – Management Responsibility Requirements Clause 5.1 – Management Commitment How does management ensure that quality objectives are established in the organization? Are their periodic management reviews of the QMS? Clause 5.2 – Customer Focus How are customer requirements determined? What verification do you perform to ensure that specific customer requirements have been met? Clause 5.3 – Quality Policy How do you communicate the quality policy to employees? Are employees aware of the quality policy? Do they understand how it relates to their job? Is the quality policy adhered to in the organization (or is it merely a marketing slogan)?
43 Sample Audit Questions for ISO 9001:2000 certification (4)Clause 5 – Management Responsibility Requirements (cont.) Clause Planning Subclause – Quality objectives Are the quality objectives quantitative (measurable)? Do employees know what quality objectives apply to their jobs? Subclause – Quality management system planning How does management ensure that the organization plans for achievement of defined quality objectives? How does management ensure that integrity of the QMS and compliance to ISO 9001:2000 requirements is maintained when there are changes to the QMS, or there are organizational changes that impact the QMS?
44 Sample Audit Questions for ISO 9001:2000 certification (5)Clause 5 – Management Responsibility Requirements (cont.) Clause 5.5 – Responsibility, Authority and Communication Subclause – Responsibility and authority Are organization responsibilities clearly defined and communicated? How is this done? Subclause – Management representative Who is your management representative? Subclause – Internal communication Are employees informed about the effectiveness of the QMS? For example, are measurement data and customer satisfaction data communicated to relevant employees?
45 Sample Audit Questions for ISO 9001:2000 certification (6)Clause 5 – Management Responsibility Requirements (cont.) Clause 5.6 – Management Review Subclause General What are some examples of typical agenda items for discussion at management reviews? Subclause – Review input Do management reviews follow up on the status of open action items from past management reviews? Are results of first-party, second-party, and third-party audits presented at management reviews? Subclause – Review output Are records of management reviews maintained, such as meeting minutes?
46 Sample Audit Questions for ISO 9001:2000 certification (7)Clause 6 – Resource Management Requirements Clause 6.1 – Provision of Resources Are there adequate resources to implement and maintain the QMS? How are resources estimated and secured in a timely manner? Clause 6.2 – Human Resources Subclause General Do employees have the needed competency to perform the job? Subclause – Competence, awareness and training Is the necessary competence for every employee position identified? Do you provide training or use other mechanisms to address deficiency in required competency in employees? How do you evaluate that the action taken to bridge the competency deficiency was effective? What records do you maintain of employee competency?
47 Sample Audit Questions for ISO 9001:2000 certification (8)Clause 6 – Resource Management Requirements (cont.) Clause 6.3 – Infrastructure Is the infrastructure provided adequate for the task performed? Clause 6.4 – Work Environment Is the work environment appropriately maintained?
48 Sample Audit Questions for ISO 9001:2000 certification (9)Clause 7 – Product Realization Requirements Clause 7.1 – Planning and Product Realization Is there a defined process for developing the product? Clause 7.2 – Customer-Related Processes Are records maintained to demonstrate that the executed processes and developed product comply with applicable requirements? Subclause – Determination of requirements related to the product Are the requirements for the product adequately defined? Subclause – Review of requirements related to the product Are the product requirements reviewed? Does the organization have the ability to meet the approved requirements? Are records maintained for the requirements reviews, including disposition of action items?
49 Sample Audit Questions for ISO 9001:2000 certification (10)Clause 7 – Product Realization Requirements (cont.) Clause 7.2 – Customer-Related Processes (cont.) Subclause – Customer communication What mechanisms do you use to communicate with the customer? Is this communication periodic or sporadic? Is the customer provided feedback on the status of customer complaints? Clause 7.3 – Design and Development Subclause – Design and development planning How do you plan for design and development of a product? What mechanisms do you have in place to control project execution, manage interfaces between departments, and coordinate tasks? Are project plans revised, when appropriate, during the product development process? Subclause – Design and development inputs Are functional and nonfunctional requirements for the product defined?
50 Sample Audit Questions for ISO 9001:2000 certification (11)Clause 7 – Product Realization Requirements (cont.) Clause 7.3 – Design and Development (cont.) Subclause Design and development outputs How are the design and development outputs documented? Are these approved prior to release? Subclause Design and development review Is the design and development reviewed with appropriate parties during project execution as per plans? Subclause Design and development verification During project execution, is the design and development output verified against design and development input as per plans?
51 Sample Audit Questions for ISO 9001:2000 certification (12)Clause 7 – Product Realization Requirements (cont.) Clause 7.3 – Design and Development (cont.) Subclause Design and development validation Is the design and development validated against product requirements as per plans? Subclause – Control of design and development changes In the event that a change to the design or development is required, is the change identified and approved prior to implementation? Is an assessment made of the impact of the proposed changes on the design and development already completed (or under way)?
52 Sample Audit Questions for ISO 9001:2000 certification (13)Clause 7 – Product Realization Requirements (cont.) Clause Purchasing Subclause – Purchasing process How do you ensure that the purchased product conforms to requirements? Is there a defined process and established criteria for the evaluation and selection of potential suppliers? Subclause Purchasing information Are the purchase requirements adequately defined? Subclause – Verification of purchased product Is the purchased product verified against purchase requirements? How is the verification performed?
53 Sample Audit Questions for ISO 9001:2000 certification (14)Clause 7 – Product Realization Requirements (cont.) Clause 7.5 – Production and Service Provision Subclause – Control of production and service provision Is the master copy of the software product uniquely identified and stored in a secure location? Are there controls in place to ensure that the replicated copy of the software product is a true copy of the master? Note: This question is only applicable if the product is delivered on physical media. Subclause – Validation of processes for production and service provision When the software product is delivered without certain requirements being tested, are alternate means used to validate the product against those requirements? Subclause – Identification and traceability Has your organization determined need for traceability during product requirements? Can you show examples of traceability, such as traceability of test cases to requirements?
54 Sample Audit Questions for ISO 9001:2000 certification (15)Clause 7 – Product Realization Requirements (cont.) Clause 7.5 – Production and Service Provision (cont.) Subclause – Customer property How do you identify customer property? How do you protect customer property from unauthorized access and abuse, or corruption? Subclause – Preservation of product How do you ensure that the product is adequately preserved during packaging and delivery to the customer? Clause 7.6 – Control and Monitoring of Measuring Devices Have you identified monitoring and measurement equipment requiring calibration? When and how is this calibration performed? What calibration records do you maintain?
55 Sample Audit Questions for ISO 9001:2000 certification (16)Clause 8 - Measurement, Analysis, and Improvement Requirements Clause 8.1 – General Have you implemented measurements processes in your organization? Clause Monitoring and Measurement Subclause – Customer satisfaction Do you collect customer satisfaction information? How do you analyze and use customer satisfaction information for continual improvement?
56 Sample Audit Questions for ISO 9001:2000 certification (17)Clause 8 - Measurement, Analysis, and Improvement Requirements (cont.) Clause Monitoring and Measurement (cont.) Subclause – Internal audit Do you have an internal audit plan? How many internal audits have you conducted? Are the internal auditors appropriately qualified and independent of the audited area? What records do you maintain of the audits performed? Do you present results of audits at management reviews?
57 Sample Audit Questions for ISO 9001:2000 certification (18)Clause 8 - Measurement, Analysis, and Improvement Requirements (cont.) Clause Monitoring and Measurement (cont.) Subclause Monitoring and measurement of processes What process measurements do you use? Subclause Monitoring and measurement of product What product measurements do you use? Can you show records that prove that the product meets the originally identified acceptance criteria? Can you show records to prove that the product was released by authorized personnel?
58 Sample Audit Questions for ISO 9001:2000 certification (19)Clause 8 - Measurement, Analysis, and Improvement Requirements (cont.) Clause 8.3 – Control of Nonconforming Product How do you control a product that is determined to be nonconforming? What do you do if a product is determined to be nonconforming after release to the customer? Clause 8.4 – Analysis of Data What techniques do you use to analyze measurement data?
59 Sample Audit Questions for ISO 9001:2000 certification (20)Clause 8 - Measurement, Analysis, and Improvement Requirements (cont.) Clause Improvement Subclause – Continual improvement How do you continually improve the effectiveness of your QMS? Subclause – Corrective action Do you have a documented procedure for handling of corrective actions? Do you determine the root cause of nonconformities? How do you determine that a proposed corrective action is adequate? Do you maintain records of implemented corrective actions? Subclause – Preventive action Do you have a documented procedure for handling of preventive actions? Once a preventive action has been implemented, do you perform a follow-up to verify the implemented action?
60 Index Introduction Overview of the ISO 9000 – International Set of Standards for Quality Management Overview of the Capability Maturity Model Integration for Software Engineering (CMMI-SW) Overview of the ISO/IEC TR International Standard for Software Process Assessment How to write a quality manual How to write a quality plan Quality control - measurements and reviews Product certification, warranties and liabilities, and quality aspects in customer support and maintenance services
61 Capability Maturity Model Integration for Software Engineering (CMMI-SW), Version 1.1Developed by the Software Engineering Institute, Carnegie Mellon University, sponsored by the U.S. Department of Defense Designed to help organizations improve their product and service development, acquisition, and maintenance processes Judge the maturity of the software processes of an organization (based on maturity questionnaires) Identify the key practices required to increase the maturity of these processes Staged representation - measures process-improvement achievement across an organizational unit using maturity levels (as in traditional SW-CMM) At maturity level n, an organization has achieved all the specific goals of the process areas assigned to maturity levels n (and generic goals assigned to levels n) To achieve each goal a set of (best) practices must be implemented Continuous representation - measures process-improvement achievement within individual process areas (e.g. configuration management) using capability levels The CMMI models are the result of an effort of integration of the "traditional" CMM models (SW-CMM, etc.)
62 CMMI-SW, v1.1, staged version Process Areas by Maturity Levels and Process Area CategoriesLevel 1) Initial Level 4) Quantitatively Managed Level 2) Managed Level 3) Defined (*) Level 5) Optimizing Process Managem. Organizational Process Focus Organizational Process Performance Organizational Innovation and Deployment Organizational Process Definition Organizational Training Project Planning Integrated Project Management for Integrated Product and Process Development (IPPD) Quantitative Project Management Project Managem. Project Monitoring and Control Supplier Agreement Management Risk Management Requirements Management (REQM) Requirements Development (RD) Technical Solution (TS) Engineering Product Integration (PI) Verification (VER) Validation (VAL) Configuration Management (CM) Decision Analysis and Resolution (DAR) Causal Analysis and Resolution (CAR) Measurement and Analysis (MA) Support Process and Product Quality Assurance (PPQA) (*) also includes practices to institutionalize a defined process for each process area of level 2
63 CMMI-SW, v1.1, staged version Maturity Levels and Process Areas5. Optimizing Organizational Innovation and Deployment Causal Analysis and Resolution (CAR) 4. Quantitatively Managed Organizational Process Performance Quantitative Project Management 3. Defined Organizational Process Focus Organizational Process Definition Organizational Training Integrated Project Management for IPPD Risk Management Requirements Development (RD) Technical Solution (TS) Product Integration Verification (VER) Validation (VAL) Decision Analysis and Resolution (DAR) 2. Managed Project Planning Project Monitoring and Control Supplier Agreement Management Requirements Management (REQM) Configuration Management (CM) Measurement and Analysis (MA) Process and Product Quality Assurance (PPQA) 1.Initial
64 CMMI-SW, v1.1, staged version Maturity Levels1) Initial. The software process is characterized as ad hoc, and occasionally even chaotic. Few processes are defined, and success depends on individual effort and heroics. 2) Managed. Basic project management processes are established to track cost, schedule, and functionality. The necessary process discipline is in place to repeat earlier successes on projects with similar applications. 3) Defined. The organization’s set of standard processes is established and improved over time. Projects and organizational units establish their defined processes by tailoring the organization’s set of standard processes according to tailoring guidelines. 4) Quantitatively Managed. Detailed measures of the software process and product quality are collected. Both the software process and products are quantitatively understood and controlled. 5) Optimizing. Continuous process improvement is enabled by quantitative feedback from the process and from piloting innovative ideas and technologies.
65 CMMI-SW, v1.1, staged version Maturity Levels (1/5)1: Initial Processes are usually ad hoc and chaotic The organization usually does not provide a stable environment Success depends on the competence and heroics of the people in the organization and not on the use of proven processes Products and services that work are often produced; however, they frequently exceed the budget and schedule of their projects Tendency to over commit, abandon processes in the time of crisis, and not be able to repeat their past successes
66 CMMI-SW, v1.1, staged version Maturity Levels (2/5)2: Managed Requirements, processes, work products, and services are managed (on a per project basis) Processes are planned, performed, measured, and controlled (but processes may be quite different from instance to instance, i.e., from project to project) Existing practices are retained during times of stress Projects are performed and managed according to their documented plans The status of the work products and the delivery of services are visible to management at defined points (for example, at major milestones and at the completion of major tasks) Commitments are established among relevant stakeholders and are revised as needed Work products are reviewed with stakeholders and are controlled The work products and services satisfy their specified requirements, standards, and objectives
67 CMMI-SW, v1.1, staged version Maturity Levels (3/5)3: Defined The organization’s set of standard processes is established and improved over time Projects and organizational units establish their defined processes by tailoring the organization’s set of standard processes according to tailoring guidelines As a result, the processes that are performed across the organization are consistent except for the differences allowed by the tailoring guidelines The organization’s management establishes process objectives based on the organization’s set of standard processes and ensures that these objectives are appropriately addressed Processes are typically described in more detail and more rigorously than at maturity level 2 (and are organization-wide instead of project-wide) Processes are managed more proactively than at maturity level 2 using an understanding of the interrelationships of the process activities and detailed measures of the process, its work products, and its services
68 CMMI-SW, v1.1, staged version Maturity Levels (4/5)4: Quantitatively Managed Subprocesses are selected that significantly contribute to overall process performance. These selected subprocesses are controlled using statistical and other quantitative techniques. Quantitative objectives for quality and process performance are established and used as criteria in managing processes. Quantitative objectives are based on the needs of the customer, end users, organization, and process implementers. Quality and process performance are understood in statistical terms and are managed throughout the life of the processes. For these processes, detailed measures of process performance are collected and statistically analyzed. Special causes of process variation are identified and, where appropriate, the sources of special causes are corrected to prevent future occurrences. Quality and process performance measures are incorporated into the organization’s measurement repository to support fact-based decision making in the future. A critical distinction between maturity level 3 and maturity level 4 is the predictability of process performance. At maturity level 4, the performance of processes is controlled using statistical and other quantitative techniques, and is quantitatively predictable. At maturity level 3, processes are only qualitatively predictable.
69 CMMI-SW, v1.1, staged version Maturity Levels (5/5)5: Optimizing Processes are continually improved based on a quantitative understanding of the common causes of variation inherent in processes. Maturity level 5 focuses on continually improving process performance through both incremental and innovative technological improvements. Quantitative process-improvement objectives for the organization are established, continually revised to reflect changing business objectives, and used as criteria in managing process improvement. The effects of deployed process improvements are measured and evaluated against the quantitative process-improvement objectives. Both the defined processes and the organization’s set of standard processes are targets of measurable improvement activities. Process improvements to address common causes of process variation and measurably improve the organization’s processes are identified, evaluated, and deployed. Improvements are selected based on a quantitative understanding of their expected contribution to achieving the organization’s process-improvement objectives versus the cost and impact to the organization. The performance of the organization’s processes is continually improved. Optimizing processes that are agile and innovative depends on the participation of an empowered workforce aligned with the business values and objectives of the organization. The organization’s ability to rapidly respond to changes and opportunities is enhanced by finding ways to accelerate and share learning. Improvement of the processes is inherently part of everybody’s role, resulting in a cycle of continual improvement.
70 CMMI-SW, v1.1 Process Area CategoriesProcess Management - contains the cross-project activities related to defining, planning, resourcing, deploying, implementing, monitoring, controlling, appraising, measuring, and improving processes Project Management - covers the project management activities related to planning, monitoring, and controlling the project Engineering - covers the development and maintenance activities that are shared across engineering disciplines (e.g., systems engineering and software engineering) Support - covers the activities that support product development and maintenance
71 CMMI-SW, v1.1 Verification and Validation (V&V)Verification - ensures that the specified requirements are satisfied Confirms that (intermediate and final) work products properly reflect the requirements specified for them Ensures that “you built it right” Occurs throughout the development of the product and work products, beginning with verification of the requirements, progressing through the verification of the evolving work products, and culminating in the verification of the completed product Validation - ensures that the product will fulfill its intended use Confirms that the (final) product, as provided, will fulfill its intended use Ensures that “you built the right thing” Applied mainly to the product and product components in its intended environment, but can also be applied to work products (e.g., requirements, designs, prototypes) selected on the basis of which are the best predictors of how well the product and product component will satisfy user needs Both validation and verification activities often run concurrently and may use portions of the same environment
72 CMMI-SW, v1.1 Interactions Among V&V and other Engineering Process AreasREQM Requirements Product and product component requirements RD Alternative TS PI solutions Product components Product Customer Require - ments Product components, work products, verification and validation reports VER VAL Customer needs
73 CMMI-SW, v1.1 Process and Product Quality Assurance (PPQA)Quality Assurance - ensures that planned processes are implemented Involves objectively evaluating performed processes and work products against applicable process descriptions, standards, and procedures, and communicating and ensuring resolution of noncompliance issues Usually performed by a QA group that is independent of the project, or by peers not directly involved in the development or maintenance of the work product subject to evaluation Should begin in the early phases of a project (with the participation of the QA group) to establish plans, processes, standards, and procedures that will add value to the project and satisfy its requirements and the organizational policies and that will be useable for performing QA evaluations, and to designate the specific processes and work products that will be evaluated V&V and PPQA may on occasion address the same work product but from different perspectives; care should be taken to minimize duplication of effort
74 CMMI-SW, v1.1 Interactions Among PPQA and other Process AreasOrganization IPPD OEI Infrastructure organizational environment for integration Ability to develop CAR and deploy IPPD processes and supporting assets Process improvement Integrated work proposals IPPD environment and knowledge people practices and skill Defects and needs other problems Quality and Measurements, noncompliance analyses Process Management process areas Project Management process areas issues MA PPQA Information Processes and work products, needs standards and procedures All process areas Selected issues Configuration Baselines, Formal DAR items, audit reports evaluations change CM requests
75 CMMI-SW, v1.1 Terminology - artifactsProduct - any tangible output or service that is a result of a process and that is intended for delivery to a customer or end user A product is a work product that is delivered to the customer Product components - lower level components of the product, that are integrated to “build” the product There may be multiple levels of product components A product component is any work product that must be engineered (requirements defined and designs developed and implemented) to achieve the intended use of the product throughout its life Work product - any artifact produced by a process, including files, documents, parts of the product, services, processes, specifications, and invoices Examples of processes to be considered as work products include a manufacturing process, a training process, and a disposal process for the product
76 CMMI-SW, v1.1 Terminology – actors and activitiesCustomer - the party (individual, project, or organization) responsible for accepting the product or for authorizing payment The customer is external to the project, but not necessarily external to the organization The customer may be a higher level project Customers are a subset of stakeholders Stakeholder - a group or individual that is affected by or in some way accountable for the outcome of an undertaking Stakeholders may include project members, suppliers, customers, end users, and others Project manager - the person responsible for planning, directing, controlling, structuring, and motivating the project and satisfying the customer Project - a managed set of interrelated resources that delivers one or more products to a customer or end user This set of resources has a definite beginning and end and typically operates according to a plan Such a plan is frequently documented and specifies the product to be delivered or implemented, the resources and funds used, the work to be done, and a schedule for doing the work A project can be composed of projects
77 CMMI-SW, v1.1 Terminology – defined and managedManaged process - a performed process that is planned and executed in accordance with policy; employs skilled people having adequate resources to produce controlled outputs; involves relevant stakeholders; is monitored, controlled, and reviewed; and is evaluated for adherence to its process description Institutionalize a managed process is a generic goal for all process areas at maturity level 2 Defined process - a managed process that is tailored from the organization’s set of standard processes according to the organization’s tailoring guidelines; has a maintained process description; and contributes work products, measures, and other process-improvement information to the organizational process assets Institutionalize a defined process is a generic goal for all process areas at maturity level 3 or higher
78 Example of specific goals and practicesSpecific goals and practices in the Verification process area: SG 1 Prepare for Verification SP 1.1 Select Work Products for Verification SP 1.2 Establish the Verification Environment SP 1.3 Establish Verification Procedures and Criteria SG 2 Perform Peer Reviews SP 2.1 Prepare for Peer Reviews SP 2.2 Conduct Peer Reviews SP 2.3 Analyze Peer Review Data SG 3 Verify Selected Work Products SP 3.1 Perform Verification SP 3.2 Analyze Verification Results and Identify Corrective Action
79 Example of generic goals and practicesGeneric goals and practices that apply to the Verification process area: GG 3 Institutionalize a Defined Process GP 2.1 (CO 1) Establish an Organizational Policy GP 3.1 (AB 1) Establish a Defined Process GP 2.2 (AB 2) Plan the Process GP 2.3 (AB 3) Provide Resources GP 2.4 (AB 4) Assign Responsibility GP 2.5 (AB 5) Train People GP 2.6 (DI 1) Manage Configurations GP 2.7 (DI 2) Identify and Involve Relevant Stakeholders GP 2.8 (DI 3) Monitor and Control the Process GP 3.2 (DI 4) Collect Improvement Information GP 2.9 (VE 1) Objectively Evaluate Adherence GP 2.10 (VE 2) Review Status with Higher Level Management (Note: because this process area is introduced at maturity level 3, it inherits the practices 2.* of the generic goal GG 2 Institutionalize a Defined Process)
80 A sample maturity questionnaire (CMU/SEI-94-SR-7)Each question has the following possible answers: Yes No Does Not Apply Don't Know There is a space for Comments in each question This version has about 6 to 8 questions per Key Process Area (KPA), and covers 18 KPA's, and can usually be completed in one hour
81 A sample maturity questionnaire (CMU/SEI-94-SR-7)Example of questions for one KPA - Requirements Management: Are system requirements allocated to software used to establish a baseline for software engineering and management use? As the systems requirements allocated to software change, are the necessary adjustments to software plans, work products, and activities made? Does the project follow a written organizational policy for managing the system requirements allocated to software? Are the people in the project who are charged with managing the allocated requirements trained in the procedures for managing allocated requirements? Are measurements used to determine the status of the activities performed for managing the allocated requirements (e.g., total number of requirements changes that are proposed, open, approved, and incorporated into the baseline)? Are the activities for managing allocated requirements on the project subjected to SQA review?
82 The CMM and ISO 9000 There is a clear correlation between the key processes in the CMM and the quality management processes in ISO 9000 The CMM is more detailed and prescriptive and includes a more detailed framework for improvement Organisations rated as level 2 in the CMM are likely to be ISO 9000 compliant
83 Index Introduction Overview of the ISO 9000 – International Set of Standards for Quality Management Overview of the Capability Maturity Model Integration for Software Engineering (CMMI-SW) Overview of the ISO/IEC TR International Standard for Software Process Assessment How to write a quality manual How to write a quality plan Quality control - measurements and reviews Product certification, warranties and liabilities, and quality aspects in customer support and maintenance services
84 ISO/IEC TR 15504 – Purpose and scopeAligned with CMMI Provides a framework for the assessment of software processes This framework can be used by organizations involved in planning, managing, monitoring, controlling, and improving the acquisition, supply, development, operation, evolution and support of software Provides a structured approach for the assessment of software processes for the following purposes: by or on behalf of an organization with the objective of understanding the state of its own processes for process improvement by or on behalf of an organization with the objective of determining the suitability of its own processes for a particular requirement or class of requirements (process capability determination) by or on behalf of one organization with the objective of determining the suitability of another organization's processes for a particular contract or class of contracts (process capability determination)
85 ISO/IEC TR 15504 – Structure (1)Part 1 : Concepts and introductory guide (informative) Part 2 : A reference model for processes and process capability (normative) Defines a two dimensional reference model for describing processes and process capability used in a process assessment Part 3 : Performing an assessment (normative) Defines the requirements for performing an assessment in such a way that the outcomes will be repeatable, reliable and consistent Part 4 : Guide to performing assessments (informative) Provides guidance on performing software process assessments, interpreting the requirements of parts 2 and 3 for different assessment contexts Part 5 : An assessment model and indicator guidance (informative) Provides an exemplar model for performing process assessments that is based upon and directly compatible with the reference model in part 2 The assessment model(s) extend the reference model through the inclusion of a comprehensive set of indicators of process performance and capability
86 ISO/IEC TR 15504 – Structure (2)Part 6 : Guide to competency of assessors (informative) Describes the competence, education, training and experience of assessors that are relevant to conducting process assessments Part 7 : Guide for use in process improvement (informative) Describes how to define the inputs to and use the results of an assessment for the purposes of process improvement Part 8 : Guide for use in determining supplier process capability (informative) Describes how to define the inputs to and use the results of an assessment for the purpose of process capability determination Part 9 : Vocabulary (normative) Consolidated vocabulary of all terms specifically defined for the purposes of ISO/IEC TR 15504
87 ISO/IEC TR 15504 – Reference modelThe process dimension defines 57 processes grouped into the following 6 process categories: Customer-Supplier – processes that directly impact the customer, support development and transition of the software to the customer, and provide for the correct operation and use of the software product and/or service Engineering - processes that directly specify, implement, or maintain the software product, its relation to the system and its customer documentation Support - processes which may be employed by any of the other processes (including other supporting processes) at various points in the software life cycle Management - processes which contain practices of a generic nature which may be used by anyone who manages any type of project or process within a software life cycle Organization - processes that establish the business goals of the organization and develop process, product, and resource assets which, when used by the projects in the organization, will help the organization achieve its business goals Acquirer - processes that directly impact the acquirer and is generally driven by the acquirer
88 ISO/IEC TR 15504 – Reference modelProcess capability levels and process attributes: Level 0 – Incomplete - There is general failure to attain the purpose of the process. There are few or no easily identifiable work products or outputs of the process. Level 1- Performed - The purpose of the process is generally achieved. The achievement may not be rigorously planned and tracked. Individuals within the organization recognize that an action should be performed, and there is general agreement that this action is performed as and when required. There are identifiable work products for the process, and these testify to the achievement of the purpose. 1.1 Process Performance - The extent to which the process achieves the process outcomes by transforming identifiable input work products to produce identifiable output work products.
89 ISO/IEC TR 15504 – Reference modelProcess capability levels and process attributes (cont.) Level 2 – Managed - The process delivers work products according to specified procedures and is planned and tracked. Work products conform to specified standards and requirements. 2.1 Performance Management - The extent to which the performance of the process is managed to produce work products that meet the defined objectives. 2.2 Work Product Management - The extent to which the performance of the process is managed to produce work products that are appropriately documented, controlled and verified.
90 ISO/IEC TR 15504 – Reference modelProcess capability levels and process attributes (cont.) Level 3 – Established - The process is performed and managed using a defined process based upon good software engineering principles. Individual implementations of the process use approved, tailored versions of standard, documented processes to achieve the process outcomes. The resources necessary to establish the process definition are also in place. 31. Process Definition - The extent to which the performance of the process uses a process definition based upon a standard process to achieve the process outcomes. 3.2 Process Resource - The extent to which the process draws upon suitable resources (for example, human resources and process infrastructure) that is appropriately allocated to deploy the defined process.
91 ISO/IEC TR 15504 – Reference modelProcess capability levels and process attributes (cont.) Level 4 – Predictable - The defined process is performed consistently in practice within defined control limits, to achieve its defined process goals. Detailed measures of performance are collected and analyzed. This leads to a quantitative understanding of process capability and an improved ability to predict and manage performance. Performance is quantitatively managed. The quality of work products is quantitatively known. 4.1 Process Measurement - The extent to which product and process goals and measures are used to ensure that performance of the process supports the achievement of the defined goals in support of the relevant business goals. 4.2 Process Control - The extent to which the process is controlled through the collection, analysis, and use of product and process measures to correct, where necessary, the performance of the process to achieve the defined product and process goals.
92 ISO/IEC TR 15504 – Reference modelProcess capability levels and process attributes (cont.) Level 5 - Optimizing - Performance of the process is optimized to meet current and future business needs, and the process achieves repeatability in meeting its defined business goals. Quantitative process effectiveness and efficiency goals (targets) for performance are established, based on the business goals of the organization. Continuous process monitoring against these goals is enabled by obtaining quantitative feedback and improvement is achieved by analysis of the results. Optimizing a process involves piloting innovative ideas and technologies and changing non-effective processes to meet defined goals or objectives. 5.1 Process Change - The extent to which changes to the definition, management and performance of the process are controlled to achieve the relevant business goals of the organization. 5.2 Continuous Improvement - The extent to which changes to the process are identified and implemented to ensure continuous improvement in the fulfillment of the relevant business goals of the organization.
93 ISO/IEC TR 15504 – Reference modelRating of process attributes - process attributes are rated on a percentage scale from zero to one hundred percent that represents the extent of achievement of the attribute N Not achieved – (0% to 15%) - There is little or no evidence of achievement of the defined attribute in the assessed process. P Partially achieved (>15% to 50%) - There is evidence of a sound systematic approach to and achievement of the defined attribute in the assessed process. L Largely achieved (>50% to 85%) - There is evidence of a sound systematic approach to and significant achievement of the defined attribute in the assessed process. F Fully achieved (>85% to 100%) - There is evidence of a complete and systematic approach to and full achievement of the defined attribute in the assessed process. Question: what ratings should get process attributes in levels 1 to n to assert a given process capability level n?
94 Index Introduction Overview of the ISO 9000 – International Set of Standards for Quality Management Overview of the Capability Maturity Model Integration for Software Engineering (CMMI-SW) Overview of the ISO/IEC TR International Standard for Software Process Assessment How to write a quality manual How to write a quality plan Quality control - measurements and reviews Product certification, warranties and liabilities, and quality aspects in customer support and maintenance services
95 A sample quality manual template (1)Section 1: Introduction Purpose
96 A sample quality manual template (2)Section 2: QMS Overview Quality Policy
97 A sample quality manual template (3)Section 3: QMS Structure
98 A sample quality manual template (4)Section 4: QMS Content (ISO 9001:2000 based) Brief Overview of ISO 9001:2000 Clause 4: Quality management system requirements
99 A sample quality manual template (5)Section 4: QMS Content (meta-process based) Brief Overview of ISO 9001:2000 Overview of the Meta-Process
100 A sample quality manual template (6)Milestone Reviews
101 A sample quality manual template (7)Section 5: ISO 9001:2000 to QMS Documentation Traceability Matrix
102 Index Introduction Overview of the ISO 9000 – International Set of Standards for Quality Management Overview of the Capability Maturity Model Integration for Software Engineering (CMMI-SW) Overview of the ISO/IEC TR International Standard for Software Process Assessment How to write a quality manual How to write a quality plan Quality control - measurements and reviews Product certification, warranties and liabilities, and quality aspects in customer support and maintenance services
103 The quality plan A quality plan sets out (within a particular project) the desired product qualities and how these are assessed and define the most significant quality attributes It should define the quality assessment process It should set out which organisational standards should be applied and, if necessary, define new standards Quality plans should be short, succinct documents If they are too long, no-one will read them
104 The quality plan in contextSoftware project (management) plan (note: for many authors, QA V&V) (adapted from: Ilene Burnstein, Practical Software Testing)
105 Index Introduction Overview of the ISO 9000 – International Set of Standards for Quality Management Overview of the Capability Maturity Model Integration for Software Engineering (CMMI-SW) Overview of the ISO/IEC TR International Standard for Software Process Assessment How to write a quality manual How to write a quality plan Quality control - measurements and reviews Product certification, warranties and liabilities, and quality aspects in customer support and maintenance services
106 Quality metrics A software metric is a property of a software product, process or related documentation that takes a numeric value that can be measured Lines of code in a program, number of person-days required to develop a component, etc. We are interested here in measuring (quantifying) the quality of a software product Main difficulty: distance between what we want to know (usually an external quality attribute) and what we are able to measure (usually an internal attribute) higher with static metrics – see next Although some companies have introduced measurement programmes, the systematic use of measurement is still uncommon and there are few standards in this area
107 Types of product metricsDynamic metrics Are collected by measurements made of a program in execution Are closely related to software quality attributes, such as efficiency and reliability It is relatively easy to measure the response time of a system (performance attribute) or the number of failures (reliability attribute) Static metrics Are collected by measurements made of the system representations (source files, documentation, etc.) Have an indirect (and difficult to establish) relationship with quality attributes, such as complexity, understandability and maintainability
108 Examples of static metrics
109 Relationship between static metrics and quality attributesInternal attribute (static metric) External attribute (quality attribute) ?
110 Product measurement processselect metrics collected data
111 Measurement analysis It is not always obvious what data meansAnalysing collected data is very difficult Data analysis must take local circumstances into account Example of measurement surprises: Reducing the number of faults in a program leads to an increased number of help desk calls The program is now thought of as more reliable and so has a wider more diverse market. The percentage of users who call the help desk may have decreased but the total may increase A more reliable system is used in a different way from a system where users work around the faults. This leads to more help desk calls
112 Index Introduction Overview of the ISO 9000 – International Set of Standards for Quality Management Overview of the Capability Maturity Model Integration for Software Engineering (CMMI-SW) Overview of the ISO/IEC TR International Standard for Software Process Assessment How to write a quality manual How to write a quality plan Quality control - measurements and reviews Product certification, warranties and liabilities, and quality aspects in customer support and maintenance services
113 Liability, warranty and certificateLiability - the state of being legally obliged and responsible (http://www.thefreedictionary.com/liability) Legal liability - a term used to describe situations in which a person is liable, for, say, damage to property and is therefore responsible to pay compensation for any damage incurred; liability may be civil or criminal. (http://en.wikipedia.org/wiki/Liability) Warranty - a written assurance that some product or service will be provided or will meet certain specifications (http://www.thefreedictionary.com/warranty ) Certificate - a document attesting to the truth of certain stated facts (http://www.thefreedictionary.com/certificate )
114 Example of a software warrantyMicrosoft Windows XP End-User License Agreement: 11. LIMITED WARRANTY FOR PRODUCT ACQUIRED IN THE US AND CANADA. Microsoft warrants that the Product will perform substantially in accordance with the accompanying materials for a period of ninety days from the date of receipt. (…) If an implied warranty or condition is created by your state/jurisdiction and federal or state/provincial law prohibits disclaimer of it, you also have an implied warranty or condition, BUT ONLY AS TO DEFECTS DISCOVERED DURING THE PERIOD OF THIS LIMITED WARRANTY (NINETY DAYS). (…) Some states/jurisdictions do not allow limitations on how long an implied warranty or condition lasts, so the above limitation may not apply to you. (…) YOUR EXCLUSIVE REMEDY. Microsoft's and its suppliers' entire liability and your exclusive remedy shall be, at Microsoft's option from time to time exercised subject to applicable law, (a) return of the price paid (if any) for the Product, or (b) repair or replacement of the Product, that does not meet this Limited Warranty and that is returned to Microsoft with a copy of your receipt. (..) This Limited Warranty is void if failure of the Product has resulted from accident, abuse, misapplication, abnormal use or a virus.
115 Quality of service Some product-related services and their quality attributes User Training User Help Quick and useful response (avoid “Help does not Help”) Product repair and enhancement, and deployment of new versions Quick and effective repair Conservation qualities: Things that worked well in the old version, continue to work well in the new version (regression tests are very important here), and don’t require new user training Installation of the new version doesn’t cause loss of user data (backward compatibility) Installation of the new version doesn’t require system down for too much time Progress qualities: Things that worked wrong or didn’t work at all in the old version, now work well in the new version, or new useful features have been added
116 References and further reading (1)ISO 9001:2000 Achieving Compliance and Continuous Improvement in Software Development Companies, Vivek Nanda, American Society for Quality (ASQ), Quality Press, 2003 existe na biblioteca da FEUP Ian Sommerville, “Software Engineering”, 6th edition, Addison-Wesley, 2001 ISO 9000 – Series of standards for quality management Capability Maturity Model Integration for Software Engineering (CMMI-SW), Software Engineering Institute (SEI), Carnegie Mellon University (CMU) ISO/IEC TR International Standard for Software Process Assessment SPICE project - Software Process Improvement and Capability dEtermination Model
117 References and further reading (2)ISO/IEC 90003:2004, Software engineering - Guidelines for the application of ISO 9001:2000 to computer software ISO TR 10013: Guidelines for quality management system documentation ISO/IEC 12207:1995 – Software life cycle processes Establishes a system for software life cycle processes with well-defined terminology. Contains processes, activities and tasks that are to be applied during the acquisition of a system that contains software, a stand-alone software product and software services. ISO/IEC – Series of standards on software functional size measurement, ISO/IEC Series of standards on software product evaluation, general overview, planning and management, process for developers, process for acquirers, process for evaluators, documentation of evaluation modules ISO/IEC Series of standards on software product quality metrics – quality model, external metrics, internal metrics, quality in use metrics