Metrics and Models in Software Testing

poetic rhythm and Models in package running playingHow do we treasure the progress of interrogation? When do we release the softw ar system? Why do we devote to a greater extent conviction and re characters for evidenceing a particular faculty? What is the dependableness of packet at the while of release? Who is responsible for the filling of a poor mental exam suite? How m both work shifts do we c at onceptualise during test? How much meter and re extensions be undeniable to test a softw atomic subject 18 system? How do we raise up along the potential of test suite? We whitethorn keep on framing much(prenominal)(prenominal)(prenominal) questions without much driving? However, conclusion answers to such questions be not easy and whitethorn subscribe epochal amount of effort. Softwargon testing cargonfuls whitethorn avail us to invoice and quantify many things which whitethorn pit approximately answers to such important questions.10.1 Softwargo n rhythmic patternWhat targetnot be visord, piece of tailnot be controlled is a mankind in this world. If we want to control something we should first be able to bar it. Therefore, boththing should be measurable. If a thing is not measurable, we should win an effort to make it measurable. The argona of measurement is truly important in e really subject field and we take for mature and establish poetic rhythm to quantify respective(a) things. However, in software engineering this area of measurement is still in its evolution stage and may regard signifi stackt effort to make it mature, scientific and effective.10.1.1 Measure, Measurement and MetricsThese terms are often used interchangeably. However, we should understand the difference amongst these terms. Pressman explained this clearly as PRES05A measure provides a quantitative indication of the extent, amount, dimension, capacity or size of some attri entirelyes of a mathematical product or movement. Measuremen t is the act of determining a measure. The metric unit is a quantitative measure of the degree to which a product or butt possesses a presumption attribute. For example, a measure is the list of strokes undergo during testing. Measurement is the way of recording such misfortunes. A software metric may be add up way out of strokes experient per bit during testing.Fenton FENT04 has define measurement asIt is the process by which scraps or symbols are assigned to attributes of entities in the real world in such a way as to c all(prenominal) them according to clearly outlined rules.The grassroots issue is that we want to measure every attribute of an entity. We should embed out established prosody to do so. However, we are in the process of evolution poetic rhythm for many attributes of various entities used in software engineering.Software rhythmic pattern can be specify as GOOD93 The continuous finishing of measurement found techniques to the software growth p rocess and its products to supply purposeful and sequencely management training, together with the use of those techniques to improve that process and its products. umteen things are covered in this definition. Software rhythmic pattern are think to measures which, in turn, involve issue forths for quantification, these functions are used to take a crap better product and improve its related process. We may handle to measure tonicity attributes such as testability, complexity, dependableness, maintainability, expertness, portability, enhanceability, usability etc.for a software. We may besides want to measure size, effort, development season and re reference points for a software.10.1.2 ApplicationsSoftware inflection are applicable in all phases of software development life cycle. In software requirements and analysis phase, where output is the SRS entry, we may pose to estimate the cost, workforce requirement and development age for the software. The client ma y homogeneous to contend cost of the software and development age before signing the contract. As we all know, the SRS document acts as a contract amidst customer and developer. The readability and effectiveness of SRS document may swear out to accession the assumption level of the customer and may provide better foundations for designing the product. rough prosody are acquirable for cost and size estimation interchangeable COCOMO, Putnam resource allocation imitate, function point estimation manakin etc. rough inflection are excessively available for the SRS document equal usual figure of mistakes found during verification, change request frequency, readability etc. In the design phase, we may standardized to measure stability of a design, coupling amongst modules, cohesion of a module etc. We may also handle to measure the amount of entropy input to a software, processed by the software and also produced by the software. A count of the amount of info input t o, processed in, and output from software is called a info social organisation metric. many another(prenominal) such metrics are available like hail of variables, anatomy of operators, material body of operands, play of live variables, variable spans, module answerlessness etc. Some information flow metrics are also fashionable like FANIN, FAN OUT etc.Use compositors cases may also be used to design metrics like tally actors, counting use cases, counting crook of links etc. Some metrics may also be knowing for various applications of websites like sum up of static web pages, number of dynamic web pages, number of internal page links, word count, number of static and dynamic kernel objects, while taken to search a web page and come up the desired information, similarity of web pages etc. Software metrics render number of applications during murder phase and after the completion of such a phase. Halstead software size measures are applicable after coding like tok en count, course length, course of instruction volume, program level, difficulty, estimation of sequence and effort, style level etc. Some complexity measures are also prevalent like cyclomatic complexity, knot count, feature count etc. Software metrics have found good number of applications during testing. One area is the dependableness estimation where popular models are Musas basic death penalty succession model and Logarithmic Poisson action time model. Jelinski Moranda model JELI72 is also used for the calculation of reliableness. Source code coverage metrics are available that consider the region of source code covered during testing. Test suite effectiveness may also be metrical. look of mischances experient per unit of time, number of paths, number of independent paths, number of du paths, percentage of statement coverage, percentage of break condition covered are also recyclable software metrics. sustenance phase may have many metrics like number of faults reported per year, number of requests for changes per year, percentage of source code modified per year, percentage of obsolete source code per year etc.We may puzzle number of applications of software metrics in every phase of software development life cycle. They provide meaningful and timely information which may help us to take corrective actions as and when postulate. Effective implementation of metrics may improve the timbre of software and may help us to deliver the software in time and within budget.10.2 Categories of MetricsThere are two broad categories of software metrics to wit product metrics and process metrics. Product metrics describe the characteristics of the product such as size, complexity, design features, performance, aptitude, reliability, portability, etc. Process metrics describe the effectiveness and whole gradation of the processes that produce the software product. Examples are effort inevitable in the process, time to produce the product, effect iveness of break removal during development, number of defects found during testing, maturity of the process AGGA08.10.2.1 Product metrics for testingThese metrics provide information about the testing status of a software product. The info for such metrics are also generated during testing and may help us to know the quality of the product. Some of the basic metrics are devoten as(i) result of ill lucks experient in a time legal separation(ii) Time interval between mishaps(iii) Cumulative misadventures experienced upto a specified time(iv) Time of visitation(v) Estimated time for testing(vi) Actual testing timeWith these basic metrics, we may find some surplus metrics as devoted below(i)(ii) Average time interval between misfortunes(iii) upper limit and minimum failures experienced in any time interval(iv) Average number of failures experienced in time intervals(v) Time remaining to complete the testing.We may design similar metrics to find the indications about the q uality of the product.10.2.2 Process metrics for testingThese metrics are developed to monitor the progress of testing, status of design and development of test cases and outcome of test cases after murder.Some of the basic process metrics are assumption below(i) subprogram of test cases designed(ii) Number of test cases pull throughd(iii) Number of test cases passed(iv) Number of test cases failed(v) Test case performance time(vi) innate consummation time(vii) Time spent for the development of a test case(viii) centre time spent for the development of all test casesOn the posterior of above direct measures, we may design by-line additional metrics which may convert the base metric data into more(prenominal) useful information.(i) % of test cases executed(ii) % of test cases passed(iii) % of test cases failed(iv) rack up actual execution time / score estimated execution time(v) Average execution time of a test caseThese metrics, although simple, may help us to know the progress of testing and may provide meaningful information to the testers and reckon manager.An effective test plan may force us to fix data and convert it into useful metrics for process and product both. This document also guides the organization for future projects and may also suggest changes in the existing processes in order to produce a good quality maintainable software product.10.3 Object Oriented Metrics used in TestingObject oriented metrics capture many attributes of a software and some of them are relevant in testing. Measuring structural design attributes of a software system, such as coupling, cohesion or complexity, is a promising approach towards early quality assessments. There are several metrics available in the literature to capture the quality of design and source code.10.3.1 Coupling MetricsCoupling similaritys increase complexity, reduce encapsulation, potential reuse, and limit understanding and maintainability. The coupling metrics requires information about attribute usage and method invocations of opposite circlees. These metrics are disclosen in table 10.1. Higher nurtures of coupling metrics indicate that a carve up under test will require more number of stubs during testing. In addition, each interface will require to be tested thoroughly.MetricDefinitionSourceCoupling between Objects. (CBO)CBO for a class is count of the number of other classes to which it is coupled.CHID94Data stimulus generalisation Coupling (DAC)Data Abstraction is a technique of creating new data tokens suited for an application to be programmed.DAC = number of ADTs defined in a class.LI93Message Passing Coupling. (MPC)It counts the number of send statements defined in a class.Response for a Class (RFC)It is defined as set of methods that can be potentially executed in response to a pass along received by an object of that class. It is presumption byRFC=RS, where RS, the response set of the class, is granted byCHID94Information flow-establish c oupling (ICP)The number of methods invoked in a class, weighted by the number of parameters of the methods invoked.LEE95Information flow-establish inheritance coupling. (IHICP)Same as ICP, but only counts methods invocations of ancestors of classes.Information flow-based non-inheritance coupling (NIHICP)Same as ICP, but only counts methods invocations of classes not related through inheritance.Fan-inCount of modules (classes) that call a stipulation class, plus the number of global data elements.BINK98Fan-outCount of modules (classes) called by a given module plus the number of global data elements altered by the module (class).BINK98Table 10.1 Coupling Metrics10.3.3 heritage MetricsInheritance metrics requires information about ancestors and descendants of a class. They also collect information about methods overridden, contractable and added (i.e. neither inherited nor overrided). These metrics are summarized in table 10.3. If a class has more number of children (or sub classes) , more amount of testing may be required in testing the methods of that class. More is the depth of inheritance tree, more complex is the design as more number of methods and classes are involved. Thus, we may test all the inherited methods of a class and testing effort well increase accordingly.MetricDefinitionSourcesNumber of Children (NOC)The NOC is the number of nimble subclasses of a class in a hierarchy.CHID94Depth of Inheritance head (DIT)The depth of a class within the inheritance hierarchy is the upper limit number of steps from the class node to the root of the tree and is measurable by the number of ancestor classes.Number of Parents (NOP)The number of classes that a class directly inherits from (i.e. multiple inheritance).LORE94Number of Descendants (NOD)The number of subclasses (both direct and indirectly inherited) of a class.Number of Ancestors (NOA)The number of superclasses (both direct and indirectly inherited) of a class.TEGA92Number of Methods Overridden (NMO) When a method in a subclass has the same name and type signature as in its superclass, then the method in the superclass is give tongue to to be overridden by the method in the subclass.LORE94Number of Methods Inherited (NMI)The number of methods that a class inherits from its super (ancestor) class.Number of Methods Added (NMA)The number of new methods added in a class (neither inherited, nor overriding).Table 10.3 Inheritance Metrics10.3.4 Size MetricsSize metrics indicate the length of a class in terms of lines of source code and methods used in the class. These metrics are given in table 10.4. If a class has more number of methods with greater complexity, then more number of test cases will be required to test that class. When a class with more number of methods with greater complexity is inherited, it will require more rigorous testing. Similarly, a class with more number of public methods will require thorough testing of public methods as they may be used by other classes.Met ricDefinitionSourcesNumber of Attributes per Class (NA)It counts the enumerate number of attributes defined in a class.Number of Methods per Class (NM)It counts number of methods defined in a class.Weighted Methods per Class (WMC)The WMC is a count of sum of complexities of all methods in a class. Consider a class K1, with methods M1,.. Mn that are defined in the class. Let C1,.Cn be the complexity of the methods.CHID94Number of public methods (PM)It counts number of public methods defined in a class.Number of non-public methods (NPM)It counts number of private methods defined in a class.Lines Of Code (LOC)It counts the lines in the source code. Table 10.4 Size Metrics10.4 What should we measure during testing?We should measure every thing (if possible) which we want to control and which may help us to find answers to the questions given in the beginning of this chapter. Test metrics may help us to measure the current performance of any project. The collected data may become histor ical data for future projects. This data is very important because in the absence of historical data, all estimates are middling the guesses. Hence, it is essential to record the key information about the current projects. Test metrics may become an important indicator of the effectiveness and efficiency of a software testing process and may also notice risky areas that may need more testing.10.4.1 TimeWe may measure many things during testing with respect to time and some of them are given as1) Time required to run a test case.2) Total time required to run a test suite.3) Time available for testing4) Time interval between failures5) Cumulative failures experienced upto a given time6) Time of failure7) nonstarters experienced in a time intervalA test case requires some time for its execution. A measurement of this time may help to estimate the total time required to execute a test suite. This is the simplest metric and may estimate the testing effort. We may calculate the time av ailable for testing at any point in time during testing, if we know the total allotted time for testing. Generally unit of time is seconds, minutes or hours, per test case. Total testing time may be defined in terms of hours. Time needed to execute a plotted test suite may also be defined in terms of hours.When we test a software, we experience failures. These failures may be record in different ways like time of failure, time interval between failures, cumulative failures experienced upto given time and failures experienced in a time interval. Consider the table 10.5 and table 10.6 where time based failure specification and failure based failure specification are givenSr. No. of failure occurrencesFailure time measured in minutesFailure intervals in minutes1121222614335094380355012670207106368125199 peerless hundred fifty-five301020045Table 10.5 Time based failure specificationTime in minutesCumulative failuresFailures in interval of 20 minutes20010140040360050180060110006001200701 1400801160090118009002001001Table 10.6 Failure based failure specificationThese two tables give us the idea about failure pattern and may help us to define the following1) Time taken to experience n failures2) Number of failures in a particular time interval3) Total number of failures experienced after a specified time4) Maximum / minimum number of failures experienced in any well-ordered time interval.10.4.2 Quality of source codeWe may know the quality of the delivered source code after reasonable time of release employ the following formulaWhere WDB Number of weighted defects found before releaseWDA Number of weighted defects found after releaseThe weight for each defect is defined on the basis of defect abrasiveness and removal cost. A severity is assigned to each defect by testers based on how important or right is the defect. A lower value of this metric indicates the less number of error detection or less serious error detection.We may also calculate the number of defects per execution test case. This may also be used as an indicator of source code quality as the source code progressed through the series of test activities STEP03.10.4.3 Source Code CoverageWe may like to execute every statement of a program at least once before its release to the customer. Hence, percentage of source code coverage may be calculated asThe higher value of this metric given confidence about the effectiveness of a test suite. We should write additional test cases to cover the uncovered portions of the source code.10.4.4 Test Case faulting DensityThis metric may help us to know the efficiency and effectiveness of our test cases.Where Failed test case A test case that when executed, produced an undesired output.Passed test case A test case that when executed, produced a desired outputHigher value of this metric indicates that the test cases are effective and efficient because they are able to detect more number of defects.10.4.5 Review EfficiencyReview efficiency is a m etric that gives brainstorm on the quality of review process carried out during verification.Higher the value of this metric, better is the review efficiency.10.5 Software Quality Attributes Prediction ModelsSoftware quality is dependent on many attributes like reliability, maintainability, fault proneness, testability, complexity, etc. Number of models are available for the yellion of one or more such attributes of quality. These models are especially beneficial for large-scale systems, where testing experts need to tenseness their attention and resources to problem areas in the system under development.10.5.1 Reliability ModelsMany reliability models for software are available where emphasis is on failures instead than faults. We experience failures during execution of any program. A fault in the program may lead to failure(s) depending upon the input(s) given to a program with the purpose of execution it. Hence, time of failure and time between failures may help us to find r eliability of software. As we all know, software reliability is the fortune of failure free operation of software in a given time under specified conditions. Generally, we consider the calendar time. We may like to know the probability that a given software will not fail in one month time or one week time and so on. However, most of the available models are based on execution time. The execution time is the time for which the computer actually executes the program. Reliability models based on execution time normally give better results than those based on calendar time. In many cases, we have a mapping table that converts execution time to calendar time for the purpose of reliability studies. In order to differentiate both the timings, execution time is represented byand calendar time by t.Most of the reliability models are applicable at system testing level. Whenever software fails, we note the time of failure and also try to locate and correct the fault that caused the failure. D uring system testing, software may not fail at regular intervals and may also not follow a particular pattern. The fun in time between successive failures may be exposit in terms of following functions () average number of failures upto time () average number of failures per unit time at time and is known as failure impregnation function.It is expect that the reliability of a program increases collectible to fault detection and correction over time and hence the failure military strength minifys accordingly.(i) Basic Execution Time ModelThis is one of the popular model of software reliability assessment and was developed by J.D. genus Musa MUSA79 in 1979. As the name indicates, it is based on execution time (). The basic assumption is that failures may occur according to a non-homogeneous poisson process (NHPP) during testing. Many examples may be given for real world events where poisson processes are used. Few examples are given as* Number of users using a website in a giv en period of time.* Number of persons requesting for railway tickets in a given period of time* Number of e-mails expected in a given period of time.The failures during testing represents a non-homogeneous process, and failure frenzy decreases as a function of time. J.D. Musa assumed that the decrease in failure excitement as a function of the number of failures observed, is eonian and is given asWhere Initial failure rapture at the belong of testing. Total number of failures experienced upto infinite time Number of failures experienced upto a given point in time.Musa MUSA79 has also given the kinship between failure passion () and the mean failures experienced () and is given in 10.1.If we take the first derivative of compare given above, we get the slope of the failure strong point as given belowThe blackball sign shows that there is a negative slope indicating a decreaseing panache in failure eagerness.This model also assumes a uniform failure pattern meaning thereby equal probability of failures due to various faults. The kin between execution time () and mean failures experienced () is given in 10.2The derivation of the relationship of 10.2 may be obtained asThe failure intensity as a function of time is given in 10.3.This relationship is useful for calculating present failure intensity at any given value of execution time. We may find this relationshipTwo additional equations are given to calculate additional failures required to be experienced to reach a failure intensity intention (F) and additional time required to reach the physical object. These equations are given as Where Expected number of additional failures to be experienced to reach failure intensity object glass. Additional time required to reach the failure intensity objective. Present failure intensity Failure intensity objective. and are very interesting metrics to know the additional time and additional failures required to get a failure intensity objective.Example 10.1 A program will experience 100 failures in infinite time. It has now experienced 50 failures. The initial failure intensity is 10 failures/hour. Use the basic execution time model for the following(i) Find the present failure intensity.(ii) Calculate the decrement of failure intensity per failure.(iii) Determine the failure experienced and failure intensity after 10 and 50 hours of execution.(iv) Find the additional failures and additional execution time needed to reach the failure intensity objective of 2 failures/hour.Solution(a) Present failure intensity can be calculated using the following equation(b) reduction of failure intensity per failure can be calculated using the following(c) Failures experienced and failure intensity after 10 and 50 hours of execution can be calculated as(i) After 10 hours of execution(ii) After 50 hours of execution(d) and with failure intensity objective of 2 failures/hour(ii) Logarithmic Poisson Execution time modelWith a small modification in the failure intensity function, Musa presented logarithmic poisson execution time model. The failure intensity function is given asWhere Failure intensity decay parameter which represents the relative change of failure intensity per failure experienced.The slope of failure intensity is given asThe expected number of failures for this model is always infinite at infinite time. The relation for mean failures experienced is given asThe expression for failure intensity with respect to time is given asThe relationship for additional number of failures and additional execution time are given asWhen execution time is more, the logarithmic poisson model may give large value of failure intensity than the basic model.Example 10.2 The initial failure intensity of a program is 10 failures/hour. The program has experienced 50 failures. The failure intensity decay parameter is 0.01/failure. Use the logarithmic poisson execution time model for the following(a) Find present failure intensity.(b) Calcu late the decrement of failure intensity per failure.(c) Determine the failure experienced and failure intensity after 10 and 50 hours of execution.(d) Find the additional failures and additional and failure execution time needed to reach the failure intensity objective of 2 failures/hour.Solution(a) Present failure intensity can be calculated as= 50 failures= 50 failures= 0.01/faluresHence= 6.06 failures/hour(b) Decrement of failure intensity per failure can be calculated as(c) Failure experienced and failure intensity after 10 and 50 hours of execution can be calculated as(i) After 10 hours of execution(ii) After 50 hours of execution(d) and with failure intensity objective of 2 failures/hour(iii) The Jelinski Moranda ModelThe Jelinski Moranda model JELI72 is the earliest and simples software reliability model. It proposed a failure intensity function in the form ofWhere = perpetual of proportionalityN = total number of errors presenti = number of errors found by time interval t i.This model assumes that all failures have the same failure rate. It means that failure rate is a step function and there will be an improvement in reliability after fixing an error. Hence, every failure contributes equally to the overall reliability. Here, failure intensity is directly proportional to the number of errors remaining in a software.Once we know the value of failure intensity function using any reliability model, we may calculate reliability using the equation given belowWhere is the failure intensity and t is the operating time. dismount the failure intensity and higher is the reliability and vice versa.Example 10.3 A program may experience 200 failures in infinite time of testing. It has experienced 100 failures. Use Jelinski-Moranda model to calculate failure intensity after the experience of 150 failures?SolutionTotal expected number of failures (N) = 200Failures experienced (i) =100Constant of proportionality () = 0.02We know= 2.02 failures/hourAfter 150 failur es= 0.02 (200-150+1)=1.02 failures/hourFailure intensity will decrease with every additional failure experience.10.5.2 An example of fault prediction model in practiceIt is clear that software metrics can be used to capture the quality of object oriented design and code. These metrics provide ways to evaluate the quality of software and their use in earlier phases of software development can help organizations in assessing a large software development quickly, at a low cost.To achieve help for planning and executing testing by focusing resources on the fault prone parts of the design and code, the model used to predict faulty classes should be used. The fault prediction model can also be used to identify classes that are prone to have everlasting(a) faults. One can use this model with respect to high severity of faults to focus the testing on those parts of the system that are liable(predicate) to cause serious failures. In this section, we describe models used to find relationshi p between object oriented metrics and fault proneness, and how such models can be of great help in planning and executing testing activities MALH09, SING10.In order to perform the analysis we used public land KC1 NASA data set NASA04 The data set is available on www.mdp.ivv.nasa.gov. The cxlv