Code coverage

Code coverage

Code coverage is a measure used in software testing. It describes the degree to which the source code of a program has been tested. It is a form of testing that inspects the code directly and is therefore a form of white box testing.[1]

Code coverage was among the first methods invented for systematic software testing. The first published reference was by Miller and Maloney in Communications of the ACM in 1963.[2]

Code coverage is one consideration in the safety certification of avionics equipment. The standard by which avionics gear is certified by the Federal Aviation Administration (FAA) is documented in DO-178B.[3]

Contents

Coverage criteria

To measure how well the program is exercised by a test suite, one or more coverage criteria are used.

Basic coverage criteria

There are a number of coverage criteria, the main ones being:[4]

  • Function coverage - Has each function (or subroutine) in the program been called?
  • Statement coverage - Has each node in the program been executed?
  • Decision coverage (not the same as branch coverage.[5]) - Has every edge in the program been executed? For instance, have the requirements of each branch of each control structure (such as in IF and CASE statements) been met as well as not met?
  • Condition coverage (or predicate coverage) - Has each boolean sub-expression evaluated both to true and false? This does not necessarily imply decision coverage.
  • Condition/decision coverage - Both decision and condition coverage should be satisfied.

For example, consider the following C++ function:

int foo(int x, int y)
{
    int z = 0;
    if ((x>0) && (y>0)) {
        z = x;
    }
    return z;
}

Assume this function is a part of some bigger program and this program was run with some test suite.

  • If during this execution function 'foo' was called at least once, then function coverage for this function is satisfied.
  • Statement coverage for this function will be satisfied if it was called e.g. as foo(1,1), as in this case, every line in the function is executed including z = x;.
  • Tests calling foo(1,1) and foo(0,1) will satisfy decision coverage, as in the first case the if condition and the short circuit condition are satisfied and z = x; is executed, and in the second neither conditional is satisfied and x is not assigned to z.
  • Condition coverage can be satisfied with tests that call foo(1,1), foo(1,0) and foo(0,0). These are necessary as in the first two cases (x>0) evaluates to true while in the third it evaluates false. At the same time, the first case makes (y>0) true while the second and third make it false.

In languages, like Pascal, where standard boolean operations are not short circuited, condition coverage does not necessarily imply decision coverage. For example, consider the following fragment of code:

if a and b then

Condition coverage can be satisfied by two tests:

  • a=true, b=false
  • a=false, b=true

However, this set of tests does not satisfy decision coverage as in neither case will the if condition be met.

Fault injection may be necessary to ensure that all conditions and branches of exception handling code have adequate coverage during testing.

Modified condition/decision coverage

For safety-critical applications (e.g., for avionics software) it is often required that modified condition/decision coverage (MC/DC) is satisifed. This criteria extends condition/decision criteria with requirements that each condition should affect the decision outcome independently. For example, consider the following code:

if (a or b) and c then

The condition/decision criteria will be satisfied by the following set of tests:

  • a=true, b=true, c=true
  • a=false, b=false, c=false

However, the above tests set will not satisfy modified condition/decision coverage, since in the first test, the value of 'b' and in the second test the value of 'c' would not influence the output. So, the following test set is needed to satisfy MC/DC:

  • a=false, b=false, c=true
  • a=true, b=false, c=true
  • a=false, b=true, c=true
  • a=true, b=true, c=false

The bold values influence the output, each variable must be present as an influencing value at least once with false and once with true.

Multiple condition coverage

This criteria requires that all combinations of conditions inside each decision are tested. For example, the code fragment from the previous section will require eight tests:

  • a=false, b=false, c=false
  • a=false, b=false, c=true
  • a=false, b=true, c=false
  • a=false, b=true, c=true
  • a=true, b=false, c=false
  • a=true, b=false, c=true
  • a=true, b=true, c=false
  • a=true, b=true, c=true

Other coverage criteria

There are further coverage criteria, which are used less often:

  • Linear Code Sequence and Jump (LCSAJ) coverage - has every LCSAJ been executed?
  • JJ-Path coverage - have all jump to jump paths [6] (aka LCSAJs) been executed?
  • Path coverage - Has every possible route through a given part of the code been executed?
  • Entry/exit coverage - Has every possible call and return of the function been executed?
  • Loop coverage - Has every possible loop been executed zero times, once, and more than once?

Safety-critical applications are often required to demonstrate that testing achieves 100% of some form of code coverage.

Some of the coverage criteria above are connected. For instance, path coverage implies decision, statement and entry/exit coverage. Decision coverage implies statement coverage, because every statement is part of a branch.

Full path coverage, of the type described above, is usually impractical or impossible. Any module with a succession of n decisions in it can have up to 2n paths within it; loop constructs can result in an infinite number of paths. Many paths may also be infeasible, in that there is no input to the program under test that can cause that particular path to be executed. However, a general-purpose algorithm for identifying infeasible paths has been proven to be impossible (such an algorithm could be used to solve the halting problem).[7] Methods for practical path coverage testing instead attempt to identify classes of code paths that differ only in the number of loop executions, and to achieve "basis path" coverage the tester must cover all the path classes.

In practice

The target software is built with special options or libraries and/or run under a special environment such that every function that is exercised (executed) in the program(s) is mapped back to the function points in the source code. This process allows developers and quality assurance personnel to look for parts of a system that are rarely or never accessed under normal conditions (error handling and the like) and helps reassure test engineers that the most important conditions (function points) have been tested. The resulting output is then analyzed to see what areas of code have not been exercised and the tests are updated to include these areas as necessary. Combined with other code coverage methods, the aim is to develop a rigorous, yet manageable, set of regression tests.

In implementing code coverage policies within a software development environment one must consider the following:

  • What are coverage requirements for the end product certification and if so what level of code coverage is required? The typical level of rigor progression is as follows: Statement, Branch/Decision, Modified Condition/Decision Coverage(MC/DC), LCSAJ (Linear Code Sequence and Jump)
  • Will code coverage be measured against tests that verify requirements levied on the system under test (DO-178B)?
  • Is the object code generated directly traceable to source code statements? Certain certifications, (i.e. DO-178B Level A) require coverage at the assembly level if this is not the case: "Then, additional verification should be performed on the object code to establish the correctness of such generated code sequences" (DO-178B) para-6.4.4.2.[3]

Test engineers can look at code coverage test results to help them devise test cases and input or configuration sets that will increase the code coverage over vital functions. Two common forms of code coverage used by testers are statement (or line) coverage and path (or edge) coverage. Line coverage reports on the execution footprint of testing in terms of which lines of code were executed to complete the test. Edge coverage reports which branches or code decision points were executed to complete the test. They both report a coverage metric, measured as a percentage. The meaning of this depends on what form(s) of code coverage have been used, as 67% path coverage is more comprehensive than 67% statement coverage.

Generally, code coverage tools and libraries exact a performance and/or memory or other resource cost which is unacceptable to normal operations of the software. Thus, they are only used in the lab. As one might expect, there are classes of software that cannot be feasibly subjected to these coverage tests, though a degree of coverage mapping can be approximated through analysis rather than direct testing.

There are also some sorts of defects which are affected by such tools. In particular, some race conditions or similar real time sensitive operations can be masked when run under code coverage environments; and conversely, some of these defects may become easier to find as a result of the additional overhead of the testing code.

Software tools

Tools for C / C++

Tools for C# .NET

Tools for Java

Tools for JavaScript

Tools for Perl

  • Devel::Cover is a complete suite for generating code coverage reports in HTML and other formats.

Tools for PHP

  • PHPUnit, also need Xdebug to make coverage reports

Tools for Python

Hardware tools

See also

References

  1. ^ Kolawa, Adam; Huizinga, Dorota (2007). Automated Defect Prevention: Best Practices in Software Management. Wiley-IEEE Computer Society Press. p. 254. ISBN 0470042125. http://www.wiley.com/WileyCDA/WileyTitle/productCd-0470042125.html. 
  2. ^ Joan C. Miller, Clifford J. Maloney (February 1963). "Systematic mistake analysis of digital computer programs". Communications of the ACM (New York, NY, USA: ACM) 6 (2): 58–63. doi:10.1145/366246.366248. ISSN 0001-0782. 
  3. ^ a b RTCA/DO-178B, Software Considerations in Airborne Systems and Equipment Certification, Radio Technical Commission for Aeronautics, December 1, 1992.
  4. ^ Glenford J. Myers (2004). The Art of Software Testing, 2nd edition. Wiley. ISBN 0471469122. 
  5. ^ Position Paper CAST-10 (June 2002). What is a “Decision” in Application of Modified Condition/Decision Coverage (MC/DC) and Decision Coverage (DC)?
  6. ^ M. R. Woodward, M. A. Hennell, "On the relationship between two control-flow coverage criteria: all JJ-paths and MCDC", Information and Software Technology 48 (2006) pp. 433-440
  7. ^ Dorf, Richard C.: Computers, Software Engineering, and Digital Devices, Chapter 12, pg. 15. CRC Press, 2006. ISBN 0849373409, 9780849373404; via Google Book Search

External links


Wikimedia Foundation. 2010.

Игры ⚽ Поможем решить контрольную работу

Look at other dictionaries:

  • Code Coverage — Couverture de code La couverture de code (en anglais code coverage) est une mesure utilisée en génie logiciel pour décrire le taux de code source testé d un programme. Ceci permet de mesurer la qualité des tests effectués. La mesure de ce taux… …   Wikipédia en Français

  • Code Coverage — Die kontrollflussorientierten Testverfahren, auch Überdeckungstests genannt, gehören zu der Gruppe der strukturorientierten Testmethoden. Die kontrollflussorientierten Testverfahren orientieren sich am Kontrollflussgraphen des Programms. Es… …   Deutsch Wikipedia

  • EMMA (code coverage tool) — EMMA is an open source toolkit for measuring and reporting Java code coverage. EMMA is distributed under the terms of Common Public License v1.0.EMMA features at a glance: * EMMA can instrument classes for coverage either offline (before they are …   Wikipedia

  • Code integrity — is a measurement used in software testing. It measures the how high is the source code s quality when it is passed on to the QA, and is affected by how extensively the code was unit tested and integration tested. Code integrity is a combination… …   Wikipedia

  • Coverage — may refer to: Contents 1 Filmmaking 2 Media and journalism 3 Music …   Wikipedia

  • Code of ethics in media — The code of ethics in media was created by a suggestion from the 1947 Hutchins Commission. They suggested that newspapers, broadcasters and journalists had started to become more responsible for journalism and thought they should be held… …   Wikipedia

  • Couverture De Code — La couverture de code (en anglais code coverage) est une mesure utilisée en génie logiciel pour décrire le taux de code source testé d un programme. Ceci permet de mesurer la qualité des tests effectués. La mesure de ce taux implique souvent l… …   Wikipédia en Français

  • Branch Coverage — Die kontrollflussorientierten Testverfahren, auch Überdeckungstests genannt, gehören zu der Gruppe der strukturorientierten Testmethoden. Die kontrollflussorientierten Testverfahren orientieren sich am Kontrollflussgraphen des Programms. Es… …   Deutsch Wikipedia

  • Decision Coverage — Die kontrollflussorientierten Testverfahren, auch Überdeckungstests genannt, gehören zu der Gruppe der strukturorientierten Testmethoden. Die kontrollflussorientierten Testverfahren orientieren sich am Kontrollflussgraphen des Programms. Es… …   Deutsch Wikipedia

  • Path Coverage — Die kontrollflussorientierten Testverfahren, auch Überdeckungstests genannt, gehören zu der Gruppe der strukturorientierten Testmethoden. Die kontrollflussorientierten Testverfahren orientieren sich am Kontrollflussgraphen des Programms. Es… …   Deutsch Wikipedia

Share the article and excerpts

Direct link
Do a right-click on the link above
and select “Copy Link”