Coverage Testing - good and bad

  • WRITE SOME CODE! (happy)
  • Does it work? (sad)

Coverage Measurement

  • Shows which lines of code are executing
  • How much of your code is covered by your tests?
  • Your tests test your product
  • Coverage testing tests your tests

Coverage tools

  • in standard library
  • figleaf by Dr. Titus Brown
  • coverage

Running coverage


$ coverage -e x arg1 arg2
$ coverage -r -m

Can also annotate source files. Can run as a Nose plugin can run it programmatically:

import coverage

Good side of things

  • Statements are marked as executed or not
  • window into your code.
  • Fine tune by marking clauses to ignore via pragma
  • Clauses can be ignored by regex
  • Write more tests

Bad: blunt tool

  • Everything considered important
  • Leads to many false alarms
  • Excluding code to boost coverage is too easy
    • Tempting
    • You’ll never come back
    • You are only hurting yourself

Goals of coverage measurement

  • 100% coverage
    • ideal
    • Not always possible
    • real world issues are thorny. Hardware failures for example
  • Practical goals
    • more coverage is better
    • Actual number doesn’t matter
    • False quantifiability: bad! Only cover what matters!
    • Use coverage results to understand your code.

100% coverage issue

  • You can fool yourself into thinking you are bug free.
  • Tests can have bugs too!
  • Doesn’t deal with path coverage

How coverage works

  • sys.stack_trace()
  • you can trick the trace function
  • Lie to python about where the lines are
  • Trace byte codes rater than statements