Coverage Testing - good and bad¶
- WRITE SOME CODE! (happy)
- Does it work? (sad)
- Shows which lines of code are executing
- How much of your code is covered by your tests?
- Your tests test your product
- Coverage testing tests your tests
- trace.py in standard library
- figleaf by Dr. Titus Brown
$ coverage -e x test_mycode.py arg1 arg2 $ coverage -r -m
Can also annotate source files. Can run as a Nose plugin can run it programmatically:
import coverage coverage.blah()
Good side of things¶
- Statements are marked as executed or not
- window into your code.
- Fine tune by marking clauses to ignore via pragma
- Clauses can be ignored by regex
- Write more tests
Bad: blunt tool¶
- Everything considered important
- Leads to many false alarms
- Excluding code to boost coverage is too easy
- You’ll never come back
- You are only hurting yourself
Goals of coverage measurement¶
- 100% coverage
- Not always possible
- real world issues are thorny. Hardware failures for example
- Practical goals
- more coverage is better
- Actual number doesn’t matter
- False quantifiability: bad! Only cover what matters!
- Use coverage results to understand your code.
100% coverage issue¶
- You can fool yourself into thinking you are bug free.
- Tests can have bugs too!
- Doesn’t deal with path coverage
How coverage works¶
- you can trick the trace function
- Lie to python about where the lines are
- Trace byte codes rater than statements