python-testing/index.html at gh-pages · swcarpentry/python-testing · GitHub

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
<!DOCTYPE html>
<html>
  <head>
    <meta charset="utf-8">
    <meta name="generator" content="pandoc">
    <title>Software Carpentry: Testing</title>
    <link rel="shortcut icon" type="image/x-icon" href="/favicon.ico" />
    <meta name="viewport" content="width=device-width, initial-scale=1.0" />
    <link rel="stylesheet" type="text/css" href="css/bootstrap/bootstrap.css" />
    <link rel="stylesheet" type="text/css" href="css/bootstrap/bootstrap-theme.css" />
    <link rel="stylesheet" type="text/css" href="css/swc.css" />
    <link rel="alternate" type="application/rss+xml" title="Software Carpentry Blog" href="http://software-carpentry.org/feed.xml"/>
    <meta charset="UTF-8" />
    <!-- HTML5 shim, for IE6-8 support of HTML5 elements -->
    <!--[if lt IE 9]>
      <script src="http://html5shim.googlecode.com/svn/trunk/html5.js"></script>
    <![endif]-->
  </head>
  <body class="lesson">
    <div class="container card">
      <div class="banner">
        <a href="http://software-carpentry.org" title="Software Carpentry">
          <img alt="Software Carpentry banner" src="img/software-carpentry-banner.png" />
        </a>
      </div>
      <article>
      <div class="row">
        <div class="col-md-10 col-md-offset-1">
                    <h1 class="title">Testing</h1>
          <section class="prereq panel panel-warning">
<div class="panel-heading">
<h2 id="prerequisites"><span class="glyphicon glyphicon-education"></span>Prerequisites</h2>
</div>
<div class="panel-body">
<p>In this lesson we use a python library called pytest. Basic understanding of python variables and functions are a necessary prerequisite. Some previous experience with the shell is expected, <em>but isn’t mandatory</em>.</p>
</div>
</section>
<section class="getready panel panel-warning">
<div class="panel-heading">
<h2 id="getting-ready"><span class="glyphicon glyphicon-check"></span>Getting ready</h2>
</div>
<div class="panel-body">
<p>Nothing to do: you’re ready to go!</p>
</div>
</section>
<h2 id="topics">Topics</h2>
<ol style="list-style-type: decimal">
<li><a href="01-basics.html">Basics of Testing</a></li>
<li><a href="02-assertions.html">Assertions</a></li>
<li><a href="03-exceptions.html">Exceptions</a></li>
<li><a href="04-units.html">Unit Tests</a></li>
<li><a href="05-pytest.html">Running Tests with pytest</a></li>
<li><a href="06-edges.html">Edge and Corner Cases</a></li>
<li><a href="07-integration.html">Integration and Regression Tests</a></li>
<li><a href="08-ci.html">Continuous Integration</a></li>
<li><a href="09-tdd.html">Test Driven Development</a></li>
<li><a href="10-fixtures.html">Fixtures</a></li>
</ol>
<p>Before relying on a new experimental device, an experimental scientist always establishes its accuracy. A new detector is calibrated when the scientist observes its responses to to known input signals. The results of this calibration are compared against the <em>expected</em> response. <strong>An experimental scientist would never conduct an experiment with uncalibrated detectors - the that would be unscientific. So too, simulations and analysis with untested software do not constitute science.</strong></p>
<aside class="callout panel panel-info">
<div class="panel-heading">
<h2 id="how-do-you-know-your-code-is-right"><span class="glyphicon glyphicon-pushpin"></span>How Do You Know Your Code is Right?</h2>
</div>
<div class="panel-body">
<p><strong>You can only know by testing it.</strong> Software bugs are hiding in all nontrivial software. Testing is the process by which those bugs are systematically exterminated before they have a chance to cause a paper retraction. In software tests, just like in device calibration, expected results are compared with observed results in order to establish accuracy.</p>
</div>
</aside>
<p>The collection of all of the tests for a given code is known as the <em>test suite</em>. You can think of the test suite as a bunch of pre-canned experiments that anyone can run. If all of the test pass, then the code is at least partially trustworthy. If any of the tests fail then the code is known to be incorrect with respect to whichever case failed. After this lesson, you will know to not trust software when its tests do not <em>cover</em> its claimed capabilities and when its tests do not pass.</p>
<aside class="callout panel panel-info">
<div class="panel-heading">
<h2 id="but-how-close-is-close-enough"><span class="glyphicon glyphicon-pushpin"></span>But How Close is Close Enough?</h2>
</div>
<div class="panel-body">
<p>In the same way that your scientific domain has expectations concerning experimental accuracy, it likely also has expectations concerning allowable computational accuracy. These considerations should surely come into play when you evaluate the acceptability of your own or someone else’s software.</p>
</div>
</aside>
<p>In most other programming endeavors, if code is fundamentally wrong - even for years at a time - the impact of this error can be relatively small. Perhaps a website goes down, or a game crashes, or a days worth of writing is lost to a bug in your word processor. Scientific code, on the other hand, controls planes, weapons systems, satellites, agriculture, and most importantly scientific simulations and experiments. If the software that governs the computational or physical experiment is wrong, then disasters (such as false claims in a publication) will result.</p>
<p>This is not to say that scientists have a monopoly on software testing, simply that software cannot be called <em>scientific</em> unless it has been validated.</p>
<aside class="callout panel panel-info">
<div class="panel-heading">
<h2 id="why-untested-code-is-legacy-code"><span class="glyphicon glyphicon-pushpin"></span>Why Untested Code is Legacy Code</h2>
</div>
<div class="panel-body">
<p>In <em><a href="http://www.amazon.com/Working-Effectively-Legacy-Michael-Feathers/dp/0131177052/">Working Effectively with Legacy Code</a></em>, Michael Feathers defines legacy code as “any code without tests”. This definition draws on the fact that after its initial creation, tests provide a powerful guide to other developers (and to your forgetful self, a few months in the future) about how each function is meant to be used. Without runnable tests to provide examples of code use, even brand new programs are unsustainable.</p>
</div>
</aside>
<p>Testing is the calibration step of the computational simulation and analysis world: it lets the scientist trust their own work on a fundamental level and helps others to understand and trust their work as well. Furthermore, tests help you to never fix a bug a second time. Once a bug has been caught and a test has been written, that particular bug can never again re-enter the codebase unnoticed. So, whether motivated by principles or a desire to work more efficiently, all scientists can benefit from testing.</p>
<aside class="callout panel panel-info">
<div class="panel-heading">
<h2 id="effective-computation-in-physics"><span class="glyphicon glyphicon-pushpin"></span>Effective Computation In Physics</h2>
</div>
<div class="panel-body">
<p>Note that this testing lesson was adapted from the Testing chapter in <em><a href="http://physics.codes">Effective Computation In Physics</a></em> by Anthony Scopatz and Kathryn Huff. It is often quoted directly.</p>
</div>
</aside>
<h2 id="other-resources">Other Resources</h2>
<ul>
<li><a href="reference.html">Reference</a></li>
<li><a href="discussion.html">Discussion</a></li>
<li><a href="instructors.html">Instructor’s Guide</a></li>
</ul>
        </div>
      </div>
      </article>
      <div class="footer">
        <a class="label swc-blue-bg" href="http://software-carpentry.org">Software Carpentry</a>
        <a class="label swc-blue-bg" href="https://github.com/swcarpentry/lesson-template">Source</a>
        <a class="label swc-blue-bg" href="mailto:admin@software-carpentry.org">Contact</a>
        <a class="label swc-blue-bg" href="LICENSE.html">License</a>
      </div>
    </div>
    <!-- Javascript placed at the end of the document so the pages load faster -->
    <script src="http://software-carpentry.org/v5/js/jquery-1.9.1.min.js"></script>
    <script src="css/bootstrap/bootstrap-js/bootstrap.js"></script>
  </body>
</html>