2 Marco Baringer <mb@bese.it>
8 :website: http://common-lisp.net/project/fiveam
9 :stylesheet: fiveam.css
14 === The Super Brief Introduction ===
16 |================================
17 | (xref:OP_DEF-TEST[`def-test`] `NAME` () &body `BODY`) | define tests
18 | (xref:OP_IS[`is`] (`PREDICATE` `EXPECTED` `ACTUAL`)) | check that, according to `PREDICATE` our `ACTUAL` is the same as our `EXPECTED`
19 | (xref:OP_IS[`is-true`] VALUE) | check that a value is non-NIL
20 | (xref:OP_RUN![`run!`] TEST-NAME) | run one (or more) tests and print the results
21 |================================
23 See the xref:API_REFERENCE[api] for details.
25 === An Ever So Slightly Longer Introduction ===
27 You use define some xref:TESTS[tests] (using
28 xref:OP_DEF-TEST[`def-test`]), each of which consists of some
29 xref:CHECKS[checks] (with xref:OP_IS[`is`] and friends) which can pass
32 --------------------------------
35 (is-false (= 5 (+ 2 2))))
36 --------------------------------
38 you xref:RUNNING_TESTS[run] some tests (using xref:OP_RUN[run] and
39 friends) and you look at the results (using using
40 xref:OP_EXPLAIN[explain]); or you do both at once (using
41 xref:OP_RUN-EPOINT-[run!]):
43 --------------------------------
44 CL-USER> (run! 'a-test)
50 --------------------------------
52 Lather, rinse, repeat:
54 --------------------------------
61 --------------------------------
63 === The Real Introduction ===
65 FiveAM is a testing framework, this is a rather vague concept, so
66 before talking about how to use FiveAM it's worth knowing what task(s)
67 FiveAM was built to do and, in particular, which styles of testing
68 FiveAM was designed to facilitate:
70 `test driven development`:: sometimes you know what you're trying to
71 do (lucky you) and you can figure out what your code should do
72 before you've written the code itself. The idea here is that you
73 write a bunch of tests and when all these test pass your code is
76 `interactive testing`:: sometimes as you're writing code you'll see
77 certain constraints that your code has to meet. For example you'll
78 realize there's a specific border case your code, which you're
79 probably not even done writing, has to deal with. In this work flow
80 you'll write code and tests more or less simultaneously and by the
81 time you're satisfied that your code does what it should you'll have
82 a set of tests which prove that it does what you think it does.
84 `regression testing`:: sometimes you're pretty confident, just by
85 looking at the code, that your program does what it should, but you
86 want an automatic way to make sure that it continues to do what it
87 does even if (when) you change other parts of the code.
90 There's also `beaviour driven development`. this works under
91 the assumption that you can write tests in a natural-ish lanugage and
92 they'll be easier to maintain than tests writen in code (have we
93 learned nothing from cobol?). FiveAM does not, in its current
94 implementation, support link:http://cukes.info/[cucumber] like
95 behaviour driven development. patches welcome (they'll get laughed at
96 at first, but they'll get applied, and then they'll get used, and then
97 they'll be an essential part of fiveam itself...)
101 Since there are far many more testing frameworks than there are words
102 for talking about testing frameworks, the same words end up meaning
103 different things in different frameworks. Just to be clear, here are
104 the words fiveam uses:
106 `check`:: a single expression which has an expected value.
108 `test`:: a set of checks which we want to always run together.
110 `suite`:: a group of tests we often want to run all at once.
115 Tests are created with the xref:OP_DEF-TEST[`def-test`] macro and
120 Because everything deserves a name. Names in FiveAM are symbols (or
121 anything that can be sensibly put in an `eql` hash table) and they are
122 used both to select which test to run (as arguments to `run!` and
123 family) and when reporting test failures.
127 Every test has a function which is the actual code that gets executed
128 when the test is run. This code, whatever it is, will, bugs aside,
129 xref:CHECKS[create a set of test result objects] (failures, successes
130 and skips) and store these in a few dynamic variables (you don't need
131 to worry about those).
133 The body is actually the only real part of the test, everything else
134 is administrativia. Sometimes usefel administrativia, but none the
139 Generally speaking you'll have so many tests that you'll not want to
140 run them all every single time you need to run one of them (automated
141 regression testing is another use case). Tests can be grouped into
142 suites, and suites can also be grouped into suites, and suites have
143 names, so by specfying the name of a suite we only run those tests
144 that are a part of that suite.
146 Unless otherwise specified tests add themselves to the xref:THE_CURRENT_SUITE[current suite].
148 There are two other properties, also set via parameters to
149 xref:OP_DEF-TEST[`def-test`], which influence how the tests are
152 When to compile the test::
154 Often enough, when working with lisp macros especially, it's useful to
155 delay compilation of the test's body until the test is run. A useful
156 side effect of this delay is that the code will be recompiled every
157 time its run, so if the macro definition has changed that will be
158 picked up at the next run of the test. While this is the default mode
159 of operation for FiveAM it can be turned off and tests will be
160 compiled at the 'normal' time (when the enclosing def-test form is
163 Whether to run the test at all::
165 Sometimes, but far less often than the designer of FiveAM expected,
166 it's useful to run a test only when some other test passes. The
167 assumption being that if the lower level tests have failed there's no
168 point in cluttering up the output by running the higher level tests as
171 YMMV. (i got really bad mileage out of this feature)
176 At the heart of every test is something which compares the result of
177 some code to some expected value, in FiveAM these are called
178 checks. All checks in FiveAM do something, exactly what depends on the
179 check, and then either:
181 . generate a "this check passed" result
183 . generate a "this check failed" result and a corresponding failure
186 . generate a "for some reason this check was skipped" result.
188 All checks take, as an optional argument, so called "reason format
189 control arguments." Should the check fail (or be skipped) these
190 arguments will be passed to format, via something like `(curry
191 #'format nil)`, and the result will be used as the
192 explanation/description of the failure.
194 When it comes to the actual check functions themeselves, there are
197 . xref:CHECKING_RETURN_VALUES[those that take a value and compare it
200 . xref:CHECKING_CONTROL_FLOW[those that make sure the program's
201 execution takes, or does not take, a certain path]
203 . xref:ARBITRARY_CHECK_RESULTS[those that just force a success or
204 failure to be recorded].
206 [[CHECKING_RETURN_VALUES]]
207 === Checking return values ===
209 xref:OP_IS[`IS`], xref:OP_IS-TRUE[`IS-TRUE`],
210 xref:OP_IS[`IS-FALSE`] will take one form and compare its return
211 value to some known value (the so called expected vaule) and report an
212 error if these two are not equal.
214 --------------------------------
215 ;; Pass if (+ 2 2) is = to 5
217 ;; Pass if (zerop 0) is not-NIL
219 ;; Pass if (zerop 1) is NIL
221 --------------------------------
223 Often enough we want to test a set of expected values against a set of
224 test values using the same operator. If, for example, we were
225 implementing a string formatting functions, then `IS-EVERY` provides a
226 concise way to line up N different inputs along with their expected
227 outputs. For example, let's say we were testing `cl:+`, we could setup
228 a list of tests like this:
230 --------------------------------
231 (is-every #'= (5 (+ 2 2))
236 --------------------------------
238 We'd do this instead of writing out 5 seperate `IS` or `IS-TRUE`
241 [[CHECKING_CONTROL_FLOW]]
242 === Checking control flow ===
244 xref:OP_SIGNALS[`SIGNALS`] and xref:OP_FINISHES[`FINISHES`] create
245 pass/fail results depending on whether their body code did or did not
248 Both of these checks assume that there is a single block of code and
249 it either runs to completion or it doesn't. Sometimes though the logic
250 is more complex and you can't easily represent it as a single progn
251 with a flag at the end. See xref:ARBITRARY_CHECK_RESULTS[below].
253 [[ARBITRARY_CHECK_RESULTS]]
254 === Recording arbitrary test results ===
256 Very simply these three checks, xref:OP_PASS[`PASS`],
257 xref:OP_FAIL[`FAIL`] and xref:OP_SKIP[`SKIP`] generate the specified
258 result. They're intended to be used when what we're trying to test
259 doesn't quite fit into any of the two preceding ways of working.
263 Suites serve to group tests into managable (and runnable) chunks, they
264 make it easy to have many tests defined, but only run those that
265 pertain to what we're currently working on. Suites, like tests, have a
266 name which can be used to retrieve the suite, and running a suite
267 simply causes all of the suite's tests to be run, if the suite
268 contains other suites, then those are run as well (and so on and so
271 There is one suite that's a little special (in so far as it always
272 exists), the `T` suite. If you ignore suites completely, which is a
273 good idea at first or for small(ish) code bases, you're actually
274 putting all your tests into the `T` suite.
276 === Creating Suites ===
278 Suites are created in one of two ways: Either explicitly via the
279 xref:OP_DEF-SUITE[`def-suite`] macro, or implicity via the
280 xref:OP_DEF-SUITE-STAR-[`def-suite*`] and/or
281 xref:OP_IN-SUITE-STAR-[`in-suite*`] macros:
283 Suites, very much like tests, have a name (which is globally unique)
284 which can be used to retrieve the suite (so that you can run it), and,
285 most of the time, suites are part of a suite (the exception being the
286 special suite `T`, which is never a part of any suite).
288 For example these two forms will first define a suite called
289 `:my-project`, then define a second suite called `:my-db-layer`, which
290 is a sub suite of `:my-project` and set the current suite to
293 --------------------------------
294 (def-suite :my-project)
296 (in-suite* :my-db-layer :in :my-project)
297 --------------------------------
299 [[THE_CURRENT_SUITE]]
300 === The Current Suite ===
302 FiveAM also has the concept of a current suite and everytime a test is
303 created it adds itself to the current suite's set of tests. The
304 `IN-SUITE` and `IN-SUITE*` macros, in a similar fashion to
305 `IN-PACKAGE`, change the current suite.
307 Unless changed via `IN-SUITE` and `IN-SUITE*` the current suite is the
310 Having a default current suite allows developers to ignore suites
311 completly and still have FiveAM's suite mechanism in place if they
312 want to add suites in later.
315 === Running Suites ===
317 When a suite is run we do nothing more than run all the tests (and any
318 other suites) in the named suite. And, on one level, that's it, suites
319 allow you run a whole set of tests at once just by passing in the name
323 === Per-suite Fixtures ===
325 xref:FIXTURES[Fixtures] can also be associated with suite. Often
326 enough when testing an external component, a database or a network
327 server or something, we'll have multiple tests which all use a mock
328 version of this component. It is often easier to associate the fixture
329 with the suite directly than have to do this for every individual
330 test. Associating a fixture to a suite doesn't change the suite at
331 all, only when a test is then defined in that suite, then the fixture
332 will be applied to the test's body (unless the test's own `def-test`
333 form explicitly uses another fixture).
338 The general interface is `run`, this takes a set of tests (or symbol
339 that name tests or suites) and returns a list of test results (one
340 element for each check that was executed). The output of `run` is,
341 generally, passed to the `explain` function which, given an explainer
342 object, produces some human readable text describing the test
343 failures. The 99% of the time a human will be using 5am (as opposed to
344 a continuous integration bot) they'll want to run the tests and
345 immediately see the results with detailed failure info, this can be
346 done in one step via: `run!` (see the first example).
348 If you want to run a specific test:
350 --------------------------------
352 --------------------------------
354 Where `TEST-NAME` is either a test object (as returned by `get-test`)
355 or a symbol naming a single test or a test suite.
357 === Running Tests at Test Definition Time ===
359 Often enough, especially when fixing regression bugs, we'll always
360 want to run a test right after having changed it. To facilitate this
361 set the variable `*run-test-when-defined*` to T and after compiling a
362 def-test form we'll call `run!` on the name of the test. For obvious
363 reasons you have to set this variable manually after having loaded
366 === Debugging failures and errors ===
370 Normally fiveam will simply capture unexpected errors, record them as
371 failures, and move on to the next test (any following checks in the
372 test body will not be run). However sometimes, well, all the time
373 unless you're running an automated regression test, it's better to not
374 capture the error but open up a debugger, set `*debug-on-error*` to
375 `T` to get this effect.
377 `*debug-on-failure*`::
379 Normally FiveAM will simply record a check failure and move on to the
380 next check, however it can be helpful to stop the check and use the
381 debugger to see what the state of execution is at the time of the
382 test's failure. Setting `*debug-on-failure*` to T will cause FiveAM to
383 enter the debugger whenever a test check fails. Exactly what
384 information is available is, obviously, implementation dependent.
386 [[VIEWING_TEST_RESULTS]]
387 == Viewing test results ==
389 FiveAM provides two "explainers", these are classes which, given a set
390 of results, produce some human readable/understandable
391 output. Explainers are just normal CLOS classes (and can be easily
392 subclassed) with one important method: `explain`.
394 The `run!` and `explain!` functions use the detailed-text-explainer,
395 if you want another explainer you'll have to call `run` and `explain`
398 --------------------------------
399 (explain (make-instance MY-EXPLAINER)
402 --------------------------------
404 == Random Testing (QuickCheck) ==
408 Every FiveAM test can be a random test, just use the for-all macro.
412 Fixtures are, much like macros, ways to hide common code so that the
413 essential functionality we're trying to test is easier to see. Unlike
414 normal macros fixtures are not allowed to inspect the source code of
415 their arguments, all they can really do is wrap one form (or multiple
416 forms in a progn) in something else.
419 Fixtures exist for the common case where we want to bind some
420 variables to some mock (or test) values and run our test in this
421 state. If anything more complicated than this is neccessary just use a
424 Fixtures are defined via the `def-fixture` macro and used either with
425 `with-fixture` directory or, more commonly, using the `:fixture`
426 argument to `def-test` or `def-suite`. A common example of a fixture
429 --------------------------------
430 (def-fixture mock-db ()
431 (let ((*database* (make-instance 'mock-db))
432 (*connection* (make-instance 'mock-connection)))
435 (mock-close-connection *connection*))))
437 (with-fixture mock-db ()
438 (is-true (database-p *database*)))
440 <1> This is a local macro named 5AM:&BODY (the user of def-fixture can
441 not change this name)
443 --------------------------------
445 The body of the `def-fixture` has one local function (actually a local
446 macro) called `&body` which will expand into whatever the body passed
447 to `with-fixture` is. `def-fixture` also has an argument list, but
448 there are two things to note: 1) in practice it's rarely used; 2)
449 these are arguments will be bound to values (like defun) and not
450 source code (like defmacro).
458 ================================
459 --------------------------------
460 (def-test NAME (&key DEPENDS-ON SUITE FIXTURE COMPILE-AT PROFILE) &body BODY)
461 --------------------------------
463 include::docstrings/OP_DEF-TEST.txt[]
464 ================================
469 ================================
471 (def-suite NAME &key DESCRIPTION IN FIXTURE)
474 include::docstrings/OP_DEF-SUITE.txt[]
475 ================================
478 [[OP_IN-SUITE-STAR-]]
479 === IN-SUITE / IN-SUITE* ===
481 ================================
486 include::docstrings/OP_IN-SUITE.txt[]
487 ================================
489 ================================
491 (in-suite* NAME &key IN)
494 include::docstrings/OP_IN-SUITE-STAR-.txt[]
495 ================================
500 ================================
502 (is (PREDICATE EXPECTED ACTUAL) &rest REASON-ARGS)
504 (is (PREDICATE ACTUAL) &rest REASON-ARGS)
507 include::docstrings/OP_IS.txt[]
508 ================================
512 === IS-TRUE / IS-FALSE / IS-EVERY ===
514 ================================
516 (is-true CONDITION &rest reason)
519 include::docstrings/OP_IS-TRUE.txt[]
520 ================================
522 ================================
524 (is-false CONDITION &rest reason)
527 include::docstrings/OP_IS-FALSE.txt[]
528 ================================
530 ////////////////////////////////
531 //// the actual doc string of talks about functionality i don't want
532 //// to publises (since it's just weird). se we use our own here
533 ////////////////////////////////
534 ================================
536 (is-every predicate &rest (EXPECTED ACTUAL &rest REASON))
539 Designed for those cases where you have a large set of expected/actual
540 pairs that must be compared using the same predicate function.
546 (is (,PREDICATE ,EXPECTED ,ACTUAL) ,@REASON)
551 ================================
555 === SIGNALS / FINISHES ===
557 ================================
559 (signals CONDITION &body body)
562 include::docstrings/OP_SIGNALS.txt[]
563 ================================
565 ================================
567 (finishes &body body)
570 include::docstrings/OP_FINISHES.txt[]
571 ================================
576 === PASS / FAIL / SKIP ===
578 ================================
580 (skip &rest REASON-ARGS)
583 include::docstrings/OP_SKIP.txt[]
584 ================================
586 ================================
588 (pass &rest REASON-ARGS)
591 include::docstrings/OP_PASS.txt[]
592 ================================
594 ================================
596 (fail &rest REASON-ARGS)
599 include::docstrings/OP_FAIL.txt[]
600 ================================
603 [[OP_-EPOINT--EPOINT-]]
604 [[OP_-EPOINT--EPOINT--EPOINT-]]
607 [[OP_EXPLAIN-EPOINT-]]
609 === RUN! / EXPLAIN! / DEBUG! ===
611 ================================
613 (run! &optional TEST-NAME)
616 include::docstrings/OP_RUN-EPOINT-.txt[]
617 ================================
619 ================================
621 (explain! RESULT-LIST)
624 include::docstrings/OP_EXPLAIN-EPOINT-.txt[]
625 ================================
627 ================================
632 include::docstrings/OP_DEBUG-EPOINT-.txt[]
633 ================================
638 ================================
643 include::docstrings/OP_RUN.txt[]
644 ================================
649 ================================
651 (def-fixture (NAME (&rest ARGS) &body BODY)
654 include::docstrings/OP_DEF-FIXTURE.txt[]
655 ================================
660 ================================
662 (with-fixture NAME (&rest ARGS) &body BODY)
665 include::docstrings/OP_WITH-FIXTURE.txt[]
666 ================================
671 ================================
672 --------------------------------
673 (for-all (&rest (NAME VALUE &optional GUARD))
675 --------------------------------
677 include::docstrings/OP_FOR-ALL.txt[]
678 ================================
680 [[VAR_-STAR-NUM-TRIALS-STAR-]]
681 [[VAR_-STAR-MAX-TRIALS-STAR-]]
682 === \*NUM-TRIALS* / \*MAX-TRIALS* ===
684 ================================
689 include::docstrings/VAR_-STAR-NUM-TRIALS-STAR-.txt[]
690 ================================
692 ================================
697 include::docstrings/VAR_-STAR-MAX-TRIALS-STAR-.txt[]
698 ================================
702 === GEN-INTEGER / GEN-FLOAT ===
704 ================================
706 (gen-integer &key MIN MAX)
709 include::docstrings/OP_GEN-INTEGER.txt[]
710 ================================
712 ================================
714 (gen-float &key BOUND TYPE MIN MAX)
717 include::docstrings/OP_GEN-FLOAT.txt[]
718 ================================
722 === GEN-CHARACTER / GEN-STRING ===
724 ================================
726 (gen-character &key CODE-LIMIT CODE ALPHANUMERICP)
729 include::docstrings/OP_GEN-CHARACTER.txt[]
730 ================================
732 ================================
734 (gen-string &key LENGTH ELEMENTS)
737 include::docstrings/OP_GEN-STRING.txt[]
738 ================================
743 ================================
745 (gen-buffer &key LENGTH ELEMENTS ELEMENT-TYPE)
748 include::docstrings/OP_GEN-STRING.txt[]
749 ================================
753 === GEN-LIST / GEN-TREE ===
755 ================================
757 (gen-list &key LENGTH ELEMENTS)
760 include::docstrings/OP_GEN-LIST.txt[]
761 ================================
763 ================================
766 (gen-tree &key SIZE ELEMENTS)
769 include::docstrings/OP_GEN-TREE.txt[]
770 ================================
772 [[OP_GEN-ONE-ELEMENT]]
773 === GEN-ONE-ELEMENT ===
775 ================================
777 (gen-one-element &rest ELEMENTS)
780 include::docstrings/OP_GEN-ONE-ELEMENT.txt[]
781 ================================
785 ////////////////////////////////
787 ////////////////////////////////