2 Marco Baringer <mb@bese.it>
8 :website: http://common-lisp.net/project/fiveam
9 :stylesheet: fiveam.css
14 === The Super Brief Introduction ===
16 FiveAM is a testing framework. See the xref:API_REFERENCE[api] for
19 === An Ever So Slightly Longer Introduction ===
21 You use define some xref:TESTS[tests] (using
22 xref:OP_DEF-TEST[`def-test`]), each of which consists of some
23 xref:CHECKS[checks] (with xref:OP_IS[`is`] and friends) which can pass
24 or fail; you xref:RUNNING_TESTS[run] some tests (using
25 xref:OP_RUN-EPOINT-[run!] and friends) and you look at the results
26 (probably using xref:OP_RUN-EPOINT-[run!] again). Lather, rinse,
29 === The Real Introduction ===
31 FiveAM is a testing framework, this is a rather vague concept, so
32 before talking about how to use FiveAM it's worth knowing what task(s)
33 FiveAM was built to do and, in particular, which styles of testing
34 FiveAM was designed to facilitate:
36 `test driven development`:: sometimes you know what you're trying to
37 do (lucky you) and you can figure out what your code should do
38 before you've written the code itself. The idea here is that you
39 write a bunch of tests and when all these test pass your code is
42 `interactive testing`:: sometimes as you're writing code you'll see
43 certain constraints that your code has to meet. For example you'll
44 realize there's a specific border case your code, which you're
45 probably not even done writing, has to deal with. In this work flow
46 you'll write code and tests more or less simultaneously and by the
47 time you're satisfied that your code does what it should you'll have
48 a set of tests which prove that it does what you think it does.
50 `regression testing`:: sometimes you're pretty confident, just by
51 looking at the code, that your program does what it should, but you
52 want an automatic way to make sure that it continues to do what it
53 does even if (when) you change other parts of the code.
56 There's also `beaviour driven development`. this works under
57 the assumption that you can write tests in a natural-ish lanugage and
58 they'll be easier to maintain than tests writen in code (have we
59 learned nothing from cobol?). FiveAM does not, in its current
60 implementation, support link:http://cukes.info/[cucumber] like
61 behaviour driven development. patches welcome (they'll get laughed at
62 at first, but they'll get applied, and then they'll get used, and then
63 they'll be an essential part of fiveam itself...)
67 Since there are far many more testing frameworks than there are words
68 for talking about testing frameworks, the same words end up meaning
69 different things in different frameworks. Just to be clear, here are
70 the words fiveam uses:
72 `check`:: a single expression which has an expected value.
74 `test`:: a set of checks which we want to always run together.
76 `suite`:: a group of tests we often want to run all at once.
81 Tests are created with the xref:OP_DEF-TEST[`def-test`] macro and
86 Because everything deserves a name. Names in FiveAM are symbols (or
87 anything that can be sensibly put in an `eql` hash table) and they are
88 used both to select which test to run (as arguments to `run!` and
89 family) and when reporting test failures.
93 Every test has a function which is the actual code that gets executed
94 when the test is run. This code, whatever it is, will, bugs aside,
95 xref:CHECKS[create a set of test result objects] (failures, successes
96 and skips) and store these in a few dynamic variables (you don't need
97 to worry about those).
99 The body is actually the only real part of the test, everything else
100 is administrativia. Sometimes usefel administrativia, but none the
105 Generally speaking you'll have so many tests that you'll not want to
106 run them all every single time you need to run one of them (automated
107 regression testing is another use case). Tests can be grouped into
108 suites, and suites can also be grouped into suites, and suites have
109 names, so by specfying the name of a suite we only run those tests
110 that are a part of that suite.
112 Unless otherwise specified tests add themselves to the xref:THE_CURRENT_SUITE[current suite].
114 There are two other properties, also set via parameters to
115 xref:OP_DEF-TEST[`def-test`], which influence how the tests are
118 When to compile the test::
120 Often enough, when working with lisp macros especially, it's useful to
121 delay compilation of the test's body until the test is run. A useful
122 side effect of this delay is that the code will be recompiled every
123 time its run, so if the macro definition has changed that will be
124 picked up at the next run of the test. While this is the default mode
125 of operation for FiveAM it can be turned off and tests will be
126 compiled at the 'normal' time (when the enclosing def-test form is
129 Whether to run the test at all::
131 Sometimes, but far less often than the designer of FiveAM expected,
132 it's useful to run a test only when some other test passes. The
133 assumption being that if the lower level tests have failed there's no
134 point in cluttering up the output by running the higher level tests as
137 YMMV. (i got really bad mileage out of this feature)
142 At the heart of every test is something which compares the result of
143 some code to some expected value, in FiveAM these are called
144 checks. All checks in FiveAM do something, exactly what depends on the
145 check, and then either:
147 . generate a "this check passed" result
149 . generate a "this check failed" result and a corresponding failure
152 . generate a "for some reason this check was skipped" result.
154 All checks take, as an optional argument, so called "reason format
155 control arguments." Should the check fail (or be skipped) these
156 arguments will be passed to format, via something like `(curry
157 #'format nil)`, and the result will be used as the
158 explanation/description of the failure.
160 When it comes to the actual check functions themeselves, there are
163 . xref:CHECKING_RETURN_VALUES[those that take a value and compare it
166 . xref:CHECKING_CONTROL_FLOW[those that make sure the program's
167 execution takes, or does not take, a certain path]
169 . xref:ARBITRARY_CHECK_RESULTS[those that just force a success or
170 failure to be recorded].
172 [[CHECKING_RETURN_VALUES]]
173 === Checking return values ===
175 xref:OP_IS[`IS`], xref:OP_IS-TRUE[`IS-TRUE`],
176 xref:OP_IS[`IS-FALSE`] will take one form and compare its return
177 value to some known value (the so called expected vaule) and report an
178 error if these two are not equal.
180 --------------------------------
181 ;; Pass if (+ 2 2) is = to 5
183 ;; Pass if (zerop 0) is not-NIL
185 ;; Pass if (zerop 1) is NIL
187 --------------------------------
189 Often enough we want to test a set of expected values against a set of
190 test values using the same operator. If, for example, we were
191 implementing a string formatting functions, then `IS-EVERY` provides a
192 concise way to line up N different inputs along with their expected
193 outputs. For example, let's say we were testing `cl:+`, we could setup
194 a list of tests like this:
196 --------------------------------
197 (is-every #'= (5 (+ 2 2))
202 --------------------------------
204 We'd do this instead of writing out 5 seperate `IS` or `IS-TRUE`
207 [[CHECKING_CONTROL_FLOW]]
208 === Checking control flow ===
210 xref:OP_SIGNALS[`SIGNALS`] and xref:OP_FINISHES[`FINISHES`] create
211 pass/fail results depending on whether their body code did or did not
214 Both of these checks assume that there is a single block of code and
215 it either runs to completion or it doesn't. Sometimes though the logic
216 is more complex and you can't easily represent it as a single progn
217 with a flag at the end. See xref:ARBITRARY_CHECK_RESULTS[below].
219 [[ARBITRARY_CHECK_RESULTS]]
220 === Recording arbitrary test results ===
222 Very simply these three checks, xref:OP_PASS[`PASS`],
223 xref:OP_FAIL[`FAIL`] and xref:OP_SKIP[`SKIP`] generate the specified
224 result. They're intended to be used when what we're trying to test
225 doesn't quite fit into any of the two preceding ways of working.
229 Suites serve to group tests into managable (and runnable) chunks, they
230 make it easy to have many tests defined, but only run those that
231 pertain to what we're currently working on. Suites, like tests, have a
232 name which can be used to retrieve the suite, and running a suite
233 simply causes all of the suite's tests to be run, if the suite
234 contains other suites, then those are run as well (and so on and so
237 There is one suite that's a little special (in so far as it always
238 exists), the `T` suite. If you ignore suites completely, which is a
239 good idea at first or for small(ish) code bases, you're actually
240 putting all your tests into the `T` suite.
242 === Creating Suites ===
244 Suites are created in one of two ways: Either explicitly via the
245 xref:OP_DEF-SUITE[`def-suite`] macro, or implicity via the
246 xref:OP_DEF-SUITE-STAR-[`def-suite*`] and/or
247 xref:OP_IN-SUITE-STAR-[`in-suite*`] macros:
249 Suites, very much like tests, have a name (which is globally unique)
250 which can be used to retrieve the suite (so that you can run it), and,
251 most of the time, suites are part of a suite (the exception being the
252 special suite `T`, which is never a part of any suite).
254 For example these two forms will first define a suite called
255 `:my-project`, then define a second suite called `:my-db-layer`, which
256 is a sub suite of `:my-project` and set the current suite to
259 --------------------------------
260 (def-suite :my-project)
262 (in-suite* :my-db-layer :in :my-project)
263 --------------------------------
265 [[THE_CURRENT_SUITE]]
266 === The Current Suite ===
268 FiveAM also has the concept of a current suite and everytime a test is
269 created it adds itself to the current suite's set of tests. The
270 `IN-SUITE` and `IN-SUITE*` macros, in a similar fashion to
271 `IN-PACKAGE`, change the current suite.
273 Unless changed via `IN-SUITE` and `IN-SUITE*` the current suite is the
276 Having a default current suite allows developers to ignore suites
277 completly and still have FiveAM's suite mechanism in place if they
278 want to add suites in later.
281 === Running Suites ===
283 When a suite is run we do nothing more than run all the tests (and any
284 other suites) in the named suite. And, on one level, that's it, suites
285 allow you run a whole set of tests at once just by passing in the name
289 === Per-suite Fixtures ===
291 xref:FIXTURES[Fixtures] can also be associated with suite. Often
292 enough when testing an external component, a database or a network
293 server or something, we'll have multiple tests which all use a mock
294 version of this component. It is often easier to associate the fixture
295 with the suite directly than have to do this for every individual
296 test. Associating a fixture to a suite doesn't change the suite at
297 all, only when a test is then defined in that suite, then the fixture
298 will be applied to the test's body (unless the test's own `def-test`
299 form explicitly uses another fixture).
304 The general interface is `run`, this takes a set of tests (or symbol
305 that name tests or suites) and returns a list of test results (one
306 element for each check that was executed). The output of `run` is,
307 generally, passed to the `explain` function which, given an explainer
308 object, produces some human readable text describing the test
309 failures. The 99% of the time a human will be using 5am (as opposed to
310 a continuous integration bot) they'll want to run the tests and
311 immediately see the results with detailed failure info, this can be
312 done in one step via: `run!` (see the first example).
314 If you want to run a specific test:
316 --------------------------------
318 --------------------------------
320 Where `TEST-NAME` is either a test object (as returned by `get-test`)
321 or a symbol naming a single test or a test suite.
323 === Re-running Tests ===
325 The `run!` function stores its arguments in a set of variables and,
326 via the functions `!`, `!!` and `!!!` will rerun those named
327 tests. Note that we're deliberatly talking about names, and not test
328 objects, `!` will take the last argument passed to `run!` and call
329 `run!` with that again, looking up the test again if the argument was
332 This ensures that `!` will always run the current definition of a
333 test, even if the test has been redefined since the last time `run!`
336 === Running Tests at Test Definition Time ===
338 Often enough, especially when fixing regression bugs, we'll always
339 want to run a test right after having changed it. To facilitate this
340 set the variable `*run-test-when-defined*` to T and after compiling a
341 def-test form we'll call `run!` on the name of the test. For obvious
342 reasons you have to set this variable manually after having loaded
346 Setting `*run-test-when-defined*` will cause `run!` to get called far
347 more often than normal. `!` and `!!` and `!!!` don't know that they're
348 getting called semi-automatically and will therefore tend to all
349 reduce to the same test (which still isn't totally useless behaviour).
351 === Debugging failures and errors ===
355 Normally fiveam will simply capture unexpected errors, record them as
356 failures, and move on to the next test (any following checks in the
357 test body will not be run). However sometimes, well, all the time
358 unless you're running an automated regression test, it's better to not
359 capture the error but open up a debugger, set `*debug-on-error*` to
360 `T` to get this effect.
362 `*debug-on-failure*`::
364 Normally FiveAM will simply record a check failure and move on to the
365 next check, however it can be helpful to stop the check and use the
366 debugger to see what the state of execution is at the time of the
367 test's failure. Setting `*debug-on-failure*` to T will cause FiveAM to
368 enter the debugger whenever a test check fails. Exactly what
369 information is available is, obviously, implementation dependent.
371 [[VIEWING_TEST_RESULTS]]
372 == Viewing test results ==
374 FiveAM provides two "explainers", these are classes which, given a set
375 of results, produce some human readable/understandable
376 output. Explainers are just normal CLOS classes (and can be easily
377 subclassed) with one important method: `explain`.
379 The `run!` and `explain!` functions use the detailed-text-explainer,
380 if you want another explainer you'll have to call `run` and `explain`
383 --------------------------------
384 (explain (make-instance MY-EXPLAINER)
387 --------------------------------
389 == Random Testing (QuickCheck) ==
393 Every FiveAM test can be a random test, just use the for-all macro.
397 Fixtures are, much like macros, ways to hide common code so that the
398 essential functionality we're trying to test is easier to see. Unlike
399 normal macros fixtures are not allowed to inspect the source code of
400 their arguments, all they can really do is wrap one form (or multiple
401 forms in a progn) in something else.
404 Fixtures exist for the common case where we want to bind some
405 variables to some mock (or test) values and run our test in this
406 state. If anything more complicated than this is neccessary just use a
409 Fixtures are defined via the `def-fixture` macro and used either with
410 `with-fixture` directory or, more commonly, using the `:fixture`
411 argument to `def-test` or `def-suite`. A common example of a fixture
414 --------------------------------
415 (def-fixture mock-db ()
416 (let ((*database* (make-instance 'mock-db))
417 (*connection* (make-instance 'mock-connection)))
420 (mock-close-connection *connection*))))
422 (with-fixture mock-db ()
423 (is-true (database-p *database*)))
425 <1> This is a local macro named 5AM:&BODY (the user of def-fixture can
426 not change this name)
428 --------------------------------
430 The body of the `def-fixture` has one local function (actually a local
431 macro) called `&body` which will expand into whatever the body passed
432 to `with-fixture` is. `def-fixture` also has an argument list, but
433 there are two things to note: 1) in practice it's rarely used; 2)
434 these are arguments will be bound to values (like defun) and not
435 source code (like defmacro).
443 ================================
444 --------------------------------
446 (&key DEPENDS-ON SUITE FIXTURE COMPILE-AT PROFILE)
448 --------------------------------
450 include::docstrings/OP_DEF-TEST.txt[]
451 ================================
456 ================================
458 (def-suite NAME &key DESCRIPTION IN FIXTURE)
461 include::docstrings/OP_DEF-SUITE.txt[]
462 ================================
465 [[OP_IN-SUITE-STAR-]]
466 === IN-SUITE / IN-SUITE* ===
468 ================================
473 include::docstrings/OP_IN-SUITE.txt[]
474 ================================
476 ================================
478 (in-suite* NAME &key IN)
481 include::docstrings/OP_IN-SUITE-STAR-.txt[]
482 ================================
487 ================================
489 (is (PREDICATE EXPECTED ACTUAL) &rest REASON-ARGS)
491 (is (PREDICATE ACTUAL) &rest REASON-ARGS)
494 include::docstrings/OP_IS.txt[]
495 ================================
499 === IS-TRUE / IS-FALSE / IS-EVERY ===
501 ================================
503 (is-true CONDITION &rest reason)
506 include::docstrings/OP_IS-TRUE.txt[]
507 ================================
509 ================================
511 (is-false CONDITION &rest reason)
514 include::docstrings/OP_IS-FALSE.txt[]
515 ================================
517 ////////////////////////////////
518 //// the actual doc string of talks about functionality i don't want
519 //// to publises (since it's just weird). se we use our own here
520 ////////////////////////////////
521 ================================
523 (is-every predicate &rest (EXPECTED ACTUAL &rest REASON))
526 Designed for those cases where you have a large set of expected/actual
527 pairs that must be compared using the same predicate function.
533 (is (,PREDICATE ,EXPECTED ,ACTUAL) ,@REASON)
538 ================================
542 === SIGNALS / FINISHES ===
544 ================================
546 (signals CONDITION &body body)
549 include::docstrings/OP_SIGNALS.txt[]
550 ================================
552 ================================
554 (finishes &body body)
557 include::docstrings/OP_FINISHES.txt[]
558 ================================
563 === PASS / FAIL / SKIP ===
565 ================================
567 (skip &rest REASON-ARGS)
570 include::docstrings/OP_SKIP.txt[]
571 ================================
573 ================================
575 (pass &rest REASON-ARGS)
578 include::docstrings/OP_PASS.txt[]
579 ================================
581 ================================
583 (fail &rest REASON-ARGS)
586 include::docstrings/OP_FAIL.txt[]
587 ================================
590 [[OP_-EPOINT--EPOINT-]]
591 [[OP_-EPOINT--EPOINT--EPOINT-]]
594 [[OP_EXPLAIN-EPOINT-]]
596 === RUN! / EXPLAIN! / DEBUG! ===
598 ================================
600 (run! &optional TEST-NAME)
603 include::docstrings/OP_RUN-EPOINT-.txt[]
604 ================================
606 ================================
608 (explain! RESULT-LIST)
611 include::docstrings/OP_EXPLAIN-EPOINT-.txt[]
612 ================================
614 ================================
619 include::docstrings/OP_DEBUG-EPOINT-.txt[]
620 ================================
625 ================================
630 include::docstrings/OP_RUN.txt[]
631 ================================
635 ================================
640 include::docstrings/OP_-EPOINT-.txt[]
641 ================================
643 ================================
648 include::docstrings/OP_-EPOINT--EPOINT-.txt[]
649 ================================
651 ================================
656 include::docstrings/OP_-EPOINT--EPOINT--EPOINT-.txt[]
657 ================================
662 ================================
664 (def-fixture (NAME (&rest ARGS) &body BODY)
667 include::docstrings/OP_DEF-FIXTURE.txt[]
668 ================================
673 ================================
675 (with-fixture NAME (&rest ARGS) &body BODY)
678 include::docstrings/OP_WITH-FIXTURE.txt[]
679 ================================
684 ================================
685 --------------------------------
686 (for-all (&rest (NAME VALUE &optional GUARD))
688 --------------------------------
690 include::docstrings/OP_FOR-ALL.txt[]
691 ================================
693 [[VAR_-STAR-NUM-TRIALS-STAR-]]
694 [[VAR_-STAR-MAX-TRIALS-STAR-]]
695 === \*NUM-TRIALS* / \*MAX-TRIALS* ===
697 ================================
702 include::docstrings/VAR_-STAR-NUM-TRIALS-STAR-.txt[]
703 ================================
705 ================================
710 include::docstrings/VAR_-STAR-MAX-TRIALS-STAR-.txt[]
711 ================================
715 === GEN-INTEGER / GEN-FLOAT ===
717 ================================
719 (gen-integer &key MIN MAX)
722 include::docstrings/OP_GEN-INTEGER.txt[]
723 ================================
725 ================================
727 (gen-float &key BOUND TYPE MIN MAX)
730 include::docstrings/OP_GEN-FLOAT.txt[]
731 ================================
735 === GEN-CHARACTER / GEN-STRING ===
737 ================================
739 (gen-character &key CODE-LIMIT CODE ALPHANUMERICP)
742 include::docstrings/OP_GEN-CHARACTER.txt[]
743 ================================
745 ================================
747 (gen-string &key LENGTH ELEMENTS)
750 include::docstrings/OP_GEN-STRING.txt[]
751 ================================
756 ================================
758 (gen-buffer &key LENGTH ELEMENTS ELEMENT-TYPE)
761 include::docstrings/OP_GEN-STRING.txt[]
762 ================================
766 === GEN-LIST / GEN-TREE ===
768 ================================
770 (gen-list &key LENGTH ELEMENTS)
773 include::docstrings/OP_GEN-LIST.txt[]
774 ================================
776 ================================
779 (gen-tree &key SIZE ELEMENTS)
782 include::docstrings/OP_GEN-TREE.txt[]
783 ================================
785 [[OP_GEN-ONE-ELEMENT]]
786 === GEN-ONE-ELEMENT ===
788 ================================
790 (gen-one-element &rest ELEMENTS)
793 include::docstrings/OP_GEN-ONE-ELEMENT.txt[]
794 ================================
798 ////////////////////////////////
800 ////////////////////////////////