2 Marco Baringer <mb@bese.it>
8 :website: http://common-lisp.net/project/fiveam
9 :stylesheet: fiveam.css
14 === The Super Brief Introduction ===
16 |================================
17 | (xref:OP_DEF-TEST[`def-test`] `NAME` () &body `BODY`) | define tests
18 | (xref:OP_IS[`is`] (`PREDICATE` `EXPECTED` `ACTUAL`)) | check that, according to `PREDICATE` our `ACTUAL` is the same as our `EXPECTED`
19 | (xref:OP_IS[`is-true`] VALUE) | check that a value is non-NIL
20 | (xref:OP_RUN![`run!`] TEST-NAME) | run one (or more) tests and print the results
21 | (xref:OP_RUN![`!`]) | rerun the most recently run test.
22 |================================
24 See the xref:API_REFERENCE[api] for details.
26 === An Ever So Slightly Longer Introduction ===
28 You use define some xref:TESTS[tests] (using
29 xref:OP_DEF-TEST[`def-test`]), each of which consists of some
30 xref:CHECKS[checks] (with xref:OP_IS[`is`] and friends) which can pass
33 --------------------------------
36 (is-false (= 5 (+ 2 2))))
37 --------------------------------
39 you xref:RUNNING_TESTS[run] some tests (using xref:OP_RUN[run] and
40 friends) and you look at the results (using using
41 xref:OP_EXPLAIN[explain]); or you do both at once (using
42 xref:OP_RUN-EPOINT-[run!]):
44 --------------------------------
45 CL-USER> (run! 'a-test)
51 --------------------------------
53 Lather, rinse, repeat:
55 --------------------------------
62 --------------------------------
64 === The Real Introduction ===
66 FiveAM is a testing framework, this is a rather vague concept, so
67 before talking about how to use FiveAM it's worth knowing what task(s)
68 FiveAM was built to do and, in particular, which styles of testing
69 FiveAM was designed to facilitate:
71 `test driven development`:: sometimes you know what you're trying to
72 do (lucky you) and you can figure out what your code should do
73 before you've written the code itself. The idea here is that you
74 write a bunch of tests and when all these test pass your code is
77 `interactive testing`:: sometimes as you're writing code you'll see
78 certain constraints that your code has to meet. For example you'll
79 realize there's a specific border case your code, which you're
80 probably not even done writing, has to deal with. In this work flow
81 you'll write code and tests more or less simultaneously and by the
82 time you're satisfied that your code does what it should you'll have
83 a set of tests which prove that it does what you think it does.
85 `regression testing`:: sometimes you're pretty confident, just by
86 looking at the code, that your program does what it should, but you
87 want an automatic way to make sure that it continues to do what it
88 does even if (when) you change other parts of the code.
91 There's also `beaviour driven development`. this works under
92 the assumption that you can write tests in a natural-ish lanugage and
93 they'll be easier to maintain than tests writen in code (have we
94 learned nothing from cobol?). FiveAM does not, in its current
95 implementation, support link:http://cukes.info/[cucumber] like
96 behaviour driven development. patches welcome (they'll get laughed at
97 at first, but they'll get applied, and then they'll get used, and then
98 they'll be an essential part of fiveam itself...)
102 Since there are far many more testing frameworks than there are words
103 for talking about testing frameworks, the same words end up meaning
104 different things in different frameworks. Just to be clear, here are
105 the words fiveam uses:
107 `check`:: a single expression which has an expected value.
109 `test`:: a set of checks which we want to always run together.
111 `suite`:: a group of tests we often want to run all at once.
116 Tests are created with the xref:OP_DEF-TEST[`def-test`] macro and
121 Because everything deserves a name. Names in FiveAM are symbols (or
122 anything that can be sensibly put in an `eql` hash table) and they are
123 used both to select which test to run (as arguments to `run!` and
124 family) and when reporting test failures.
128 Every test has a function which is the actual code that gets executed
129 when the test is run. This code, whatever it is, will, bugs aside,
130 xref:CHECKS[create a set of test result objects] (failures, successes
131 and skips) and store these in a few dynamic variables (you don't need
132 to worry about those).
134 The body is actually the only real part of the test, everything else
135 is administrativia. Sometimes usefel administrativia, but none the
140 Generally speaking you'll have so many tests that you'll not want to
141 run them all every single time you need to run one of them (automated
142 regression testing is another use case). Tests can be grouped into
143 suites, and suites can also be grouped into suites, and suites have
144 names, so by specfying the name of a suite we only run those tests
145 that are a part of that suite.
147 Unless otherwise specified tests add themselves to the xref:THE_CURRENT_SUITE[current suite].
149 There are two other properties, also set via parameters to
150 xref:OP_DEF-TEST[`def-test`], which influence how the tests are
153 When to compile the test::
155 Often enough, when working with lisp macros especially, it's useful to
156 delay compilation of the test's body until the test is run. A useful
157 side effect of this delay is that the code will be recompiled every
158 time its run, so if the macro definition has changed that will be
159 picked up at the next run of the test. While this is the default mode
160 of operation for FiveAM it can be turned off and tests will be
161 compiled at the 'normal' time (when the enclosing def-test form is
164 Whether to run the test at all::
166 Sometimes, but far less often than the designer of FiveAM expected,
167 it's useful to run a test only when some other test passes. The
168 assumption being that if the lower level tests have failed there's no
169 point in cluttering up the output by running the higher level tests as
172 YMMV. (i got really bad mileage out of this feature)
177 At the heart of every test is something which compares the result of
178 some code to some expected value, in FiveAM these are called
179 checks. All checks in FiveAM do something, exactly what depends on the
180 check, and then either:
182 . generate a "this check passed" result
184 . generate a "this check failed" result and a corresponding failure
187 . generate a "for some reason this check was skipped" result.
189 All checks take, as an optional argument, so called "reason format
190 control arguments." Should the check fail (or be skipped) these
191 arguments will be passed to format, via something like `(curry
192 #'format nil)`, and the result will be used as the
193 explanation/description of the failure.
195 When it comes to the actual check functions themeselves, there are
198 . xref:CHECKING_RETURN_VALUES[those that take a value and compare it
201 . xref:CHECKING_CONTROL_FLOW[those that make sure the program's
202 execution takes, or does not take, a certain path]
204 . xref:ARBITRARY_CHECK_RESULTS[those that just force a success or
205 failure to be recorded].
207 [[CHECKING_RETURN_VALUES]]
208 === Checking return values ===
210 xref:OP_IS[`IS`], xref:OP_IS-TRUE[`IS-TRUE`],
211 xref:OP_IS[`IS-FALSE`] will take one form and compare its return
212 value to some known value (the so called expected vaule) and report an
213 error if these two are not equal.
215 --------------------------------
216 ;; Pass if (+ 2 2) is = to 5
218 ;; Pass if (zerop 0) is not-NIL
220 ;; Pass if (zerop 1) is NIL
222 --------------------------------
224 Often enough we want to test a set of expected values against a set of
225 test values using the same operator. If, for example, we were
226 implementing a string formatting functions, then `IS-EVERY` provides a
227 concise way to line up N different inputs along with their expected
228 outputs. For example, let's say we were testing `cl:+`, we could setup
229 a list of tests like this:
231 --------------------------------
232 (is-every #'= (5 (+ 2 2))
237 --------------------------------
239 We'd do this instead of writing out 5 seperate `IS` or `IS-TRUE`
242 [[CHECKING_CONTROL_FLOW]]
243 === Checking control flow ===
245 xref:OP_SIGNALS[`SIGNALS`] and xref:OP_FINISHES[`FINISHES`] create
246 pass/fail results depending on whether their body code did or did not
249 Both of these checks assume that there is a single block of code and
250 it either runs to completion or it doesn't. Sometimes though the logic
251 is more complex and you can't easily represent it as a single progn
252 with a flag at the end. See xref:ARBITRARY_CHECK_RESULTS[below].
254 [[ARBITRARY_CHECK_RESULTS]]
255 === Recording arbitrary test results ===
257 Very simply these three checks, xref:OP_PASS[`PASS`],
258 xref:OP_FAIL[`FAIL`] and xref:OP_SKIP[`SKIP`] generate the specified
259 result. They're intended to be used when what we're trying to test
260 doesn't quite fit into any of the two preceding ways of working.
264 Suites serve to group tests into managable (and runnable) chunks, they
265 make it easy to have many tests defined, but only run those that
266 pertain to what we're currently working on. Suites, like tests, have a
267 name which can be used to retrieve the suite, and running a suite
268 simply causes all of the suite's tests to be run, if the suite
269 contains other suites, then those are run as well (and so on and so
272 There is one suite that's a little special (in so far as it always
273 exists), the `T` suite. If you ignore suites completely, which is a
274 good idea at first or for small(ish) code bases, you're actually
275 putting all your tests into the `T` suite.
277 === Creating Suites ===
279 Suites are created in one of two ways: Either explicitly via the
280 xref:OP_DEF-SUITE[`def-suite`] macro, or implicity via the
281 xref:OP_DEF-SUITE-STAR-[`def-suite*`] and/or
282 xref:OP_IN-SUITE-STAR-[`in-suite*`] macros:
284 Suites, very much like tests, have a name (which is globally unique)
285 which can be used to retrieve the suite (so that you can run it), and,
286 most of the time, suites are part of a suite (the exception being the
287 special suite `T`, which is never a part of any suite).
289 For example these two forms will first define a suite called
290 `:my-project`, then define a second suite called `:my-db-layer`, which
291 is a sub suite of `:my-project` and set the current suite to
294 --------------------------------
295 (def-suite :my-project)
297 (in-suite* :my-db-layer :in :my-project)
298 --------------------------------
300 [[THE_CURRENT_SUITE]]
301 === The Current Suite ===
303 FiveAM also has the concept of a current suite and everytime a test is
304 created it adds itself to the current suite's set of tests. The
305 `IN-SUITE` and `IN-SUITE*` macros, in a similar fashion to
306 `IN-PACKAGE`, change the current suite.
308 Unless changed via `IN-SUITE` and `IN-SUITE*` the current suite is the
311 Having a default current suite allows developers to ignore suites
312 completly and still have FiveAM's suite mechanism in place if they
313 want to add suites in later.
316 === Running Suites ===
318 When a suite is run we do nothing more than run all the tests (and any
319 other suites) in the named suite. And, on one level, that's it, suites
320 allow you run a whole set of tests at once just by passing in the name
324 === Per-suite Fixtures ===
326 xref:FIXTURES[Fixtures] can also be associated with suite. Often
327 enough when testing an external component, a database or a network
328 server or something, we'll have multiple tests which all use a mock
329 version of this component. It is often easier to associate the fixture
330 with the suite directly than have to do this for every individual
331 test. Associating a fixture to a suite doesn't change the suite at
332 all, only when a test is then defined in that suite, then the fixture
333 will be applied to the test's body (unless the test's own `def-test`
334 form explicitly uses another fixture).
339 The general interface is `run`, this takes a set of tests (or symbol
340 that name tests or suites) and returns a list of test results (one
341 element for each check that was executed). The output of `run` is,
342 generally, passed to the `explain` function which, given an explainer
343 object, produces some human readable text describing the test
344 failures. The 99% of the time a human will be using 5am (as opposed to
345 a continuous integration bot) they'll want to run the tests and
346 immediately see the results with detailed failure info, this can be
347 done in one step via: `run!` (see the first example).
349 If you want to run a specific test:
351 --------------------------------
353 --------------------------------
355 Where `TEST-NAME` is either a test object (as returned by `get-test`)
356 or a symbol naming a single test or a test suite.
358 === Re-running Tests ===
360 The `run!` function stores its arguments in a set of variables and,
361 via the functions `!`, `!!` and `!!!` will rerun those named
362 tests. Note that we're deliberatly talking about names, and not test
363 objects, `!` will take the last argument passed to `run!` and call
364 `run!` with that again, looking up the test again if the argument was
367 This ensures that `!` will always run the current definition of a
368 test, even if the test has been redefined since the last time `run!`
371 === Running Tests at Test Definition Time ===
373 Often enough, especially when fixing regression bugs, we'll always
374 want to run a test right after having changed it. To facilitate this
375 set the variable `*run-test-when-defined*` to T and after compiling a
376 def-test form we'll call `run!` on the name of the test. For obvious
377 reasons you have to set this variable manually after having loaded
381 Setting `*run-test-when-defined*` will cause `run!` to get called far
382 more often than normal. `!` and `!!` and `!!!` don't know that they're
383 getting called semi-automatically and will therefore tend to all
384 reduce to the same test (which still isn't totally useless behaviour).
386 === Debugging failures and errors ===
390 Normally fiveam will simply capture unexpected errors, record them as
391 failures, and move on to the next test (any following checks in the
392 test body will not be run). However sometimes, well, all the time
393 unless you're running an automated regression test, it's better to not
394 capture the error but open up a debugger, set `*debug-on-error*` to
395 `T` to get this effect.
397 `*debug-on-failure*`::
399 Normally FiveAM will simply record a check failure and move on to the
400 next check, however it can be helpful to stop the check and use the
401 debugger to see what the state of execution is at the time of the
402 test's failure. Setting `*debug-on-failure*` to T will cause FiveAM to
403 enter the debugger whenever a test check fails. Exactly what
404 information is available is, obviously, implementation dependent.
406 [[VIEWING_TEST_RESULTS]]
407 == Viewing test results ==
409 FiveAM provides two "explainers", these are classes which, given a set
410 of results, produce some human readable/understandable
411 output. Explainers are just normal CLOS classes (and can be easily
412 subclassed) with one important method: `explain`.
414 The `run!` and `explain!` functions use the detailed-text-explainer,
415 if you want another explainer you'll have to call `run` and `explain`
418 --------------------------------
419 (explain (make-instance MY-EXPLAINER)
422 --------------------------------
424 == Random Testing (QuickCheck) ==
428 Every FiveAM test can be a random test, just use the for-all macro.
432 Fixtures are, much like macros, ways to hide common code so that the
433 essential functionality we're trying to test is easier to see. Unlike
434 normal macros fixtures are not allowed to inspect the source code of
435 their arguments, all they can really do is wrap one form (or multiple
436 forms in a progn) in something else.
439 Fixtures exist for the common case where we want to bind some
440 variables to some mock (or test) values and run our test in this
441 state. If anything more complicated than this is neccessary just use a
444 Fixtures are defined via the `def-fixture` macro and used either with
445 `with-fixture` directory or, more commonly, using the `:fixture`
446 argument to `def-test` or `def-suite`. A common example of a fixture
449 --------------------------------
450 (def-fixture mock-db ()
451 (let ((*database* (make-instance 'mock-db))
452 (*connection* (make-instance 'mock-connection)))
455 (mock-close-connection *connection*))))
457 (with-fixture mock-db ()
458 (is-true (database-p *database*)))
460 <1> This is a local macro named 5AM:&BODY (the user of def-fixture can
461 not change this name)
463 --------------------------------
465 The body of the `def-fixture` has one local function (actually a local
466 macro) called `&body` which will expand into whatever the body passed
467 to `with-fixture` is. `def-fixture` also has an argument list, but
468 there are two things to note: 1) in practice it's rarely used; 2)
469 these are arguments will be bound to values (like defun) and not
470 source code (like defmacro).
478 ================================
479 --------------------------------
480 (def-test NAME (&key DEPENDS-ON SUITE FIXTURE COMPILE-AT PROFILE) &body BODY)
481 --------------------------------
483 include::docstrings/OP_DEF-TEST.txt[]
484 ================================
489 ================================
491 (def-suite NAME &key DESCRIPTION IN FIXTURE)
494 include::docstrings/OP_DEF-SUITE.txt[]
495 ================================
498 [[OP_IN-SUITE-STAR-]]
499 === IN-SUITE / IN-SUITE* ===
501 ================================
506 include::docstrings/OP_IN-SUITE.txt[]
507 ================================
509 ================================
511 (in-suite* NAME &key IN)
514 include::docstrings/OP_IN-SUITE-STAR-.txt[]
515 ================================
520 ================================
522 (is (PREDICATE EXPECTED ACTUAL) &rest REASON-ARGS)
524 (is (PREDICATE ACTUAL) &rest REASON-ARGS)
527 include::docstrings/OP_IS.txt[]
528 ================================
532 === IS-TRUE / IS-FALSE / IS-EVERY ===
534 ================================
536 (is-true CONDITION &rest reason)
539 include::docstrings/OP_IS-TRUE.txt[]
540 ================================
542 ================================
544 (is-false CONDITION &rest reason)
547 include::docstrings/OP_IS-FALSE.txt[]
548 ================================
550 ////////////////////////////////
551 //// the actual doc string of talks about functionality i don't want
552 //// to publises (since it's just weird). se we use our own here
553 ////////////////////////////////
554 ================================
556 (is-every predicate &rest (EXPECTED ACTUAL &rest REASON))
559 Designed for those cases where you have a large set of expected/actual
560 pairs that must be compared using the same predicate function.
566 (is (,PREDICATE ,EXPECTED ,ACTUAL) ,@REASON)
571 ================================
575 === SIGNALS / FINISHES ===
577 ================================
579 (signals CONDITION &body body)
582 include::docstrings/OP_SIGNALS.txt[]
583 ================================
585 ================================
587 (finishes &body body)
590 include::docstrings/OP_FINISHES.txt[]
591 ================================
596 === PASS / FAIL / SKIP ===
598 ================================
600 (skip &rest REASON-ARGS)
603 include::docstrings/OP_SKIP.txt[]
604 ================================
606 ================================
608 (pass &rest REASON-ARGS)
611 include::docstrings/OP_PASS.txt[]
612 ================================
614 ================================
616 (fail &rest REASON-ARGS)
619 include::docstrings/OP_FAIL.txt[]
620 ================================
623 [[OP_-EPOINT--EPOINT-]]
624 [[OP_-EPOINT--EPOINT--EPOINT-]]
627 [[OP_EXPLAIN-EPOINT-]]
629 === RUN! / EXPLAIN! / DEBUG! ===
631 ================================
633 (run! &optional TEST-NAME)
636 include::docstrings/OP_RUN-EPOINT-.txt[]
637 ================================
639 ================================
641 (explain! RESULT-LIST)
644 include::docstrings/OP_EXPLAIN-EPOINT-.txt[]
645 ================================
647 ================================
652 include::docstrings/OP_DEBUG-EPOINT-.txt[]
653 ================================
658 ================================
663 include::docstrings/OP_RUN.txt[]
664 ================================
668 ================================
673 include::docstrings/OP_-EPOINT-.txt[]
674 ================================
676 ================================
681 include::docstrings/OP_-EPOINT--EPOINT-.txt[]
682 ================================
684 ================================
689 include::docstrings/OP_-EPOINT--EPOINT--EPOINT-.txt[]
690 ================================
695 ================================
697 (def-fixture (NAME (&rest ARGS) &body BODY)
700 include::docstrings/OP_DEF-FIXTURE.txt[]
701 ================================
706 ================================
708 (with-fixture NAME (&rest ARGS) &body BODY)
711 include::docstrings/OP_WITH-FIXTURE.txt[]
712 ================================
717 ================================
718 --------------------------------
719 (for-all (&rest (NAME VALUE &optional GUARD))
721 --------------------------------
723 include::docstrings/OP_FOR-ALL.txt[]
724 ================================
726 [[VAR_-STAR-NUM-TRIALS-STAR-]]
727 [[VAR_-STAR-MAX-TRIALS-STAR-]]
728 === \*NUM-TRIALS* / \*MAX-TRIALS* ===
730 ================================
735 include::docstrings/VAR_-STAR-NUM-TRIALS-STAR-.txt[]
736 ================================
738 ================================
743 include::docstrings/VAR_-STAR-MAX-TRIALS-STAR-.txt[]
744 ================================
748 === GEN-INTEGER / GEN-FLOAT ===
750 ================================
752 (gen-integer &key MIN MAX)
755 include::docstrings/OP_GEN-INTEGER.txt[]
756 ================================
758 ================================
760 (gen-float &key BOUND TYPE MIN MAX)
763 include::docstrings/OP_GEN-FLOAT.txt[]
764 ================================
768 === GEN-CHARACTER / GEN-STRING ===
770 ================================
772 (gen-character &key CODE-LIMIT CODE ALPHANUMERICP)
775 include::docstrings/OP_GEN-CHARACTER.txt[]
776 ================================
778 ================================
780 (gen-string &key LENGTH ELEMENTS)
783 include::docstrings/OP_GEN-STRING.txt[]
784 ================================
789 ================================
791 (gen-buffer &key LENGTH ELEMENTS ELEMENT-TYPE)
794 include::docstrings/OP_GEN-STRING.txt[]
795 ================================
799 === GEN-LIST / GEN-TREE ===
801 ================================
803 (gen-list &key LENGTH ELEMENTS)
806 include::docstrings/OP_GEN-LIST.txt[]
807 ================================
809 ================================
812 (gen-tree &key SIZE ELEMENTS)
815 include::docstrings/OP_GEN-TREE.txt[]
816 ================================
818 [[OP_GEN-ONE-ELEMENT]]
819 === GEN-ONE-ELEMENT ===
821 ================================
823 (gen-one-element &rest ELEMENTS)
826 include::docstrings/OP_GEN-ONE-ELEMENT.txt[]
827 ================================
831 ////////////////////////////////
833 ////////////////////////////////