QuickCheck inspired property-based testing for OCaml, and combinators to generate random values to run tests on.

alt=

The documentation can be found here. This library spent some time in qtest, but is now standalone again!

To construct advanced random generators, the following libraries might be of interest:

Jan Midtgaard (@jmid) has a lecture about property-based testing that relies on QCheck.

Table of Contents

Use

See the documentation. I also wrote a blog post that explains how to use it and some design choices; however, be warned that the API changed in lots of small ways (in the right direction, I hope) so the code will not work any more. [examples] is an updated version of the blog post’s examples.

Build

$ make

You can use opam:

$ opam install qcheck

License

The code is now released under the BSD license.

An Introduction to the Library

First, let’s see a few tests. Let’s open a toplevel (e.g. utop) and type the following to load QCheck:

#require "qcheck";;
Note
 alternatively, it is now possible to locally do: dune utop src to load qcheck.

List Reverse is Involutive

We write a random test for checking that List.rev (List.rev l) = l for any list l:

let test =
  QCheck.Test.make ~count:1000 ~name:"list_rev_is_involutive"
   QCheck.(list small_nat)
   (fun l -> List.rev (List.rev l) = l);;

(* we can check right now the property... *)
QCheck.Test.check_exn test;;

In the above example, we applied the combinator list to the random generator small_nat (ints between 0 and 100), to create a new generator of lists of random integers. These builtin generators come with printers and shrinkers which are handy for outputting and minimizing a counterexample when a test fails.

Consider the buggy property List.rev l = l:

let test =
  QCheck.Test.make ~count:1000 ~name:"my_buggy_test"
   QCheck.(list small_nat)
   (fun l -> List.rev l = l);;

When we run this test we are presented with a counterexample:

# QCheck.Test.check_exn test;;
Exception:
QCheck.Test.Test_fail ("my_buggy_test", ["[0; 1] (after 23 shrink steps)"]).

In this case QCheck found the minimal counterexample [0;1] to the property List.rev l = l and it spent 23 steps shrinking it.

Now, let’s run the buggy test with a decent runner that will print the results nicely (the exact output will change at each run, because of the random seed):

# QCheck_runner.run_tests [test];;

--- Failure --------------------------------------------------------------------

Test my_buggy_test failed (10 shrink steps):

[0; 1]
================================================================================
failure (1 tests failed, 0 tests errored, ran 1 tests)
- : int = 1

For an even nicer output QCheck_runner.run_tests also accepts an optional parameter ~verbose:true.

Mirrors and Trees

QCheck provides many useful combinators to write generators, especially for recursive types, algebraic types, and tuples.

Let’s see how to generate random trees:

type tree = Leaf of int | Node of tree * tree

let leaf x = Leaf x
let node x y = Node (x,y)

let tree_gen = QCheck.Gen.(sized @@ fix
  (fun self n -> match n with
    | 0 -> map leaf nat
    | n ->
      frequency
        [1, map leaf nat;
         2, map2 node (self (n/2)) (self (n/2))]
    ));;

(* generate a few trees, just to check what they look like: *)
QCheck.Gen.generate ~n:20 tree_gen;;

let arbitrary_tree =
  let open QCheck.Iter in
  let rec print_tree = function
    | Leaf i -> "Leaf " ^ (string_of_int i)
    | Node (a,b) -> "Node (" ^ (print_tree a) ^ "," ^ (print_tree b) ^ ")"
  in
  let rec shrink_tree = function
    | Leaf i -> QCheck.Shrink.int i >|= leaf
    | Node (a,b) ->
      of_list [a;b]
      <+>
      (shrink_tree a >|= fun a' -> node a' b)
      <+>
      (shrink_tree b >|= fun b' -> node a b')
  in
  QCheck.make tree_gen ~print:print_tree ~shrink:shrink_tree;;

Here we write a generator of random trees, tree_gen, using the fix combinator. fix is sized (it is a function from int to a random generator; in particular for size 0 it returns only leaves). The sized combinator first generates a random size, and then applies its argument to this size.

Other combinators include monadic abstraction, lifting functions, generation of lists, arrays, and a choice function.

Then, we define arbitrary_tree, a tree QCheck.arbitrary value, which contains everything needed for testing on trees:

  • a random generator (mandatory), weighted with frequency to increase the chance of generating deep trees

  • a printer (optional), very useful for printing counterexamples

  • a shrinker (optional), very useful for trying to reduce big counterexamples to small counterexamples that are usually more easy to understand.

The above shrinker strategy is to

  • reduce the integer leaves, and

  • substitute an internal Node with either of its subtrees or by splicing in a recursively shrunk subtree.

A range of combinators in QCheck.Shrink and QCheck.Iter are available for building shrinking functions.

We can write a failing test using this generator to see the printer and shrinker in action:

let rec mirror_tree (t:tree) : tree = match t with
  | Leaf _ -> t
  | Node (a,b) -> node (mirror_tree b) (mirror_tree a);;

let test_buggy =
  QCheck.Test.make ~name:"buggy_mirror" ~count:200
    arbitrary_tree (fun t -> t = mirror_tree t);;

QCheck_runner.run_tests [test_buggy];;

This test fails with:

--- Failure --------------------------------------------------------------------

Test mirror_buggy failed (6 shrink steps):

Node (Leaf 0,Leaf 1)
================================================================================
failure (1 tests failed, 0 tests errored, ran 1 tests)
- : int = 1

With the (new found) understanding that mirroring a tree changes its structure, we can formulate another property that involves sequentializing its elements in a traversal:

let tree_infix (t:tree): int list =
  let rec aux acc t = match t with
    | Leaf i -> i :: acc
    | Node (a,b) ->
      aux (aux acc b) a
  in
  aux [] t;;

let test_mirror =
  QCheck.Test.make ~name:"mirror_tree" ~count:200
    arbitrary_tree
    (fun t -> List.rev (tree_infix t) = tree_infix (mirror_tree t));;

QCheck_runner.run_tests [test_mirror];;

Preconditions

The functions QCheck.assume and QCheck.(==>) can be used for tests with preconditions. For instance, List.hd l :: List.tl l = l only holds for non-empty lists. Without the precondition, the property is false and will even raise an exception in some cases.

let test_hd_tl =
  QCheck.(Test.make
    (list int) (fun l ->
      assume (l <> []);
      l = List.hd l :: List.tl l));;

QCheck_runner.run_tests [test_hd_tl];;

Long tests

It is often useful to have two version of a testsuite: a short one that runs reasonably fast (so that it is effectively run each time a projet is built), and a long one that might be more exhaustive (but whose running time makes it impossible to run at each build). To that end, each test has a long version. In the long version of a test, the number of tests to run is multiplied by the ~long_factor argument of QCheck.Test.make.

Runners

The module QCheck_runner defines several functions to run tests, including compatibility with OUnit. The easiest one is probably run_tests, but if you write your tests in a separate executable you can also use run_tests_main which parses command line arguments and exits with 0 in case of success, or an error number otherwise.

Integration within OUnit

OUnit is a popular unit-testing framework for OCaml. QCheck provides a sub-library qcheck-ounit with some helpers, in QCheck_ounit, to convert its random tests into OUnit tests that can be part of a wider test-suite.

let passing =
  QCheck.Test.make ~count:1000
    ~name:"list_rev_is_involutive"
    QCheck.(list small_nat)
    (fun l -> List.rev (List.rev l) = l);;

let failing =
  QCheck.Test.make ~count:10
    ~name:"fail_sort_id"
    QCheck.(list small_nat)
    (fun l -> l = List.sort compare l);;

let _ =
  let open OUnit in
  run_test_tt_main
    ("tests" >:::
       List.map QCheck_ounit.to_ounit_test [passing; failing])
Note
 the package qcheck contains the module QCheck_runner which contains both custom runners and OUnit-based runners.

Integration within alcotest

Alcotest is a simple and colorful test framework for OCaml. QCheck now provides a sub-library qcheck-alcotest to easily integrate into an alcotest test suite:

let passing =
  QCheck.Test.make ~count:1000
    ~name:"list_rev_is_involutive"
    QCheck.(list small_int)
    (fun l -> List.rev (List.rev l) = l);;

let failing =
  QCheck.Test.make ~count:10
    ~name:"fail_sort_id"
    QCheck.(list small_int)
    (fun l -> l = List.sort compare l);;


let () =
  let suite =
    List.map QCheck_alcotest.to_alcotest
      [ passing; failing]
  in
  Alcotest.run "my test" [
    "suite", suite
  ]

Integration within Rely

Rely is a Jest-inspire native reason testing framework. @reason-native/qcheck-rely is available via NPM and provides matchers for the easy use of qCheck within Rely.

open TestFramework;
open QCheckRely;

let {describe} = extendDescribe(QCheckRely.Matchers.matchers);

describe("qcheck-rely", ({test}) => {
  test("passing test", ({expect}) => {
    let passing =
      QCheck.Test.make(
        ~count=1000,
        ~name="list_rev_is_involutive",
        QCheck.(list(small_int)),
        l =>
        List.rev(List.rev(l)) == l
      );
    expect.ext.qCheckTest(passing);
    ();
  });
  test("failing test", ({expect}) => {
    let failing =
      QCheck.Test.make(
        ~count=10, ~name="fail_sort_id", QCheck.(list(small_int)), l =>
        l == List.sort(compare, l)
      );

    expect.ext.qCheckTest(failing);
    ();
  });
});

Deriver

A ppx_deriver is provided to derive QCheck generators from a type declaration.

type tree = Leaf of int | Node of tree * tree
[@@deriving qcheck]

See the according README for more information and examples.

Compatibility notes

Starting with 0.9, the library is split into several components:

  • qcheck-core depends only on unix and bytes. It contains the module QCheck and a QCheck_base_runner module with our custom runners.

  • qcheck-ounit provides an integration layer for OUnit

  • qcheck provides a compatibility API with older versions of qcheck, using both qcheck-core and qcheck-ounit. It provides QCheck_runner which is similar to older versions and contains both custom and Ounit-based runners.

  • qcheck-alcotest provides an integration layer with alcotest

Normally, for contributors, opam pin https://github.com/c-cube/qcheck will pin all these packages.