Generators

Generators are at the very heart of Quickcheck. They provide the test data the runner executes characteristics with.

Distributions

A distribution is a function that returns values between -1.0 and 1.0. The concrete frequency of values is determined by the distribution function. All generators provided by Quickcheck use an implementation of Distribution to derive their respective values from.

From a technical point of view all generators could be run with the uniform distribution. Other distributions can be used to increase the likelihood of certain values. This is useful to increase the significance of generators.

The supported distributions are: uniform, inverted normal, positive normal and negative normal.



The diagram shows the different distribution functions supported by Quickcheck. It is an example output of an integer generator with values ranging from 0 to 10.

All values of the uniform distribution have the same probability to occur. The inverted normal distribution is a distribution function whose border values are the most likely. The function is defined based on the normal distribution. This function is inverted and cut. This is done to provide a pragmatic distribution function to test border cases where the lowest and highest values are of the most interest. For the positive normal distribution the values near 0.0 are the most probable and values near 1.0 respectively for the negative normal distribution.

Primitive generators

Primitive generator are the basic building blocks for custom generators. The static factory methods in the net.java.quickcheck.generator.PrimitiveGenerators class can be used to create primitive generators (Strings, Integer, Long, Double, Byte, Character, Date, Enum). Typically the factory provides methods to create a generator with minimum value, maximum value and distribution for each primitive type. Additionally there are convenience methods that use default values.

There are two special factory methods that do not denote a primitive type: fixed value and null. Fixed value generators will always return the given values. Null generator always returns null.

Combined Generators

Combined generators use other generators to build more complex generators. As with primitive generators there is the net.java.quickcheck.generator.CombinedGenerators class with static factory methods.

There are number of generators that create subtypes of java.util.Collection and arrays (List, Set, object Arrays, int arrays, byte array).

Two generators generate tuples: the pair and triple generators. (The Java generics somewhat discourages the creation of lager tuples).

The ensured values generator can be used to create a deterministic generator. This can be combined with a random generator that kicks in when the deterministic generator runs out of values.

The nullsAnd generator can be used to create a random mixture of null values and values returned by an other generator. (The generator has this funny name to form readable statements in test code. For example: nullsAnd(integers()).)

The unique values generator can be used to decorate another generator to ensure that a value is not created repeatedly. When the base generator returns a redundant value it will be retried. (At some point the unique generator will give up and throw an exception.)

The frequency generator can be used to combine multiple generators to one generator. Every added generator will have a weight and values are taken from of the generators according to their respective weight. The oneOf generator factory method is a convenience method to create a frequency generator from a set of generators with the same weight.

Building custom generators

Armed with this knowledge about primitive and combined generators you can now start to create your own custom generators.

The following example defines a generator for java.io.File objects.

class FileGenerator implements Generator<File> {

  Generator<File> roots = fixedValues(File.listRoots());
  Generator<List<String>> paths = nonEmptyLists(letterStrings());

  @Override
  public File next() {
    File f = roots.next();

    for (String p : paths.next()) f = new File(f, p);
    return f;
   }
}

An implementation of the Generator interface has to define the next method. This method returns the next generated value. It will be called by the Quickcheck runner when a test executes.

It uses the fixedValues generator to create the root node. The number of file system roots depends on the OS. Windows systems can have multiple roots whereas Unix systems have only one. The path of the file after the root is created from a list of strings. The nonEmptyList generator will create lists with at least one entry.

The next method implementation creates the file from a file representing the root node and adds to it the relative paths generated with the paths generator. The relative paths are repeatedly added in the for loop to create the absolute path of the resulting file.

It is always a good practice to keep the base generators as members and reuse these for multiple runs of the next method.

Iterable adapter - Ways to define characteristics

The current development version of Quickcheck for Java contains new Iterable adapter. This adapter lets you convert a Generator implementation into an Iterable. This Iteraterable can then be used in Java for loops. This makes definitions of tests much shorter as no inner class has to be used to define a characteristic.

When I started porting Quickcheck from Haskell I thought that there will be some closure-like facility in the Java language in 2 years. (After all more and more developers seemed to get exposed to the concept of functions and higher order functions. Most recent languages have some form of language construct to build new control structures in libraries.) So I did not bother myself too much with the usability problems that arise from the use of inner classes. In the meantime time it is obvious that Java will not have closures soon and maybe never will. So as long as we are using Java we have to get around this.

Quickcheck has now three different ways to express a characteristic: inner classes, annotations and for-loop expressions. Each of these solutions has its advantages and disadvantages. The example shows a characteristic defined for all three approaches.

@RunWith(QuickCheckRunner.class)
public class IterableExample {

  @Test
  public void classic() {
    forAll(PrimitiveGenerators.integers(), new AbstractCharacteristic<Integer>() {

      @Override
      protected void doSpecify(Integer any) throws Throwable {
        assertEquals(any * 2, any + any);
      }
    });
  }

  @ForAll(generatorMethod = "net.java.quickcheck.generator.PrimitiveGenerators.integers")
  public void annotation(Integer any) {
    assertEquals(any * 2, any + any);
  }

  @Test
  public void iterable() {
    for (Integer any : Iterables.toIterable(PrimitiveGenerators.integers())) {
      assertEquals(any * 2, any + any);
    }
  }
}

Inner classes are the standard implementation and fine if one can ignore the boilerplate code that is used to express a characteristic as an inner class.
Annotations are not type-safe (and therefore refactoring safe), they cannot be navigated in IDEs and the generator definition and use is separated.
The Iterable adapter is type-safe and has concise definition of the characteristic but it lacks all features of the Quickcheck runner. So the Iterator<T> created by the Iterable<T> adapter will produce a fixed number of values. As the Java for loop construct is used the context information for the test run is missing. Setup and tear down have to be called explicitly.

Defining the Collection.add contract in Quickcheck for Java

The Collection add method does define a contract that is expressed in javadoc.
It states:

• A collection implementation can throw an exception. The collection will stay unchanged and contains() will return false.
  
• The add method return value indicates if the collection changed as a result of the call. The collection will contain the element (based on equals()) but only if the collection changed the very instance will be part of the collection.
  

This textual form can be expressed as a Quickcheck characteristic.

@Test
public void collectionAdd() {
 for (Pair<Integer, List<Integer>> any : somePairs(integers(), lists(integers()))) {
  Integer element = any.getFirst();
  Collection<Integer> collection = any.getSecond();

  boolean changedCollection = false;
  boolean exceptionThrown = false;
  try {
   changedCollection = collection.add(element);
  } catch (Exception e) {
   assertException(e);
   exceptionThrown = true;
  }
  assertTrue(collection.contains(element) != exceptionThrown);
  assertTrue(changedCollection == containsInstance(collection,
    element));
 }
}

private void assertException(Exception e) {
 assertTrue(e instanceof UnsupportedOperationException
   || e instanceof ClassCastException
   || e instanceof IllegalArgumentException
   || e instanceof IllegalStateException);
}

private boolean containsInstance(Collection<?> collection, Object element) {
 for (Object e : collection) if (e == element) return true;
 return false;
}

So in this way a specification can be provided of an interface some third party has to implement. It's then up to the implementer to provide a generator for the implementation besides the working code.

This has obviously some overlap with contract-driven development and the assertions facility. One interesting reuse of the characteristic could be to take an implementation that states it fulfills a given contract and monitor the behaviour of the implementation at run time (i.e. not while test time) and to report violations of the contract.