JUnit4

Description

This task runs tests from the JUnit testing framework in versions 4.10 and above. It to some extent mimics the style of the default ANT's junit task, but adds the following features:

Note: The latest version of the JUnit framework can be found at http://www.junit.org. This task has been tested with JUnit 4.10 and later. Appropriate junit*.jar must be available in the task's classpath and in the tested code's classpath. See the examples below.

External references: More information about randomized testing.

Parameters

junit4 attempts to be a drop-in replacement for ANT's default junit task, but so attributes that make little sense have been deprecated and are ignored. Attributes specific to junit4 are highlighted in light green.

Attribute Description Required
printsummary Prints the summary of all executed, ignored and failed tests after all suites have been executed. To get per-suite summaries, configure a console report properly (see below). No; default is on.
fork Not used. ANT-junit drop-in compatibility only. Tests are always forked. --
forkmode Not used. ANT-junit drop-in compatibility only. Tests are always forked into one more more forked JVMs. Each suite is executed entirely on a single JVM. --
haltonerror Not used. ANT-junit drop-in compatibility only. Use haltonfailure which handles both errors and failures. --
errorproperty Not used. ANT-junit drop-in compatibility only. Use failureproperty which handles both errors and failures. --
haltonfailure Stop the build process if a test fails (errors are considered failures as well). No; default is on (ANT's junit has the default set to 'off').
failureproperty The name of a property to set in the event of a failure (errors are considered failures as well). No.
filtertrace Not used. ANT-junit drop-in compatibility only. Reports may provide their own filtering capabilities. --
timeout Not used. ANT-junit drop-in compatibility only. Test runners provide their own timeout facilities. --
maxmemory Maximum amount of memory to allocate to each forked VM (be careful if using lots of forks). Note: If you get java.lang.OutOfMemoryError: Java heap space in some of your tests then you need to raise the size like maxmemory="128m" No
jvm The command used to invoke the forked java virtual machines, the default is 'java'. The command is resolved by java.lang.Runtime.exec(). No; default is java.
dir The directory in which to invoke the forked JVMs in. This is by default the project's basedir. Because multiple JVMs may clash if they are generating files in the current working directory, one can use per-JVM directory by setting isolateWorkingDirectories attribute to true; in such case each JVM gets a sub-directory under whatever dir is defined. No
isolateWorkingDirectories If true current working directories are isolated for each forked JVM. See dir attribute too. No; default is true.
newenvironment Do not propagate the old environment when new environment variables are specified. No; default is false.
includeantruntime Not used. ANT-junit drop-in compatibility only. --
showoutput Not used. ANT-junit drop-in compatibility only. Configure console reports appropriately. --
outputtoformatters Not used. ANT-junit drop-in compatibility only. Configure console reports appropriately. --
tempdir Specify where to store temporary files. These temporary files include a list of suites passed to each slave JVM so it is usually wise to just leave this attribute unset - the default is to take the value of the dir attribute or the project's basedir. No
reloading Not used. ANT-junit drop-in compatibility only. --
clonevm Not used. ANT-junit drop-in compatibility only. --
logfailedtests Not used. ANT-junit drop-in compatibility only. Configure console reports appropriately to get just the failing tests, their output etc. --
enableTestListenerEvents Not used. ANT-junit drop-in compatibility only. --
parallelism The number of parallel slaves. Can be set to a constant max for the maximum number of cores returned from Runtime.availableProcessors or auto for sensible defaults depending on the number of cores.

Note: this setting forks physical JVM processes so it multiplies the requirements for heap memory, IO, etc.

No; Can be set to any integer or: 'max', 'auto'. The default is '1' (sequential execution in a forked JVM).
dynamicAssignmentRatio Specifies the ratio of suites moved to dynamic assignment list (job-stealing). A dynamic assignment list dispatches suites to the first idle slave JVM. Theoretically this is an optimal strategy, but it is usually better to have some static assignments to avoid communication costs.

A ratio of 0 means only static assignments are used. A ratio of 1 means only dynamic assignments are used.

The list of dynamic assignments is sorted by decreasing cost (always) and is inherently prone to race conditions in distributing suites. Should there be an error based on suite-dependency it will not be directly repeatable. In such case use the per-slave-jvm list of suites file dumped to disk for each slave JVM. See leaveTemporary attribute.

No; default is '0.25' (25% of all suites are assigned dynamically).
seed Specify random seed for anything that is randomized in junit4. The order of suites execution and suite-JVM assignments are a result of this seed for example.

The master seed is also passed to slave JVMs as a system property (to bootstrap randomized runner).

No; default is a randomly generated seed.
prefix Initializes custom prefix for all randomized runner properties. This must be consistent across all junit4 invocations if done from the same classpath. Use only when REALLY needed. No; default is randomized runner's prefix.
shuffleOnSlave Predictably shuffle tests order after balancing. This will help in spreading lighter and heavier tests over a single slave's execution timeline while still keeping the same tests order depending on the seed. See nested elements for configuring load balancers. No; the default is 'true'.
leaveTemporary Leave temporary junit4 files after the task quits. This can be used to trace the exact order of suites executed on each forked JVM for example. No; default is 'false'.

Nested Elements

The <junit4> task supports a nested elements. Some of them are compatible with ANT's default <junit> task while others are unique to <junit4>.

Specifying classpath for tests

The <classpath> element represents a PATH like structure to be used for each forked JVM. Note that classpath should include suiteable junit*.jar, otherwise your tests wont run.

Specifying test suite classes

Unlike standard <junit> task, <junit4> defines suite classes as resources to .class files. These resources can originate from any resource collection and there can be any number of resource collections with such resources. For example this definition includes all suite classes from a ZIP file (JAR file), excluding nested classes:

jvmarg

Additional parameters may be passed to the new JVM via nested <jvmarg> elements. For example:


would run the JVM with a serial garbage collector on HotSpot JVM.

sysproperty

Use nested <sysproperty> elements to specify system properties required by the tests. These properties will be made available to the forked JVMs during their execution. The attributes for this element are the same as for environment variables.

would run the test in ANT's VM and make the project's basedir property available to the test (note the current working directory of a forked JVM will not be the basedir typically).

syspropertyset

You can specify a set of properties to be used as system properties with syspropertysets.

env

It is possible to specify environment variables to pass to the forked VM via nested <env> elements. For a description of the <env> element's attributes, see the description in the exec task.

bootclasspath

The location of bootstrap class files can be specified using this PATH like structure.

assertions

You can control enablement of Java 1.4 assertions with an <assertions> subelement.

listeners and reports

There is no notion of "reports" in junit4. Reports are just listeners that attach to the even stream and can produce something (to the console, file or otherwise). You can also attach your own listeners if you like (although you'd have to peek at the code of existing listeners to get the feeling how this is done -- communication is via events not interfaces).

report-text

There are a few predefined listeners to choose from. By far the most common will be report-text which produces console output (or plain text file). A plain text report configuration may look like this:

The level of verbosity of the output can be fine-tuned to one's needs by disabling or enabling test statuses and suite summaries or each test's output. Experimenting highly recommended.

report-ant-xml

Another listener is report-ant-xml which can produce XML files suitable for junitreport task (although we highly recommend using the built-in JSON report.

report-json

Yet another built-in listener is producing a modern JSON output with test data and an accompanying HTML5 file for visualizing it:

An example of a HTML5 report produced by the above can be seen here, for example.

Load balancing of test suites

Suites can be scattered across forked JVMs randomly or using a greedy load-balancing algorithm (and followed by job-stealing if needed). For load balancing, previous execution statistics will be needed. These statistics should ideally come from a single machine and multiple executions so that they average the influence of the environment etc.

A dedicated listener is used to collect and update statistics. The configuration below shows both the listener and a balancer configuration. The balancer uses multiple sources for collecting statistics (for example precomputed statistics and previous local runs):