Dynamic Tests
The standard @Test annotation in JUnit Jupiter described in
Annotations is very similar to the @Test annotation in JUnit 4. Both
describe methods that implement test cases. These test cases are static in the sense that
they are fully specified at compile time, and their behavior cannot be changed by
anything happening at runtime. Assumptions provide a basic form of dynamic behavior but
are intentionally rather limited in their expressiveness.
In addition to these standard tests a completely new kind of test programming model has
been introduced in JUnit Jupiter. This new kind of test is a dynamic test which is
generated at runtime by a factory method that is annotated with @TestFactory.
In contrast to @Test methods, a @TestFactory method is not itself a test case but
rather a factory for test cases. Thus, a dynamic test is the product of a factory.
Technically speaking, a @TestFactory method must return a single DynamicNode or a
stream of DynamicNode instances or any of its subclasses. In this context, a "stream"
is anything that JUnit can reliably convert into a Stream, such as Stream,
Collection, Iterator, Iterable, an array of objects, or any type that provides an
iterator(): Iterator method (such as, for example, a kotlin.sequences.Sequence).
Instantiable subclasses of DynamicNode are DynamicContainer and DynamicTest.
DynamicContainer instances are composed of a display name and a list of dynamic child
nodes, enabling the creation of arbitrarily nested hierarchies of dynamic nodes.
DynamicTest instances will be executed lazily, enabling dynamic and even
non-deterministic generation of test cases.
Any Stream returned by a @TestFactory will be properly closed by calling
stream.close(), making it safe to use a resource such as Files.lines().
As with @Test methods, @TestFactory methods must not be private or static and may
optionally declare parameters to be resolved by ParameterResolvers.
A DynamicTest is a test case generated at runtime. It is composed of a display name
and an Executable. Executable is a @FunctionalInterface which means that the
implementations of dynamic tests can be provided as lambda expressions or method
references.
|
Dynamic Test Lifecycle
The execution lifecycle of a dynamic test is quite different than it is for a
standard @Test case. Specifically, there are no lifecycle callbacks for individual
dynamic tests. This means that @BeforeEach and @AfterEach methods and their
corresponding extension callbacks are executed for the @TestFactory method but not for
each dynamic test. In other words, if you access fields from the test instance within a
lambda expression for a dynamic test, those fields will not be reset by callback methods
or extensions between the execution of individual dynamic tests generated by the same
@TestFactory method.
|
Dynamic Test Examples
The following DynamicTestsDemo class demonstrates several examples of test factories
and dynamic tests.
The first method returns an invalid return type and will cause a warning to be reported by JUnit during test discovery. Such methods are not executed.
The next six methods demonstrate the generation of a Collection, Iterable, Iterator,
array, or Stream of DynamicTest instances. Most of these examples do not really
exhibit dynamic behavior but merely demonstrate the supported return types in principle.
However, dynamicTestsFromStream() and dynamicTestsFromIntStream() demonstrate how to
generate dynamic tests for a given set of strings or a range of input numbers.
The next method is truly dynamic in nature. generateRandomNumberOfTests() implements an
Iterator that generates random numbers, a display name generator, and a test executor
and then provides all three to DynamicTest.stream(). Although the non-deterministic
behavior of generateRandomNumberOfTests() is of course in conflict with test
repeatability and should thus be used with care, it serves to demonstrate the
expressiveness and power of dynamic tests.
The next method is similar to generateRandomNumberOfTests() in terms of flexibility;
however, dynamicTestsFromStreamFactoryMethod() generates a stream of dynamic tests from
an existing Stream via the DynamicTest.stream() factory method.
For demonstration purposes, the dynamicNodeSingleTest() method generates a single
DynamicTest instead of a stream, and the dynamicNodeSingleContainer() method generates
a nested hierarchy of dynamic tests utilizing DynamicContainer.
import static example.util.StringUtils.isPalindrome;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertFalse;
import static org.junit.jupiter.api.Assertions.assertNotNull;
import static org.junit.jupiter.api.Assertions.assertTrue;
import static org.junit.jupiter.api.DynamicContainer.dynamicContainer;
import static org.junit.jupiter.api.DynamicTest.dynamicTest;
import static org.junit.jupiter.api.parallel.ExecutionMode.CONCURRENT;
import static org.junit.jupiter.api.parallel.ExecutionMode.SAME_THREAD;
import java.util.Arrays;
import java.util.Collection;
import java.util.Iterator;
import java.util.List;
import java.util.Random;
import java.util.function.Function;
import java.util.stream.IntStream;
import java.util.stream.Stream;
import example.util.Calculator;
import org.junit.jupiter.api.DynamicNode;
import org.junit.jupiter.api.DynamicTest;
import org.junit.jupiter.api.Tag;
import org.junit.jupiter.api.TestFactory;
import org.junit.jupiter.api.function.ThrowingConsumer;
import org.junit.jupiter.api.parallel.Execution;
class DynamicTestsDemo {
private final Calculator calculator = new Calculator();
// This method will not be executed but produce a warning
@TestFactory
List<String> dynamicTestsWithInvalidReturnType() {
return Arrays.asList("Hello");
}
@TestFactory
Collection<DynamicTest> dynamicTestsFromCollection() {
return Arrays.asList(
dynamicTest("1st dynamic test", () -> assertTrue(isPalindrome("madam"))),
dynamicTest("2nd dynamic test", () -> assertEquals(4, calculator.multiply(2, 2)))
);
}
@TestFactory
Iterable<DynamicTest> dynamicTestsFromIterable() {
return Arrays.asList(
dynamicTest("3rd dynamic test", () -> assertTrue(isPalindrome("madam"))),
dynamicTest("4th dynamic test", () -> assertEquals(4, calculator.multiply(2, 2)))
);
}
@TestFactory
Iterator<DynamicTest> dynamicTestsFromIterator() {
return Arrays.asList(
dynamicTest("5th dynamic test", () -> assertTrue(isPalindrome("madam"))),
dynamicTest("6th dynamic test", () -> assertEquals(4, calculator.multiply(2, 2)))
).iterator();
}
@TestFactory
DynamicTest[] dynamicTestsFromArray() {
return new DynamicTest[] {
dynamicTest("7th dynamic test", () -> assertTrue(isPalindrome("madam"))),
dynamicTest("8th dynamic test", () -> assertEquals(4, calculator.multiply(2, 2)))
};
}
@TestFactory
Stream<DynamicTest> dynamicTestsFromStream() {
return Stream.of("racecar", "radar", "mom", "dad")
.map(text -> dynamicTest(text, () -> assertTrue(isPalindrome(text))));
}
@TestFactory
Stream<DynamicTest> dynamicTestsFromIntStream() {
// Generates tests for the first 10 even integers.
return IntStream.iterate(0, n -> n + 2).limit(10)
.mapToObj(n -> dynamicTest("test" + n, () -> assertEquals(0, n % 2)));
}
@TestFactory
Stream<DynamicTest> generateRandomNumberOfTests() {
// Generates random positive integers between 0 and 100 until
// a number evenly divisible by 7 is encountered.
Iterator<Integer> inputGenerator = new Iterator<>() {
Random random = new Random();
int current;
@Override
public boolean hasNext() {
current = random.nextInt(100);
return current % 7 != 0;
}
@Override
public Integer next() {
return current;
}
};
// Generates display names like: input:5, input:37, input:85, etc.
Function<Integer, String> displayNameGenerator = (input) -> "input:" + input;
// Executes tests based on the current input value.
ThrowingConsumer<Integer> testExecutor = (input) -> assertTrue(input % 7 != 0);
// Returns a stream of dynamic tests.
return DynamicTest.stream(inputGenerator, displayNameGenerator, testExecutor);
}
@TestFactory
Stream<DynamicTest> dynamicTestsFromStreamFactoryMethod() {
// Stream of palindromes to check
Stream<String> inputStream = Stream.of("racecar", "radar", "mom", "dad");
// Generates display names like: racecar is a palindrome
Function<String, String> displayNameGenerator = text -> text + " is a palindrome";
// Executes tests based on the current input value.
ThrowingConsumer<String> testExecutor = text -> assertTrue(isPalindrome(text));
// Returns a stream of dynamic tests.
return DynamicTest.stream(inputStream, displayNameGenerator, testExecutor);
}
@TestFactory
Stream<DynamicNode> dynamicTestsWithContainers() {
return Stream.of("A", "B", "C")
.map(input -> dynamicContainer("Container " + input, Stream.of(
dynamicTest("not null", () -> assertNotNull(input)),
dynamicContainer("properties", Stream.of(
dynamicTest("length > 0", () -> assertTrue(input.length() > 0)),
dynamicTest("not empty", () -> assertFalse(input.isEmpty()))
))
)));
}
@TestFactory
DynamicNode dynamicNodeSingleTest() {
return dynamicTest("'pop' is a palindrome", () -> assertTrue(isPalindrome("pop")));
}
@TestFactory
DynamicNode dynamicNodeSingleContainer() {
return dynamicContainer("palindromes",
Stream.of("racecar", "radar", "mom", "dad")
.map(text -> dynamicTest(text, () -> assertTrue(isPalindrome(text)))
));
}
}
Dynamic Tests and Named
In some cases, it can be more natural to specify inputs together with a descriptive name
using the Named API and the corresponding stream() factory methods on DynamicTest as
shown in the first example below. The second example takes it one step further and allows
to provide the code block that should be executed by implementing the Executable
interface along with Named via the NamedExecutable base class.
import static example.util.StringUtils.isPalindrome;
import static org.junit.jupiter.api.Assertions.assertTrue;
import static org.junit.jupiter.api.Named.named;
import java.util.stream.Stream;
import org.junit.jupiter.api.DynamicTest;
import org.junit.jupiter.api.NamedExecutable;
import org.junit.jupiter.api.TestFactory;
public class DynamicTestsNamedDemo {
@TestFactory
Stream<DynamicTest> dynamicTestsFromStreamFactoryMethodWithNames() {
// Stream of palindromes to check
var inputStream = Stream.of(
named("racecar is a palindrome", "racecar"),
named("radar is also a palindrome", "radar"),
named("mom also seems to be a palindrome", "mom"),
named("dad is yet another palindrome", "dad")
);
// Returns a stream of dynamic tests.
return DynamicTest.stream(inputStream, text -> assertTrue(isPalindrome(text)));
}
@TestFactory
Stream<DynamicTest> dynamicTestsFromStreamFactoryMethodWithNamedExecutables() {
// Stream of palindromes to check
var inputStream = Stream.of("racecar", "radar", "mom", "dad")
.map(PalindromeNamedExecutable::new);
// Returns a stream of dynamic tests based on NamedExecutables.
return DynamicTest.stream(inputStream);
}
record PalindromeNamedExecutable(String text) implements NamedExecutable {
@Override
public String getName() {
return "'%s' is a palindrome".formatted(text);
}
@Override
public void execute() {
assertTrue(isPalindrome(text));
}
}
}
URI Test Sources for Dynamic Tests
The JUnit Platform provides TestSource, a representation of the source of a test or
container used to navigate to its location by IDEs and build tools.
The TestSource for a dynamic test or dynamic container can be constructed from a
java.net.URI which can be supplied via the DynamicTest.dynamicTest(String, URI,
Executable) or DynamicContainer.dynamicContainer(String, URI, Stream) factory method,
respectively. The URI will be converted to one of the following TestSource
implementations.
ClasspathResourceSource-
If the
URIcontains theclasspathscheme — for example,classpath:/test/foo.xml?line=20,column=2. DirectorySource-
If the
URIrepresents a directory present in the file system. FileSource-
If the
URIrepresents a file present in the file system. MethodSource-
If the
URIcontains themethodscheme and the fully qualified method name (FQMN) — for example,method:org.junit.Foo#bar(java.lang.String, java.lang.String[]). Please refer to the Javadoc forDiscoverySelectors.selectMethodfor the supported formats for a FQMN. ClassSource-
If the
URIcontains theclassscheme and the fully qualified class name — for example,class:org.junit.Foo?line=42. UriSource-
If none of the above
TestSourceimplementations are applicable.
Parallel Execution
Dynamic tests and containers support
parallel execution. You can configure their
ExecutionMode by using the dynamicTest(Consumer) and dynamicContainer(Consumer)
factory methods as illustrated by the following example.
@TestFactory
@Execution(CONCURRENT) (1)
Stream<DynamicNode> dynamicTestsWithConfiguredExecutionMode() {
return Stream.of("A", "B", "C")
.map(input ->
dynamicContainer(outer -> outer
.displayName("Container " + input)
.children(
dynamicTest(config -> config
.displayName("not null")
.executionMode(SAME_THREAD) (2)
.executable(() -> assertNotNull(input))
),
dynamicContainer(inner -> inner
.displayName("properties")
.executionMode(CONCURRENT) (3)
.childExecutionMode(SAME_THREAD) (4)
.children(
dynamicTest(config -> config
.displayName("length > 0")
.executionMode(CONCURRENT) (5)
.executable(() -> assertTrue(input.length() > 0))
),
dynamicTest(config -> config
.displayName("not empty")
.executable(() -> assertFalse(input.isEmpty()))
)
)
)
)
)
);
}
Executing the above test factory method results in the following test tree and execution modes:
-
dynamicTestsWithConfiguredExecutionMode() —
CONCURRENT(from@Executionannotation)-
Container A —
CONCURRENT(from@Executionannotation)-
not null —
SAME_THREAD(fromexecutionMode(…)call) -
properties —
CONCURRENT(from@Executionannotation)-
length > 0 —
CONCURRENT(fromexecutionMode(…)call) -
not empty —
SAME_THREAD(fromchildExecutionMode(…)call)
-
-
-
… (same for "Container B" and "Container C")
-