Dynamic Tests

The standard @Test annotation in JUnit Jupiter described in Annotations is very similar to the @Test annotation in JUnit 4. Both describe methods that implement test cases. These test cases are static in the sense that they are fully specified at compile time, and their behavior cannot be changed by anything happening at runtime. Assumptions provide a basic form of dynamic behavior but are intentionally rather limited in their expressiveness.

In addition to these standard tests a completely new kind of test programming model has been introduced in JUnit Jupiter. This new kind of test is a dynamic test which is generated at runtime by a factory method that is annotated with @TestFactory.

In contrast to @Test methods, a @TestFactory method is not itself a test case but rather a factory for test cases. Thus, a dynamic test is the product of a factory. Technically speaking, a @TestFactory method must return a single DynamicNode or a stream of DynamicNode instances or any of its subclasses. In this context, a "stream" is anything that JUnit can reliably convert into a Stream, such as Stream, Collection, Iterator, Iterable, an array of objects, or any type that provides an iterator(): Iterator method (such as, for example, a kotlin.sequences.Sequence).

Instantiable subclasses of DynamicNode are DynamicContainer and DynamicTest. DynamicContainer instances are composed of a display name and a list of dynamic child nodes, enabling the creation of arbitrarily nested hierarchies of dynamic nodes. DynamicTest instances will be executed lazily, enabling dynamic and even non-deterministic generation of test cases.

Any Stream returned by a @TestFactory will be properly closed by calling stream.close(), making it safe to use a resource such as Files.lines().

As with @Test methods, @TestFactory methods must not be private or static and may optionally declare parameters to be resolved by ParameterResolvers.

A DynamicTest is a test case generated at runtime. It is composed of a display name and an Executable. Executable is a @FunctionalInterface which means that the implementations of dynamic tests can be provided as lambda expressions or method references.

Dynamic Test Lifecycle
The execution lifecycle of a dynamic test is quite different than it is for a standard @Test case. Specifically, there are no lifecycle callbacks for individual dynamic tests. This means that @BeforeEach and @AfterEach methods and their corresponding extension callbacks are executed for the @TestFactory method but not for each dynamic test. In other words, if you access fields from the test instance within a lambda expression for a dynamic test, those fields will not be reset by callback methods or extensions between the execution of individual dynamic tests generated by the same @TestFactory method.

Dynamic Test Examples

The following DynamicTestsDemo class demonstrates several examples of test factories and dynamic tests.

The first method returns an invalid return type and will cause a warning to be reported by JUnit during test discovery. Such methods are not executed.

The next six methods demonstrate the generation of a Collection, Iterable, Iterator, array, or Stream of DynamicTest instances. Most of these examples do not really exhibit dynamic behavior but merely demonstrate the supported return types in principle. However, dynamicTestsFromStream() and dynamicTestsFromIntStream() demonstrate how to generate dynamic tests for a given set of strings or a range of input numbers.

The next method is truly dynamic in nature. generateRandomNumberOfTests() implements an Iterator that generates random numbers, a display name generator, and a test executor and then provides all three to DynamicTest.stream(). Although the non-deterministic behavior of generateRandomNumberOfTests() is of course in conflict with test repeatability and should thus be used with care, it serves to demonstrate the expressiveness and power of dynamic tests.

The next method is similar to generateRandomNumberOfTests() in terms of flexibility; however, dynamicTestsFromStreamFactoryMethod() generates a stream of dynamic tests from an existing Stream via the DynamicTest.stream() factory method.

For demonstration purposes, the dynamicNodeSingleTest() method generates a single DynamicTest instead of a stream, and the dynamicNodeSingleContainer() method generates a nested hierarchy of dynamic tests utilizing DynamicContainer.

import static example.util.StringUtils.isPalindrome;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertFalse;
import static org.junit.jupiter.api.Assertions.assertNotNull;
import static org.junit.jupiter.api.Assertions.assertTrue;
import static org.junit.jupiter.api.DynamicContainer.dynamicContainer;
import static org.junit.jupiter.api.DynamicTest.dynamicTest;

import java.util.Arrays;
import java.util.Collection;
import java.util.Iterator;
import java.util.List;
import java.util.Random;
import java.util.function.Function;
import java.util.stream.IntStream;
import java.util.stream.Stream;

import example.util.Calculator;

import org.junit.jupiter.api.DynamicNode;
import org.junit.jupiter.api.DynamicTest;
import org.junit.jupiter.api.Tag;
import org.junit.jupiter.api.TestFactory;
import org.junit.jupiter.api.function.ThrowingConsumer;

class DynamicTestsDemo {

	private final Calculator calculator = new Calculator();

	// This method will not be executed but produce a warning
	@TestFactory
	List<String> dynamicTestsWithInvalidReturnType() {
		return Arrays.asList("Hello");
	}

	@TestFactory
	Collection<DynamicTest> dynamicTestsFromCollection() {
		return Arrays.asList(
			dynamicTest("1st dynamic test", () -> assertTrue(isPalindrome("madam"))),
			dynamicTest("2nd dynamic test", () -> assertEquals(4, calculator.multiply(2, 2)))
		);
	}

	@TestFactory
	Iterable<DynamicTest> dynamicTestsFromIterable() {
		return Arrays.asList(
			dynamicTest("3rd dynamic test", () -> assertTrue(isPalindrome("madam"))),
			dynamicTest("4th dynamic test", () -> assertEquals(4, calculator.multiply(2, 2)))
		);
	}

	@TestFactory
	Iterator<DynamicTest> dynamicTestsFromIterator() {
		return Arrays.asList(
			dynamicTest("5th dynamic test", () -> assertTrue(isPalindrome("madam"))),
			dynamicTest("6th dynamic test", () -> assertEquals(4, calculator.multiply(2, 2)))
		).iterator();
	}

	@TestFactory
	DynamicTest[] dynamicTestsFromArray() {
		return new DynamicTest[] {
			dynamicTest("7th dynamic test", () -> assertTrue(isPalindrome("madam"))),
			dynamicTest("8th dynamic test", () -> assertEquals(4, calculator.multiply(2, 2)))
		};
	}

	@TestFactory
	Stream<DynamicTest> dynamicTestsFromStream() {
		return Stream.of("racecar", "radar", "mom", "dad")
			.map(text -> dynamicTest(text, () -> assertTrue(isPalindrome(text))));
	}

	@TestFactory
	Stream<DynamicTest> dynamicTestsFromIntStream() {
		// Generates tests for the first 10 even integers.
		return IntStream.iterate(0, n -> n + 2).limit(10)
			.mapToObj(n -> dynamicTest("test" + n, () -> assertEquals(0, n % 2)));
	}

	@TestFactory
	Stream<DynamicTest> generateRandomNumberOfTests() {

		// Generates random positive integers between 0 and 100 until
		// a number evenly divisible by 7 is encountered.
		Iterator<Integer> inputGenerator = new Iterator<>() {

			Random random = new Random();
			int current;

			@Override
			public boolean hasNext() {
				current = random.nextInt(100);
				return current % 7 != 0;
			}

			@Override
			public Integer next() {
				return current;
			}
		};

		// Generates display names like: input:5, input:37, input:85, etc.
		Function<Integer, String> displayNameGenerator = (input) -> "input:" + input;

		// Executes tests based on the current input value.
		ThrowingConsumer<Integer> testExecutor = (input) -> assertTrue(input % 7 != 0);

		// Returns a stream of dynamic tests.
		return DynamicTest.stream(inputGenerator, displayNameGenerator, testExecutor);
	}

	@TestFactory
	Stream<DynamicTest> dynamicTestsFromStreamFactoryMethod() {
		// Stream of palindromes to check
		Stream<String> inputStream = Stream.of("racecar", "radar", "mom", "dad");

		// Generates display names like: racecar is a palindrome
		Function<String, String> displayNameGenerator = text -> text + " is a palindrome";

		// Executes tests based on the current input value.
		ThrowingConsumer<String> testExecutor = text -> assertTrue(isPalindrome(text));

		// Returns a stream of dynamic tests.
		return DynamicTest.stream(inputStream, displayNameGenerator, testExecutor);
	}

	@TestFactory
	Stream<DynamicNode> dynamicTestsWithContainers() {
		return Stream.of("A", "B", "C")
			.map(input -> dynamicContainer("Container " + input, Stream.of(
				dynamicTest("not null", () -> assertNotNull(input)),
				dynamicContainer("properties", Stream.of(
					dynamicTest("length > 0", () -> assertTrue(input.length() > 0)),
					dynamicTest("not empty", () -> assertFalse(input.isEmpty()))
				))
			)));
	}

	@TestFactory
	DynamicNode dynamicNodeSingleTest() {
		return dynamicTest("'pop' is a palindrome", () -> assertTrue(isPalindrome("pop")));
	}

	@TestFactory
	DynamicNode dynamicNodeSingleContainer() {
		return dynamicContainer("palindromes",
			Stream.of("racecar", "radar", "mom", "dad")
				.map(text -> dynamicTest(text, () -> assertTrue(isPalindrome(text)))
		));
	}
}

Dynamic Tests and Named

In some cases, it can be more natural to specify inputs together with a descriptive name using the Named API and the corresponding stream() factory methods on DynamicTest as shown in the first example below. The second example takes it one step further and allows to provide the code block that should be executed by implementing the Executable interface along with Named via the NamedExecutable base class.

import static example.util.StringUtils.isPalindrome;
import static org.junit.jupiter.api.Assertions.assertTrue;
import static org.junit.jupiter.api.Named.named;

import java.util.stream.Stream;

import org.junit.jupiter.api.DynamicTest;
import org.junit.jupiter.api.NamedExecutable;
import org.junit.jupiter.api.TestFactory;

public class DynamicTestsNamedDemo {

	@TestFactory
	Stream<DynamicTest> dynamicTestsFromStreamFactoryMethodWithNames() {
		// Stream of palindromes to check
		var inputStream = Stream.of(
			named("racecar is a palindrome", "racecar"),
			named("radar is also a palindrome", "radar"),
			named("mom also seems to be a palindrome", "mom"),
			named("dad is yet another palindrome", "dad")
		);

		// Returns a stream of dynamic tests.
		return DynamicTest.stream(inputStream, text -> assertTrue(isPalindrome(text)));
	}

	@TestFactory
	Stream<DynamicTest> dynamicTestsFromStreamFactoryMethodWithNamedExecutables() {
		// Stream of palindromes to check
		var inputStream = Stream.of("racecar", "radar", "mom", "dad")
				.map(PalindromeNamedExecutable::new);

		// Returns a stream of dynamic tests based on NamedExecutables.
		return DynamicTest.stream(inputStream);
	}

	record PalindromeNamedExecutable(String text) implements NamedExecutable {

		@Override
		public String getName() {
			return "'%s' is a palindrome".formatted(text);
		}

		@Override
		public void execute() {
			assertTrue(isPalindrome(text));
		}
	}
}

URI Test Sources for Dynamic Tests

The JUnit Platform provides TestSource, a representation of the source of a test or container used to navigate to its location by IDEs and build tools.

The TestSource for a dynamic test or dynamic container can be constructed from a java.net.URI which can be supplied via the DynamicTest.dynamicTest(String, URI, Executable) or DynamicContainer.dynamicContainer(String, URI, Stream) factory method, respectively. The URI will be converted to one of the following TestSource implementations.

ClasspathResourceSource

If the URI contains the classpath scheme — for example, classpath:/test/foo.xml?line=20,column=2.

DirectorySource

If the URI represents a directory present in the file system.

FileSource

If the URI represents a file present in the file system.

MethodSource

If the URI contains the method scheme and the fully qualified method name (FQMN) — for example, method:org.junit.Foo#bar(java.lang.String, java.lang.String[]). Please refer to the Javadoc for DiscoverySelectors.selectMethod for the supported formats for a FQMN.

ClassSource

If the URI contains the class scheme and the fully qualified class name — for example, class:org.junit.Foo?line=42.

UriSource

If none of the above TestSource implementations are applicable.