JUnit Platform Test Kit

The junit-platform-testkit artifact provides support for executing a test plan on the JUnit Platform and then verifying the expected results. As of JUnit Platform 1.14.2-SNAPSHOT, this support is limited to the execution of a single TestEngine (see Engine Test Kit).

Engine Test Kit

The org.junit.platform.testkit.engine package provides support for discovering and executing a TestPlan for a given TestEngine running on the JUnit Platform and then accessing the results via convenient result objects. For execution, a fluent API may be used to verify the expected execution events were received. The key entry point into this API is the EngineTestKit which provides static factory methods named engine(), discover(), and execute(). It is recommended that you select one of the engine() variants to benefit from the fluent API for building a LauncherDiscoveryRequest.

If you prefer to use the LauncherDiscoveryRequestBuilder from the Launcher API to build your LauncherDiscoveryRequest, you must use one of the discover() or execute() variants in EngineTestKit.

The following test class written using JUnit Jupiter will be used in subsequent examples.

import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assumptions.assumeTrue;

import example.util.Calculator;

import org.junit.jupiter.api.Disabled;
import org.junit.jupiter.api.MethodOrderer.OrderAnnotation;
import org.junit.jupiter.api.Order;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.TestMethodOrder;

@TestMethodOrder(OrderAnnotation.class)
public class ExampleTestCase {

	private final Calculator calculator = new Calculator();

	@Test
	@Disabled("for demonstration purposes")
	@Order(1)
	void skippedTest() {
		// skipped ...
	}

	@Test
	@Order(2)
	void succeedingTest() {
		assertEquals(42, calculator.multiply(6, 7));
	}

	@Test
	@Order(3)
	void abortedTest() {
		assumeTrue("abc".contains("Z"), "abc does not contain Z");
		// aborted ...
	}

	@Test
	@Order(4)
	void failingTest() {
		// The following throws an ArithmeticException: "/ by zero"
		calculator.divide(1, 0);
	}

}

For the sake of brevity, the following sections demonstrate how to test JUnit’s own JupiterTestEngine whose unique engine ID is "junit-jupiter". If you want to test your own TestEngine implementation, you need to use its unique engine ID. Alternatively, you may test your own TestEngine by supplying an instance of it to the EngineTestKit.engine(TestEngine) static factory method.

Verifying Test Discovery

The following test demonstrates how to verify that a TestPlan was discovered as expected by the JUnit Jupiter TestEngine.

import static java.util.Collections.emptyList;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.platform.engine.discovery.DiscoverySelectors.selectClass;

import example.ExampleTestCase;

import org.junit.jupiter.api.Test;
import org.junit.platform.testkit.engine.EngineDiscoveryResults;
import org.junit.platform.testkit.engine.EngineTestKit;

class EngineTestKitDiscoveryDemo {

	@Test
	void verifyJupiterDiscovery() {
		EngineDiscoveryResults results = EngineTestKit.engine("junit-jupiter") (1)
				.selectors(selectClass(ExampleTestCase.class)) (2)
				.discover(); (3)

		assertEquals("JUnit Jupiter", results.getEngineDescriptor().getDisplayName()); (4)
		assertEquals(emptyList(), results.getDiscoveryIssues()); (5)
	}

}

1	Select the JUnit Jupiter `TestEngine`.
2	Select the `ExampleTestCase` test class.
3	Discover the `TestPlan`.
4	Assert engine root descriptor has expected display name.
5	Assert no discovery issues were encountered.

Asserting Execution Statistics

One of the most common features of the Test Kit is the ability to assert statistics against events fired during the execution of a TestPlan. The following tests demonstrate how to assert statistics for containers and tests in the JUnit Jupiter TestEngine. For details on what statistics are available, consult the Javadoc for EventStatistics.

import static org.junit.platform.engine.discovery.DiscoverySelectors.selectClass;

import example.ExampleTestCase;

import org.junit.jupiter.api.Test;
import org.junit.platform.testkit.engine.EngineTestKit;

class EngineTestKitStatisticsDemo {

	@Test
	void verifyJupiterContainerStats() {
		EngineTestKit
			.engine("junit-jupiter") (1)
			.selectors(selectClass(ExampleTestCase.class)) (2)
			.execute() (3)
			.containerEvents() (4)
			.assertStatistics(stats -> stats.started(2).succeeded(2)); (5)
	}

	@Test
	void verifyJupiterTestStats() {
		EngineTestKit
			.engine("junit-jupiter") (1)
			.selectors(selectClass(ExampleTestCase.class)) (2)
			.execute() (3)
			.testEvents() (6)
			.assertStatistics(stats ->
				stats.skipped(1).started(3).succeeded(1).aborted(1).failed(1)); (7)
	}

}

1	Select the JUnit Jupiter `TestEngine`.
2	Select the `ExampleTestCase` test class.
3	Execute the `TestPlan`.
4	Filter by container events.
5	Assert statistics for container events.
6	Filter by test events.
7	Assert statistics for test events.

In the verifyJupiterContainerStats() test method, the counts for the started and succeeded statistics are 2 since the JupiterTestEngine and the ExampleTestCase class are both considered containers.

Asserting Events

If you find that asserting statistics alone is insufficient for verifying the expected behavior of test execution, you can work directly with the recorded Event elements and perform assertions against them.

For example, if you want to verify the reason that the skippedTest() method in ExampleTestCase was skipped, you can do that as follows.

The assertThatEvents() method in the following example is a shortcut for org.assertj.core.api.Assertions.assertThat(events.list()) from the AssertJ assertion library.

For details on what conditions are available for use with AssertJ assertions against events, consult the Javadoc for EventConditions.

import static org.junit.platform.engine.discovery.DiscoverySelectors.selectMethod;
import static org.junit.platform.testkit.engine.EventConditions.event;
import static org.junit.platform.testkit.engine.EventConditions.skippedWithReason;
import static org.junit.platform.testkit.engine.EventConditions.test;

import example.ExampleTestCase;

import org.junit.jupiter.api.Test;
import org.junit.platform.testkit.engine.EngineTestKit;
import org.junit.platform.testkit.engine.Events;

class EngineTestKitSkippedMethodDemo {

	@Test
	void verifyJupiterMethodWasSkipped() {
		String methodName = "skippedTest";

		Events testEvents = EngineTestKit (5)
			.engine("junit-jupiter") (1)
			.selectors(selectMethod(ExampleTestCase.class, methodName)) (2)
			.execute() (3)
			.testEvents(); (4)

		testEvents.assertStatistics(stats -> stats.skipped(1)); (6)

		testEvents.assertThatEvents() (7)
			.haveExactly(1, event(test(methodName),
				skippedWithReason("for demonstration purposes")));
	}

}

1	Select the JUnit Jupiter `TestEngine`.
2	Select the `skippedTest()` method in the `ExampleTestCase` test class.
3	Execute the `TestPlan`.
4	Filter by test events.
5	Save the test `Events` to a local variable.
6	Optionally assert the expected statistics.
7	Assert that the recorded test events contain exactly one skipped test named `skippedTest` with `"for demonstration purposes"` as the reason.

If you want to verify the type of exception thrown from the failingTest() method in ExampleTestCase, you can do that as follows.

For details on what conditions are available for use with AssertJ assertions against events and execution results, consult the Javadoc for EventConditions and TestExecutionResultConditions, respectively.

import static org.junit.platform.engine.discovery.DiscoverySelectors.selectClass;
import static org.junit.platform.testkit.engine.EventConditions.event;
import static org.junit.platform.testkit.engine.EventConditions.finishedWithFailure;
import static org.junit.platform.testkit.engine.EventConditions.test;
import static org.junit.platform.testkit.engine.TestExecutionResultConditions.instanceOf;
import static org.junit.platform.testkit.engine.TestExecutionResultConditions.message;

import example.ExampleTestCase;

import org.junit.jupiter.api.Test;
import org.junit.platform.testkit.engine.EngineTestKit;

class EngineTestKitFailedMethodDemo {

	@Test
	void verifyJupiterMethodFailed() {
		EngineTestKit.engine("junit-jupiter") (1)
			.selectors(selectClass(ExampleTestCase.class)) (2)
			.execute() (3)
			.testEvents() (4)
			.assertThatEvents().haveExactly(1, (5)
				event(test("failingTest"),
					finishedWithFailure(
						instanceOf(ArithmeticException.class), message(it -> it.endsWith("by zero")))));
	}

}

1	Select the JUnit Jupiter `TestEngine`.
2	Select the `ExampleTestCase` test class.
3	Execute the `TestPlan`.
4	Filter by test events.
5	Assert that the recorded test events contain exactly one failing test named `failingTest` with an exception of type `ArithmeticException` and an error message that ends with `"/ by zero"`.

Although typically unnecessary, there are times when you need to verify all of the events fired during the execution of a TestPlan. The following test demonstrates how to achieve this via the assertEventsMatchExactly() method in the EngineTestKit API.

Since assertEventsMatchExactly() matches conditions exactly in the order in which the events were fired, ExampleTestCase has been annotated with @TestMethodOrder(OrderAnnotation.class) and each test method has been annotated with @Order(…). This allows us to enforce the order in which the test methods are executed, which in turn allows our verifyAllJupiterEvents() test to be reliable.

If you want to do a partial match with or without ordering requirements, you can use the methods assertEventsMatchLooselyInOrder() and assertEventsMatchLoosely(), respectively.

import static org.junit.platform.engine.discovery.DiscoverySelectors.selectClass;
import static org.junit.platform.testkit.engine.EventConditions.abortedWithReason;
import static org.junit.platform.testkit.engine.EventConditions.container;
import static org.junit.platform.testkit.engine.EventConditions.engine;
import static org.junit.platform.testkit.engine.EventConditions.event;
import static org.junit.platform.testkit.engine.EventConditions.finishedSuccessfully;
import static org.junit.platform.testkit.engine.EventConditions.finishedWithFailure;
import static org.junit.platform.testkit.engine.EventConditions.skippedWithReason;
import static org.junit.platform.testkit.engine.EventConditions.started;
import static org.junit.platform.testkit.engine.EventConditions.test;
import static org.junit.platform.testkit.engine.TestExecutionResultConditions.instanceOf;
import static org.junit.platform.testkit.engine.TestExecutionResultConditions.message;

import java.io.StringWriter;
import java.io.Writer;

import example.ExampleTestCase;

import org.junit.jupiter.api.Test;
import org.junit.platform.testkit.engine.EngineTestKit;
import org.opentest4j.TestAbortedException;

class EngineTestKitAllEventsDemo {

	@Test
	void verifyAllJupiterEvents() {
		Writer writer = // create a java.io.Writer for debug output

		EngineTestKit.engine("junit-jupiter") (1)
			.selectors(selectClass(ExampleTestCase.class)) (2)
			.execute() (3)
			.allEvents() (4)
			.debug(writer) (5)
			.assertEventsMatchExactly( (6)
				event(engine(), started()),
				event(container(ExampleTestCase.class), started()),
				event(test("skippedTest"), skippedWithReason("for demonstration purposes")),
				event(test("succeedingTest"), started()),
				event(test("succeedingTest"), finishedSuccessfully()),
				event(test("abortedTest"), started()),
				event(test("abortedTest"),
					abortedWithReason(instanceOf(TestAbortedException.class),
						message(m -> m.contains("abc does not contain Z")))),
				event(test("failingTest"), started()),
				event(test("failingTest"), finishedWithFailure(
					instanceOf(ArithmeticException.class), message(it -> it.endsWith("by zero")))),
				event(container(ExampleTestCase.class), finishedSuccessfully()),
				event(engine(), finishedSuccessfully()));
	}

}

1	Select the JUnit Jupiter `TestEngine`.
2	Select the `ExampleTestCase` test class.
3	Execute the `TestPlan`.
4	Filter by all events.
5	Print all events to the supplied `writer` for debugging purposes. Debug information can also be written to an `OutputStream` such as `System.out` or `System.err`.
6	Assert all events in exactly the order in which they were fired by the test engine.

The debug() invocation from the preceding example results in output similar to the following.

All Events:
	Event [type = STARTED, testDescriptor = JupiterEngineDescriptor: [engine:junit-jupiter], timestamp = 2018-12-14T12:45:14.082280Z, payload = null]
	Event [type = STARTED, testDescriptor = ClassTestDescriptor: [engine:junit-jupiter]/[class:example.ExampleTestCase], timestamp = 2018-12-14T12:45:14.089339Z, payload = null]
	Event [type = SKIPPED, testDescriptor = TestMethodTestDescriptor: [engine:junit-jupiter]/[class:example.ExampleTestCase]/[method:skippedTest()], timestamp = 2018-12-14T12:45:14.094314Z, payload = 'for demonstration purposes']
	Event [type = STARTED, testDescriptor = TestMethodTestDescriptor: [engine:junit-jupiter]/[class:example.ExampleTestCase]/[method:succeedingTest()], timestamp = 2018-12-14T12:45:14.095182Z, payload = null]
	Event [type = FINISHED, testDescriptor = TestMethodTestDescriptor: [engine:junit-jupiter]/[class:example.ExampleTestCase]/[method:succeedingTest()], timestamp = 2018-12-14T12:45:14.104922Z, payload = TestExecutionResult [status = SUCCESSFUL, throwable = null]]
	Event [type = STARTED, testDescriptor = TestMethodTestDescriptor: [engine:junit-jupiter]/[class:example.ExampleTestCase]/[method:abortedTest()], timestamp = 2018-12-14T12:45:14.106121Z, payload = null]
	Event [type = FINISHED, testDescriptor = TestMethodTestDescriptor: [engine:junit-jupiter]/[class:example.ExampleTestCase]/[method:abortedTest()], timestamp = 2018-12-14T12:45:14.109956Z, payload = TestExecutionResult [status = ABORTED, throwable = org.opentest4j.TestAbortedException: Assumption failed: abc does not contain Z]]
	Event [type = STARTED, testDescriptor = TestMethodTestDescriptor: [engine:junit-jupiter]/[class:example.ExampleTestCase]/[method:failingTest()], timestamp = 2018-12-14T12:45:14.110680Z, payload = null]
	Event [type = FINISHED, testDescriptor = TestMethodTestDescriptor: [engine:junit-jupiter]/[class:example.ExampleTestCase]/[method:failingTest()], timestamp = 2018-12-14T12:45:14.111217Z, payload = TestExecutionResult [status = FAILED, throwable = java.lang.ArithmeticException: / by zero]]
	Event [type = FINISHED, testDescriptor = ClassTestDescriptor: [engine:junit-jupiter]/[class:example.ExampleTestCase], timestamp = 2018-12-14T12:45:14.113731Z, payload = TestExecutionResult [status = SUCCESSFUL, throwable = null]]
	Event [type = FINISHED, testDescriptor = JupiterEngineDescriptor: [engine:junit-jupiter], timestamp = 2018-12-14T12:45:14.113806Z, payload = TestExecutionResult [status = SUCCESSFUL, throwable = null]]