Dynamic Tests – @TestFactory in Junit 5

Here in this tutorial I will tell you what are Dynamic Tests and @TestFactory in Junit 5 and how to create @TestFactory in Junit 5.

Test cases, annotated with @Test, are static in the sense that they are fully specified at compile time, and their behavior cannot be changed by anything happening at runtime.

In addition to these standard static tests a new kind of test programming model has been introduced in JUnit Jupiter. This new kind of test is a dynamic test which is generated at runtime by a factory method that is annotated with @TestFactory.

Unlike @Test methods, a @TestFactory method is not itself a test case but rather a factory for test cases. Therefore, a @TestFactory method must return a single DynamicNode or a Stream, Collection, Iterable, Iterator, or array of DynamicNode instances.

Instantiable subclasses of DynamicNode are DynamicContainer and DynamicTest. DynamicContainer instances are composed of a display name and a list of dynamic child nodes, enabling the creation of arbitrarily nested hierarchies of dynamic nodes. DynamicTest instances will be executed lazily, enabling dynamic and even non-deterministic generation of test cases.

Any Stream returned by a @TestFactory will be properly closed by calling stream.close(), making it safe to use a resource such as Files.lines().

@TestFactory methods must not be private or static and may optionally declare parameters to be resolved by ParameterResolvers.

A DynamicTest is a test case generated at runtime. It is composed of a display name and an Executable. Executable is a @FunctionalInterface which means that the implementations of dynamic tests can be provided as lambda expressions or method references.

The execution lifecycle of a dynamic test is quite different than it is for a standard @Test case. Specifically, there are no lifecycle callbacks for individual dynamic tests. This means that @BeforeEach and @AfterEach methods and their corresponding extension callbacks are executed for the @TestFactory method but not for each dynamic test. In other words, if you access fields from the test instance within a lambda expression for a dynamic test, those fields will not be reset by callback methods or extensions between the execution of individual dynamic tests generated by the same @TestFactory method.

Now I am going to show you examples on Dynamic Tests in Junit 5.

Prerequisites

Java at least 8, Junit 5.0.8-M1, Maven 3.6.3

Project Setup

Create a maven based project in your favorite IDE or tool. The name of the project is java-junit-dynamic-tests.

Dynamic Tests

The following methods are very simple examples that demonstrate the generation of a Collection, Iterable, Iterator of DynamicTest instances. However, these examples do not really exhibit dynamic behavior but merely demonstrate the supported return types in principle.

@TestFactory
Collection<DynamicTest> dynamicTestsMinMaxPalindromeCollection() {
	return Arrays.asList(
			dynamicTest("Find Max Test",
					() -> assertEquals(4, MinMaxElementFinder.findMax(new int[] { 1, 3, 4, 2 }))),
			dynamicTest("Find Min Test",
					() -> assertEquals(1, MinMaxElementFinder.findMin(new int[] { 1, 3, 4, 2 }))),
			dynamicTest("Palindrome 1", () -> assertTrue(checker.isPalindrome("madam"))),
			dynamicTest("Palindrome 2", () -> assertFalse(checker.isPalindrome("palindrome"))));
}

@TestFactory
Iterable<DynamicTest> dynamicTestsMinMaxPalindromeIterable() {
	return Arrays.asList(
			dynamicTest("Find Max Test",
					() -> assertEquals(4, MinMaxElementFinder.findMax(new int[] { 1, 3, 4, 2 }))),
			dynamicTest("Find Min Test",
					() -> assertEquals(1, MinMaxElementFinder.findMin(new int[] { 1, 3, 4, 2 }))),
			dynamicTest("Palindrome 1", () -> assertTrue(checker.isPalindrome("madam"))),
			dynamicTest("Palindrome 2", () -> assertFalse(checker.isPalindrome("palindrome"))));
}

@TestFactory
Iterator<DynamicTest> dynamicTestsMinMaxPalindromeIterator() {
	return Arrays.asList(
			dynamicTest("Find Max Test",
					() -> assertEquals(4, MinMaxElementFinder.findMax(new int[] { 1, 3, 4, 2 }))),
			dynamicTest("Find Min Test",
					() -> assertEquals(1, MinMaxElementFinder.findMin(new int[] { 1, 3, 4, 2 }))),
			dynamicTest("Palindrome 1", () -> assertTrue(checker.isPalindrome("madam"))),
			dynamicTest("Palindrome 2", () -> assertFalse(checker.isPalindrome("palindrome")))).iterator();
}

@TestFactory
DynamicTest[] dynamicTestsMinMaxPalindromeArray() {
	return new DynamicTest[] {
			dynamicTest("Find Max Test",
					() -> assertEquals(4, MinMaxElementFinder.findMax(new int[] { 1, 3, 4, 2 }))),
			dynamicTest("Find Min Test",
					() -> assertEquals(1, MinMaxElementFinder.findMin(new int[] { 1, 3, 4, 2 }))),
			dynamicTest("Palindrome 1", () -> assertTrue(checker.isPalindrome("madam"))),
			dynamicTest("Palindrome 2", () -> assertFalse(checker.isPalindrome("palindrome"))) };
}

The following methods demonstrate how easy it is to generate dynamic tests for a given set of strings or a range of input numbers.

@TestFactory
Stream<DynamicTest> dynamicTestsStream() {
	return Stream.of("madam", "mom", "dad")
			.map(str -> dynamicTest(str, () -> assertTrue(checker.isPalindrome(str))));
}

@TestFactory
Stream<DynamicTest> dynamicTestsIntStream() {
	return IntStream.iterate(0, n -> n + 2).limit(5)
			.mapToObj(n -> dynamicTest("Even Test" + n, () -> assertTrue(n % 2 == 0)));
}

@TestFactory
Stream<DynamicTest> dynamicTestsStreamFactoryMethod() {
	Stream<String> inputStream = Stream.of("madam", "mom", "dad");

	Function<String, String> displayNameGenerator = str -> str + " is a palindrome";

	ThrowingConsumer<String> testExecutor = str -> assertTrue(checker.isPalindrome(str));

	return DynamicTest.stream(inputStream, displayNameGenerator, testExecutor);
}

The following method generates a single DynamicTest instead of a stream:

@TestFactory
DynamicNode dynamicNodeSingle() {
	return dynamicTest("'push' is a not palindrome", () -> assertFalse(checker.isPalindrome("push")));
}

The following method generates a nested hierarchy of dynamic tests utilizing DynamicContainer:

@TestFactory
DynamicNode dynamicNodeSingleContainer() {
	return dynamicContainer("Palindromes", Stream.of("madam", "mom", "dad")
			.map(str -> dynamicTest(str, () -> assertTrue(checker.isPalindrome(str)))));
}

URI Test Sources for Dynamic Tests

The TestSource for a dynamic test or dynamic container can be constructed from a java.net.URI which can be supplied via the DynamicTest.dynamicTest(String, URI, Executable) or DynamicContainer.dynamicContainer(String, URI, Stream) factory method, respectively. The URI will be converted to one of the following TestSource implementations.

ClasspathResourceSource: If the URI contains the classpath scheme — for example, classpath:/test/foo.xml?line=20,column=2.

DirectorySource: If the URI represents a directory present in the file system.

FileSource: If the URI represents a file present in the file system.

MethodSource: If the URI contains the method scheme and the fully qualified method name (FQMN) — for example, method:org.junit.Foo#bar(java.lang.String, java.lang.String[]). Please refer to the Javadoc for DiscoverySelectors.selectMethod(String) for the supported formats for a FQMN.

UriSource: If none of the above TestSource implementations are applicable.

The example on URI for Test Sources for Dynamic Tests can be written as follows:

@TestFactory
Stream<DynamicTest> checkAllTextFiles() throws Exception {
	return Files.walk(Paths.get("src/test/resources/test"), 1).filter(path -> path.toString().endsWith(".txt"))
			.map(path -> dynamicTest("File: " + path.getFileName(), path.toUri(), () -> checkLineContent(path)));
}

private void checkLineContent(Path path) throws Exception {
	List<String> lines = Files.readAllLines(path);

	String expected = lines.get(0);
	String actual = new StringBuilder(lines.get(0)).reverse().toString();

	assertEquals(expected, actual, "String and its reverse should be equal");
}

The whole Junit class can be written as follows:

package com.roytuts.java.junit.dynamic.tests;

import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertFalse;
import static org.junit.jupiter.api.Assertions.assertNotNull;
import static org.junit.jupiter.api.Assertions.assertNull;
import static org.junit.jupiter.api.Assertions.assertTrue;
import static org.junit.jupiter.api.DynamicContainer.dynamicContainer;
import static org.junit.jupiter.api.DynamicTest.dynamicTest;

import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.util.Arrays;
import java.util.Collection;
import java.util.Iterator;
import java.util.List;
import java.util.function.Function;
import java.util.stream.IntStream;
import java.util.stream.Stream;

import org.junit.jupiter.api.DynamicNode;
import org.junit.jupiter.api.DynamicTest;
import org.junit.jupiter.api.TestFactory;
import org.junit.jupiter.api.function.ThrowingConsumer;

public class JunitDynamicTests {

	private final PalindromeChecker checker = new PalindromeChecker();

	@TestFactory
	Collection<DynamicTest> dynamicTestsMinMax() {
		return Arrays.asList(
				dynamicTest("Find Max Test",
						() -> assertEquals(4, MinMaxElementFinder.findMax(new int[] { 1, 3, 4, 2 }))),
				dynamicTest("Find Min Test",
						() -> assertEquals(1, MinMaxElementFinder.findMin(new int[] { 1, 3, 4, 2 }))));
	}

	@TestFactory
	Collection<DynamicTest> dynamicTestsPalindrome() {
		return Arrays.asList(dynamicTest("Palindrome 1", () -> assertTrue(checker.isPalindrome("madam"))),
				dynamicTest("Palindrome 2", () -> assertFalse(checker.isPalindrome("palindrome"))));
	}

	@TestFactory
	Collection<DynamicTest> dynamicTestsMinMaxPalindromeCollection() {
		return Arrays.asList(
				dynamicTest("Find Max Test",
						() -> assertEquals(4, MinMaxElementFinder.findMax(new int[] { 1, 3, 4, 2 }))),
				dynamicTest("Find Min Test",
						() -> assertEquals(1, MinMaxElementFinder.findMin(new int[] { 1, 3, 4, 2 }))),
				dynamicTest("Palindrome 1", () -> assertTrue(checker.isPalindrome("madam"))),
				dynamicTest("Palindrome 2", () -> assertFalse(checker.isPalindrome("palindrome"))));
	}

	@TestFactory
	Iterable<DynamicTest> dynamicTestsMinMaxPalindromeIterable() {
		return Arrays.asList(
				dynamicTest("Find Max Test",
						() -> assertEquals(4, MinMaxElementFinder.findMax(new int[] { 1, 3, 4, 2 }))),
				dynamicTest("Find Min Test",
						() -> assertEquals(1, MinMaxElementFinder.findMin(new int[] { 1, 3, 4, 2 }))),
				dynamicTest("Palindrome 1", () -> assertTrue(checker.isPalindrome("madam"))),
				dynamicTest("Palindrome 2", () -> assertFalse(checker.isPalindrome("palindrome"))));
	}

	@TestFactory
	Iterator<DynamicTest> dynamicTestsMinMaxPalindromeIterator() {
		return Arrays.asList(
				dynamicTest("Find Max Test",
						() -> assertEquals(4, MinMaxElementFinder.findMax(new int[] { 1, 3, 4, 2 }))),
				dynamicTest("Find Min Test",
						() -> assertEquals(1, MinMaxElementFinder.findMin(new int[] { 1, 3, 4, 2 }))),
				dynamicTest("Palindrome 1", () -> assertTrue(checker.isPalindrome("madam"))),
				dynamicTest("Palindrome 2", () -> assertFalse(checker.isPalindrome("palindrome")))).iterator();
	}

	@TestFactory
	DynamicTest[] dynamicTestsMinMaxPalindromeArray() {
		return new DynamicTest[] {
				dynamicTest("Find Max Test",
						() -> assertEquals(4, MinMaxElementFinder.findMax(new int[] { 1, 3, 4, 2 }))),
				dynamicTest("Find Min Test",
						() -> assertEquals(1, MinMaxElementFinder.findMin(new int[] { 1, 3, 4, 2 }))),
				dynamicTest("Palindrome 1", () -> assertTrue(checker.isPalindrome("madam"))),
				dynamicTest("Palindrome 2", () -> assertFalse(checker.isPalindrome("palindrome"))) };
	}

	@TestFactory
	Stream<DynamicTest> dynamicTestsStream() {
		return Stream.of("madam", "mom", "dad")
				.map(str -> dynamicTest(str, () -> assertTrue(checker.isPalindrome(str))));
	}

	@TestFactory
	Stream<DynamicTest> dynamicTestsIntStream() {
		return IntStream.iterate(0, n -> n + 2).limit(5)
				.mapToObj(n -> dynamicTest("Even Test" + n, () -> assertTrue(n % 2 == 0)));
	}

	@TestFactory
	Stream<DynamicTest> dynamicTestsStreamFactoryMethod() {
		Stream<String> inputStream = Stream.of("madam", "mom", "dad");

		Function<String, String> displayNameGenerator = str -> str + " is a palindrome";

		ThrowingConsumer<String> testExecutor = str -> assertTrue(checker.isPalindrome(str));

		return DynamicTest.stream(inputStream, displayNameGenerator, testExecutor);
	}

	@TestFactory
	Stream<DynamicNode> dynamicTestsContainers() {
		return Stream.of("Roytuts", "Soumitra", "Roy").map(input -> dynamicContainer("Container " + input,
				Stream.of(dynamicTest("Not Null", () -> assertNotNull(input)),
						dynamicContainer("Conditional Checks",
								Stream.of(dynamicTest("Empty", () -> assertNull(null)),
										dynamicTest("False", () -> assertFalse(false)),
										dynamicTest("Length Greater Than 0", () -> assertTrue(input.length() > 0)),
										dynamicTest("Not Empty", () -> assertFalse(input.isEmpty())))))));
	}

	@TestFactory
	DynamicNode dynamicNodeSingle() {
		return dynamicTest("'push' is a not palindrome", () -> assertFalse(checker.isPalindrome("push")));
	}

	@TestFactory
	DynamicNode dynamicNodeSingleContainer() {
		return dynamicContainer("Palindromes", Stream.of("madam", "mom", "dad")
				.map(str -> dynamicTest(str, () -> assertTrue(checker.isPalindrome(str)))));
	}

	@TestFactory
	Stream<DynamicTest> checkAllTextFiles() throws Exception {
		return Files.walk(Paths.get("src/test/resources/test"), 1).filter(path -> path.toString().endsWith(".txt"))
				.map(path -> dynamicTest("File: " + path.getFileName(), path.toUri(), () -> checkLineContent(path)));
	}

	private void checkLineContent(Path path) throws Exception {
		List<String> lines = Files.readAllLines(path);

		String expected = lines.get(0);
		String actual = new StringBuilder(lines.get(0)).reverse().toString();

		assertEquals(expected, actual, "String and its reverse should be equal");
	}

}

The whole source code can be downloaded from the Source Code section later in the below.

Testing Dynamic Tests

Running the Junit class will give you the following output:

Source Code

Download