16. April 2015
by Andreas Zitzelsberger | 1124 words | ~6 min read
This article summarizes my experiences when setting up UI tests for a new JavaFX application at QAware.
I had the following requirements for a JavaFX test framework:
Later on, the requirement to do headless tests was added. I will cover headless tests in a follow-up post.
A quick search reveals three contenders:
Of these three, MarvinFX seems dead. The last commit to MarvinFX was on May the 4th, 2013, almost two years ago, so I discarded that option.
There are several further possibilities such as Automaton and TestComplete. Test Complete is a full-blown GUI testing product and not a lean framework, and the Automaton developers recommend TestFX for testing JavaFX).
Before dealing with the test frameworks, let me mention that JavaFX has two important issues that restrict testability. To fix them, both require modifications to the JavaFX platform.
JavaFX allows to start exactly one application exactly once within a JVM. That means:
I’m not aware of any way to get around this restriction.
The Oracle and OpenJDK desktop builds to not allow testing JavaFX applications without a physical display. I will show a way to get around this restriction in a follow-up post.
TestFX promises “Simple and clean testing for JavaFX”. The TestFX Getting Started Guide provides a short primer on how to use TestFX. Currently, TestFX 4 is under active development.
TestFX is available in Maven Central, so integration is just a Maven dependency away.
To create a TestFX GUI test, you have to extend the GuiTest class:
You have to override a single method getRootNode(), which will create the node to test. In TestFX 4, currently in alpha state, the idiom is closer to the regular JavaFX application startup. The method to override has the signature void start(Stage stage). As you can see, TestFX is geared towards testing GUI components of an application. This approach comes with two important caveats:
There is a workaround for caveat #1:
setupStage() method and startup your application manually. The result of the
getRootNode() method is no longer used.
For the tests themselves, there is a nice and easy fluent API:
I like especially that TestFX exposes the lookup by CSS selector functionality of JavaFX. This makes the creation of page objects superfluous in most cases.
TestFX creates screenshots upon test failures. Unfortunately, it also creates screenshots elsewhen, for example when using the
waitUntil() method. Combine that with the fact that the screenshot location is hard-coded to the current working directory, and the screenshot feature becomes annoying instead of useful.
JemmyFX is another JavaFX UI test library and part of the offical OpenJFX project. Since 2012, it resides in the test branch of OpenJFX.
JemmyFX is not provided in binary form. To use JemmyFX, you have to checkout and compile the sources yourself:
The Jemmy web site is in disrepair which raises doubts about the project’s status. However, the test branch is still included in the newest OpenJFX versions and seems to be actively mainained at a low level.
The build produces quite a number of Jars. In total, 7 JARs are required to use JemmyFX:
You can include these JARs in a Maven or Gradle build by installing them in your local repository or on a custom Nexus.
As soon as all that is done, JemmyFX is quite easy to use. You need to create a setup method that initalizes and launches your application:
The test cases are somewhat more verbose than with TestFX:
JemmyFX requires docks to access each UI element. It offers a intuitive but non-fluent API to operate on these docks. JemmyFX does not directly expose the JavaFX lookup by CSS selector functionality. You can work around this restriction by creating the docks using the exposed JavaFX UI controls:
While JavaFX is a powerful and well-engineered GUI framework, it’s testability is severely hampered. Once you get around the fundamental restrictions, both TestFX and JemmyFX provide the means to effectively test JavaFX applications.
At the moment, I’d use JemmyFX for a new project because of it’s cleaner test case setup and the ability to do headless with Monocle. Also, I dislike abstract base classes used to provide tool-like functionality. This recommendation will probably change when TestFX 4 is released.