Run a Test Suite on a Device
In this tutorial, the target device is the Espressif ESP32-WROVER-KIT V4.1 board and the Filesystem Test Suite for FS module will be used as an example.
The tutorial should take 1 hour to complete (excluding the Platform Getting Started setup).
Intended Audience and Scope
The audience for this document is software engineers who want to validate an abstraction layer implementation or understand how to automatically run a MicroEJ Test Suite on their device.
The following topics are out of the scope of this tutorial:
This tutorial assumes the following:
- Good knowledge of the MicroEJ Glossary.
- Tutorial Understand how to build a MicroEJ Firmware and its dependencies has been followed.
- MicroEJ SDK distribution 20.07 or more (see Determine the MicroEJ Studio/SDK Version).
- The WROVER Platform has been properly setup (i.e., it can be used to generate a MicroEJ Mono-Sandbox Firmware).
The explanation can be adapted to run the test suite on any other MicroEJ Platform providing:
- An implementation of LLFS: File System version 1.0.2 in
- A partial or full BSP Connection.
This tutorial can also be adapted to run other test suites in addition to the Filesystem Test Suite presented here.
In essence, a Foundation Library provides an API to be used by a MicroEJ Application or an Add-on Library.
For example, the Java file system API
java.io.File is provided by
the MicroEJ Foundation Library named
FS. The Abstraction Layer of
each Foundation API must be implemented in C in the Board Support
Package. The Test Suite is used to validate the C code implementation
of the Abstraction Layer.
Import the Test Suite
Follow these steps to import the Filesystem Test Suite into the workspace from the Platform Qualification Tools:
- Clone or download the Platform Qualitification Tools project 2.3.0.
- Select File > Import….
- Select Existing Projects into Workspace.
- Set Select the root directory to the directory
tests/fsin the Platform Qualification Tools fetched in the previous step.
- Ensure Copy projects into workspace is checked.
- Click on Finish.
java-testsuite-fs should now be available in the
Configure the Test Suite
Configure the Platform BSP Connection
Several properties must be defined depending on the type of BSP Connection used by the MicroEJ Platform.
For a MicroEJ Application, these properties are set using the launcher
of the application. For a Test Suite, the properties are defined in a
config.properties in the root folder of the Test Suite.
For example, see this config.properties
See BSP Connection for an explanation of the properties. See
the comments in the file for a details description of each properties.
target.platform.dir properties are required.
Configure Execution Trace Redirection
When the Test Suite is executed, the Test Suite Engine must read the trace to determine the result of the execution. To do that, we will use the Serial to Socket Transmitter tool to redirect the execution traces dumped to a COM port.
The WROVER platform used in this tutorial is particular because the UART port is already used to flash the device. Thus, a separate UART port must be used for the trace output.
This platform defines the option
microej.testsuite.properties.debug.traces.uart to redirect traces
from standard input to UART.
See the WROVER Platform documentation for more details.
Start Serial To Socket
The Serial to Socket Transmitter tool can be configured to listen on a
particular COM port and redirect the output on a local socket. The
microej.testsuite.properties.testsuite.trace.port must be
Follow these steps to create a launcher for Serial To Socket Transmitter:
- Select Run > Run Configurations….
- Right-click on MicroEJ Tool > New.
- In the Execution tab:
- Set Name to
Serial To Socket Transmitter.
- Select a MicroEJ Platform available in the workspace in Target > Platform.
Serial To Socket Transmitterin Execution > Settings.
- Set the Output folder to the workspace.
- Set Name to
- In the Configuration tab:
- Set the correct COM port and baudrate for the device in Serial Options.
- Set a valid port number in Server Options >
Port. This port is the same as the one set in
Configure the Test Suite Specific Options
Depending on the Test Suite and the specificities of the device,
various properties may be required and adjusted. See the file
validation/microej-testsuite-common.properties (for example
and the README of the Test Suite for a description of each property.
On the WROVER Platform, the configuration files
microej-testsuite-common.properties are provided in
config.properties, the property
target.platform.dir must be
set to the absolute path to the platform. For example
Run the Test Suite
To run the Test Suite, right click on the Test Suite module and select
Configure the Tests to Run
It is possible to exclude some tests from being executed by the Test Suite Engine.
To speed-up the execution, let’s configure it to run only a small set
of tests. In the following example, only the classes that match
TestFilePermission are executed. This configuration goes into the
config.properties in the folder of the test suite.
# Comma separated list of patterns of files that must be included # test.run.includes.pattern=**/Test*.class test.run.includes.pattern=**/TestFilePermission*.class # Comma separated list of patterns of files that must be excluded (defaults to inner classes) test.run.excludes.pattern=**/*$*.class
Several reasons might explain why to exclude some tests:
- Iterative development. Test only the Abstraction Layer that is currently being developed. The full Test Suite must still be executed to validate the complete implementation.
- Known bugs in the Foundation Library. The latest version of the Test Suite for a given Foundation Library might contain regression tests or tests for new features. If the MicroEJ Platform doesn’t use the latest Foundation Library, then it can be necessary to exclude the new tests.
- Known bugs in the Foundation Library implementation. The project might have specific requirements that prevent a fully compliant implementation of the Foundation Library.
Examine the Test Suite Report
Once the Test Suite is completed, open the HTML Test Suite Report stored in
At the beginning of the file, a summary is displayed. Below, all execution traces for each test executed are available.
If necessary, the binaries produced and ran on the device by the Test
Suite Engine are available in
The following image shows the test suite report fully passed: