Run a Test Suite on a Device

This tutorial describes all the steps to configure and run a Platform Test Suite on a device using the Platform Qualification Tools.

In this tutorial, the target device is the Espressif ESP32-WROVER-KIT V4.1 board and the Filesystem Test Suite for FS module will be used as an example.

The tutorial should take 1 hour to complete (excluding the Platform Getting Started setup).

Intended Audience and Scope

The audience for this document is software engineers who want to validate an Abstraction Layer implementation or understand how to automatically run a MicroEJ Test Suite on their device.

The following topics are out of the scope of this tutorial:

Prerequisites

This tutorial assumes the following:

The explanation can be adapted to run the test suite on any other MicroEJ Platform providing:

Note

This tutorial can also be adapted to run other test suites in addition to the Filesystem Test Suite presented here.

Introduction

This tutorial presents a local setup of the Platform Test Suite for the FS Foundation Library on a concrete device (not on Simulator).

In essence, a Foundation Library provides an API to be used by an Application or an Add-On Library.

MicroEJ Foundation Libraries and Add-On Libraries

MicroEJ Foundation Libraries, Add-On Libraries and MicroEJ Application

For example, the Java file system API java.io.File is provided by the MicroEJ Foundation Library named FS. The Abstraction Layer of each Foundation API must be implemented in C in the Board Support Package. The Test Suite is used to validate the C code implementation of the Abstraction Layer.

Import the Test Suite

Follow these steps to import the Filesystem Test Suite into the workspace from the Platform Qualification Tools:

  • Clone or download the Platform Qualitification Tools project 2.3.0.
  • Select File > Import….
  • Select Existing Projects into Workspace.
  • Set Select the root directory to the directory tests/fs in the Platform Qualification Tools fetched in the previous step.
  • Ensure Copy projects into workspace is checked.
  • Click on Finish.

The project java-testsuite-fs should now be available in the workspace.

Configure the Test Suite

Select the Test Suite Version

For a given Foundation Library version, a specific Test Suite version should be used to validate the Abstraction Layer implementation. Please refer to Test Suite Versioning to determine the correct Test Suite version to use.

On the WROVER Platform, the FS Test Suite version to use is specified in {PLATFORM}-configuration/testsuites/fs/README.md. The Test Suite version must be set in the module.ivy of the java-testsuite-fs project (e.g. java-testsuite-fs/module.ivy). For example:

<dependency org="com.microej.pack.fs" name="fs-testsuite" rev="3.0.3"/>

Configure the Platform BSP Connection

Several properties must be defined depending on the type of BSP Connection used by the MicroEJ Platform.

For a MicroEJ Application, these properties are set using the launcher of the application. For a Test Suite, the properties are defined in a file named config.properties in the root folder of the Test Suite. For example, see this example of config.properties file.

See BSP Connection for an explanation of the properties. See the comments in the file for a details description of each properties. The microej.testsuite.properties.deploy.* and target.platform.dir properties are required.

Configure Execution Trace Redirection

When the Test Suite is executed, the Test Suite Engine must read the trace to determine the result of the execution. To do that, we will use the Serial to Socket Transmitter tool to redirect the execution traces dumped to a COM port.

The WROVER platform used in this tutorial is particular because the UART port is already used to flash the device. Thus, a separate UART port must be used for the trace output.

This platform defines the option microej.testsuite.properties.debug.traces.uart to redirect traces from standard input to UART.

Plug alternative UART on WROVER GND/D0

See the Testsuite Configuration section of the WROVER Platform documentation for more details.

Start Serial To Socket

The Serial to Socket Transmitter tool can be configured to listen on a particular COM port and redirect the output on a local socket. The properties microej.testsuite.properties.testsuite.trace.ip and microej.testsuite.properties.testsuite.trace.port must be configured.

Follow these steps to create a launcher for Serial To Socket Transmitter:

  • Select Run > Run Configurations….
  • Right-click on MicroEJ Tool > New.
  • In the Execution tab:
    • Set Name to Serial To Socket Transmitter.
    • Select a MicroEJ Platform available in the workspace in Target > Platform.
    • Select Serial To Socket Transmitter in Execution > Settings.
    • Set the Output folder to the workspace.
  • In the Configuration tab:
    • Set the correct COM port and baudrate for the device in Serial Options.
    • Set a valid port number in Server Options > Port. This port is the same as the one set in config.properties as microej.testsuite.properties.testsuite.trace.port.

Configure the Test Suite Specific Options

Depending on the Test Suite and the specificities of the device, various properties may be required and adjusted. See the file validation/microej-testsuite-common.properties (for example https://github.com/MicroEJ/PlatformQualificationTools/blob/2.3.0/tests/fs/java/java-testsuite-fs/validation/microej-testsuite-common.properties) and the README of the Test Suite for a description of each property.

On the WROVER Platform, the configuration files config.properties and microej-testsuite-common.properties are provided in {PLATFORM}-configuration/testsuites/fs/.

In config.properties, the property target.platform.dir must be set to the absolute path to the platform. For example C:/P0065_ESP32-WROVER-Platform/ESP32-WROVER-Xtensa-FreeRTOS-platform/source.

Run the Test Suite

To run the Test Suite, right click on the Test Suite module and select Build Module.

Configure the Tests to Run

It is possible to exclude some tests from being executed by the Test Suite Engine.

To speed-up the execution, let’s configure it to run only a small set of tests. In the following example, only the classes that match TestFilePermission are executed. This configuration goes into the file config.properties in the folder of the test suite.

# Comma separated list of patterns of files that must be included
# test.run.includes.pattern=**/Test*.class
test.run.includes.pattern=**/TestFilePermission*.class
# Comma separated list of patterns of files that must be excluded (defaults to inner classes)
test.run.excludes.pattern=**/*$*.class

Several reasons might explain why to exclude some tests:

  • Iterative development. Test only the Abstraction Layer that is currently being developed. The full Test Suite must still be executed to validate the complete implementation.
  • Known bugs in the Foundation Library. The latest version of the Test Suite for a given Foundation Library might contain regression tests or tests for new features. If the MicroEJ Platform doesn’t use the latest Foundation Library, then it can be necessary to exclude the new tests.
  • Known bugs in the Foundation Library implementation. The project might have specific requirements that prevent a fully compliant implementation of the Foundation Library.

Examine the Test Suite Report

Once the Test Suite is completed, open the HTML Test Suite Report stored in java-testsuite-fs/target~/test/html/test/junit-noframes.html.

At the beginning of the file, a summary is displayed. Below, all execution traces for each test executed are available.

If necessary, the binaries produced and ran on the device by the Test Suite Engine are available in target~/test/xml/<TIMESTAMP>/bin/<FULLY-QUALIFIED-CLASSNAME>/application.out.

The following image shows the test suite report fully passed:

../_images/tuto_testsuite_fs_all_tests_passed.PNG