How to Validate GUIs

Description

This document explains how to validate Graphical User Interfaces. It describes common pitfalls that can affect GUI performances, provides tools that allow to detect performance issues and how to solve them, and finally offers ways to test GUIs automatically.

Intended Audience

This training is for developers wishing to develop GUI Applications in MICROEJ VEE.

Prerequisites

To get the most out of this training, participants should have a good knowledge of Java programming language.

Implementing GUIs Efficiently

Before using more advanced UI debugging techniques, the global application code quality should be reviewed. An overall good code quality will help to get good UI performances. It will help to get more efficient code and allow easier debugging and maintenance.

Documents and Tools to Improve Application Code Quality

Here is a list of documents or tools that help to improve the quality of application code:

Using Recent Versions of UI Libraries

Using the latest UI libraries (MicroUI, MWT, Widget, etc.) available may solve some performance issues. The most recent UI libraries fix some bugs that may affect performance and they provide tools and libraries that allow to implement more performant UIs.

Memory Management

The Java management of memory may affect UI performances:

  • Too much memory allocation/deallocation for UI resources (Images, Fonts).

  • Too much object instantiation will lead to a big Java heap size. In some use cases, the garbage collection may lead to the UI slowing down.

To avoid those pitfalls:

Format of UI Resources

One crucial aspect of optimizing an application is choosing the right image formats. Images can have a significant impact on an app’s performance and memory usage. Therefore, selecting the best image format is essential. It helps reduce memory usage, speed up the app, and improve its overall performance.

MicroUI manages two kind of images, mutable and immutable images.

Mutable images are graphical resources that can be created and modified at runtime. The application can draw into such images. More information about mutable images can be found here.

As their name suggests, immutable images can not be modified. They are the most commonly used kind of images, this section will focus on them.

Decoding Immutable Images

Immutable images can be converted for display either during the build-time process, using the Image Generator, or at run-time, utilizing the appropriate decoder library.

The decision between these two approaches depends on the project’s specific requirements. Decoding at run-time is a good choice when storage space is limited and offers greater flexibility. However, it may require more processing power and result in slower performance. Conversely, decoding at build time reduces the computational workload during run-time and is well-suited for devices with stringent performance demands, though it usually require more storage and it may sacrifice some flexibility in the process.

Format of Immutable Images

There are multiple output formats that can be used to convert the images, you can find them here: Output Formats.

Choosing the right output format is important to get the best performance:

  • For opaque images, choose a format that has no transparency, RGB565 is usually sufficient.

  • For a pictogram to colorize A4 is usually sufficient. The image can be colorized at runtime.

  • The image format can be compressed, see Compressed Output Formats

The expected result of each format can be seen here: Formats expected result

Images Heap

Mutable images and immutable images decoded at runtime require some memory to be used. Please go to the Images Heap section for more information.

Benchmarking GUIs

The process of rendering a frame of the UI consists of several parts:

  • Drawing of the UI:

    • MWT processing (layout of widgets and widget rendering process).

    • Drawing of the UI (MicroUI drawing method execution).

  • Display flush, see Buffer Policy.

  • Buffer Refresh Strategy (BRS), see Buffer Refresh Strategy.

Some tools are available to identify which part of this process affect the most the GUI performance.

SystemView

The SystemView tool can be used to trace the UI actions (drawings, flush, etc.) and detect which ones are the most time-consuming. The documentation of SystemView is available here. The MicroUI traces should be configured in SystemView in order to see the UI actions performed, it can be done by following this documentation. Custom traces can be added and logged from the Java application to record specific actions.

MicroUI Flush Visualizer

A perfect application has 100% of its display area drawn. This is the total area covered by the sum of the area drawn by the drawing operations. A value of 200% indicates the area drawn is equivalent to twice the surface of the entire display. A total area drawn between 100% to 200% is the norm in practice because widgets often overlap. However, if the total area drawn is bigger than 200%, that means that the total surface of the display was drawn more than twice, meaning that a lot of time could be spent drawing things that are never shown.

The MicroUI Flush Visualizer tool can be used to investigate potential performance bottlenecks in UI applications running on the Simulator by showing the pixel surface drawn between two MicroUI flushes.

The documentation of MicroUI Flush Visualizer is available here.

Debugging GUIs

High-level Debugging and Optimizations

This section provides insights into common issues affecting performances on the high-level side. The following advices will help reduce the MWT processing and drawing time.

Widget Hierarchy and Layout

Keeping the widget hierarchy as simple as possible will help to reduce the “MWT processing” part time. Improving the widget hierarchy design may help reduce the number of widgets or the number of them that are rendered when a certain part of the UI is updated.

Here are tools that allow to detect issues with the widget hierarchy:

  • Widget debug utilities provides tools to visualize the widget tree, count the numbers of widgets or see their bounds.

  • MWT bounds highlighting allows to visualize the bounds of the widgets, it is useful to detect overlapping widgets.

Bad Use of requestRender and requestLayout

The requestRender method requests a render of the widget on the display.

The requestLayout method requests a layout of all the widgets in the sub-hierarchy of this widget. It will compute the size and position of the widgets as setting their styles. requestLayout will trigger a render request after the layout.

A common mistake is to call requestRender just after a requestLayout. This will trigger two renders and thus affect the UI performances.

Another common issue is to request a layout where a render request would have been enough. If the size, position or style of the widgets didn’t change requestRender is enough, requestLayout would have a longer processing time. This is especially true for animation where we want each frame to be processed as fast as possible.

Documentation about rendering and layout is available here.

Animations Implementation

There are a few implementations possible for animations with MicroEJ. The way widgets are animated should be chosen according to the use case and the limitation of the hardware.

Animator

The MWT’s Animator allows to execute animations as fast as possible, it waits for the low-level screen flush to be done and directly triggers a new render. Thus the Animator will give the best framerate possible but will also consume a lot of CPU processing time.

TimerTask

A TimerTask can be used to execute an animation at a fixed framerate. This technique is very useful to set a fixed period for the animation but will cause issues if the time to render a frame is longer than that period, this will lead to missed frames. Some frames can take longer to render if their content is more complex or if the CPU is already used by another non-UI thread.

The framerate set when using a TimerTask for animation should be defined wisely, the time to render frames and the CPU utilization should be taken into consideration.

Animator and TimerTask mix

A mix of the Animator and TimeTask approaches could be implemented in order to set a fixed framerate but also to rely on the screen flush.

Hardware and Low-level Debugging and Optimizations

This section provides insights into the main spots to check regarding the low-level and the hardware.

Please see the VEE Porting Guide Graphical User Interface documentation for more information about the UI port.

At Project Level
Compiling Optimization Options

The project should be configured to bring the best performances with compiling optimization options correctly set up.

RTOS Tasks Environment

The priority of the UI task should be set high enough to avoid too many preemptions that may induce bad UI performances.

Another point that should be taken into consideration is the amount of other tasks that are running at the same time as the UI task. The total workload may be too high for the CPU, therefore, the UI task cannot get access to the required amount of computing power.

At Hardware Level
Hardware Capabilities

MCUs and SoCs may have access to various hardware IPs to speed up the UI. The UI port should exploit all of them to get the best performance. First of all, the GPU should be used if it is available on the system. Then, driving a display implies intensive memory usage, a DMA should be used whenever it’s possible.

For more information about the flush policy, please read our documentation about Display.

Hardware Configuration

Each of the hardware components such as SPI, DMA or LCD controller must be configured to bring the best performances achievable. This implies to read carefully the datasheet of the MCU and the display and determine for example the best frequency and communication mode possible.

Another example of configuration with DMAs, a DMA has often a burst mode to transfer data, the UI port should use this mode to maximize performance.

Buffers Location in Memory

An important step during the development of the UI integration is the memory location of the buffers that will use the GUI to draw to the display. In an MCU, there may be different types of RAM available that have different properties in terms of quantity and speed. The fastest RAM should be chosen for the buffers if its size allows it.

Flush Policy

As described in the Display page, there are several flush policies that can be implemented. The best flush policy should be selected according to the hardware capabilities. Generally, the best flush policy is the switch mode.

Testing GUIs

Before applying UI debugging or optimization techniques, the application behaviour should be tested. There are different ways of doing this.

Test a GUI Application with a Software Robot

It is possible to test the GUI of an application via robotic process automation (RPA). Robot tests mimic the human user behavior in the GUI and can help detect various errors by automating behavior which otherwise would cost too much effort and/or time to execute manually.

Here are the steps required to use a robot in the MicroEJ environment:

  • Record the robot input events

    • For this, you need a simple EventHandler which intercepts incoming events, for example from a Pointer, then passes them on to the real event handler.

  • Start the usage of the new ‘Watcher’ logic after the UI has started

    • With this, the watching of the Pointer envents is initiated for the whole application.

  • Create a Robot

    • The robot is a simple class which uses its own Pointer to move and press at the coordinates it has been instructed.

    • The robot should have a method which starts a series of instructions to move the Pointer.

  • Execute the Robot method containing the instructions

    • The intercepting Event Handling will record and for example log the input.

This simple way of automating GUI actions can be used to carry out real use cases and evaluate the results.

The How to test a GUI application with a (software) robot training provides detailed insight into this topic.

Test a GUI Application with the Test Automation Tool

To execute regression tests automatically and monitor minor changes in a GUI, you can use the Test Automation Tool. The Test Automation Tool allows to automatically test UIs.

The tool comparison functionality can be integrated with JUnit tests.

For detailed information about the tool usage, please check the README in the repository.