Simulation

Principle

The graphical user interface uses the Front Panel mock (see Front Panel Mock) and some extensions (widgets) to simulate the user interactions. It is the equivalent of the three embedded modules (Display, Input and LED) of the MicroEJ Platform (see MicroUI).

The Front Panel enhances the development environment by allowing User Interface  applications to be designed and tested on the computer rather than on the target device (which may not yet be built). The mock interacts with the user’s computer in two ways:

• output: LEDs, graphical displays
• input: buttons, joystick, touch, haptic sensors

Note

This chapter completes the notions described in Front Panel Mock chapter.

Module Dependencies

The Front Panel project is a regular MicroEJ Module project. Its module.ivy file should look like this example:

<ivy-module version="2.0" xmlns:ea="http://www.easyant.org" xmlns:ej="https://developer.microej.com" ej:version="2.0.0">
<info organisation="com.mycompany" module="examplePanel" status="integration" revision="1.0.0"/>

   <configurations defaultconfmapping="default->default;provided->provided">
      <conf name="default" visibility="public" description="Runtime dependencies to other artifacts"/>
      <conf name="provided" visibility="public" description="Compile-time dependencies to APIs provided by the platform"/>
   </configurations>

   <dependencies>
      <dependency org="ej.tool.frontpanel" name="widget" rev="1.0.0"/>
   </dependencies>
</ivy-module>

It depends at least on the Front Panel framework. This framework contains the Front Panel core classes. The dependencies can be reduced to:

<dependencies>
   <dependency org="ej.tool.frontpanel" name="framework" rev="1.1.0"/>
</dependencies>

To be compatible with Display module’s Graphics Engine, the project must depend on an extension of Front Panel framework. This extension provides some interfaces and classes the Front Panel is using to target simulated display and input devices. The extension does not provide any widgets. It is the equivalent of the embedded Low Level API. It fetches by transitivity the Front Panel framework, so the Front Panel framework dependency does not need to be specified explicitly:

<dependencies>
   <dependency org="com.microej.pack.ui" name="ui-pack" rev="[UI Pack version]">
      <artifact name="frontpanel" type="jar"/>
   </dependency>
</dependencies>

Warning

This extension is built for each UI pack version. By consequence, a Front Panel project is done for a platform built with the same UI pack. When the UI pack mismatch, some errors may occur during the Front Panel project export step, during the platform build, and/or during the application runtime. The current pack version is 13.4.1.

The Front Panel extension does not provide any widgets. Some compatible widgets are available in a third library. The life cycle of this library is different than the UI pack’s one. New widgets can be added to simulate new kind of displays, input devices, etc. This extension fetches by transitivity the Front Panel extension, so this extension dependency does not need to be specified explicitly:

<dependencies>
   <dependency org="ej.tool.frontpanel" name="widget" rev="2.0.0"/>
</dependencies>

Warning

The minimal version 2.0.0 is required to be compatible with UI pack 13.0.0 and higher. By default, when creating a new Front Panel project, the widget dependency version is 1.0.0.

Source code for widgets is available by expanding the library from the project view.

Front Panel Widgets

MicroUI Implementation

As described here, the Front Panel uses an equivalent of embedded side’s header files that implement MicroUI native methods.

This set of classes and interfaces is available in the module com.microej.pack.ui#ui-pack. It offers the same capacity to override some built-in drawing algorithms (internal Graphics Engine drawing algorithms), to add some custom drawing algorithms, to manipulate the MicroUI concepts (GraphicsContext, Image, etc.) in the Front Panel project, etc.

• The interface ej.microui.display.LLUIDisplay represents the MicroUI Graphics Engine (MicroUI framework). It provides some methods to map MicroUI byte arrays in MicroUI Graphics Context objects, manipulate MicroUI colors, etc. An instance of this framework is available via the field Instance.
• The interface ej.microui.display.LLUIDisplayImpl all methods required by MicroUI implementation to be compatible with the MicroUI Display class implementation. See Widget Display.
• The class ej.microui.display.LLUIPainter implements all MicroUI drawing natives. It defines some interfaces and classes to manipulate the MicroUI concepts (GraphicsContext, Image, etc.) in the Front Panel project. Like the embedded side, this class manages the synchronization with the Graphics Engine and delegates the drawing to the interface ej.microui.display.UIDrawing.
• Like the embedded side, the default implementation of the interface ej.microui.display.UIDrawing: ej.microui.display.UIDrawing.UIDrawingDefault calls the internal Graphics Engine algorithms (software algorithms). Each algorithm can be overridden independently.
• The classes in the package ej.drawing implement the native of the MicroUI extended library: Drawing
• The classes in the package ej.microui.event manage the input events, see Inputs Extensions.
• The classes in the package ej.microui.led manage the LEDs.

Widget Display

The widget Display implements the interface ej.microui.display.LLUIDisplayImpl to be compatible with the implementation of the MicroUI class Display.

Features

• Simple or double buffering (default value): doubleBufferFeature=true|false.
• LCD refresh rate: simulates the time between two visible frames on the hardware device.
• LCD flush time: simulates the time to send the frame content to the hardware device.
• Backlight (enabled by default): backlightFeature=true|false.
• Non-rectangular displays: filter="xxx.png". Some displays can have another appearance (for instance: circular).
• Standard pixel formats.
• Driver-specific pixel formats: extensionClass="xxx". This class must be added in the Front Panel project and implement the interface ej.fp.widget.Display.DisplayExtension.

Refresh Rate

Usually a LCD is cadenced around 50-60Hz. That means the LCD can display a new frame every 16-20ms. By default this widget displays a new frame as soon as possible. It can be configured to reduce this time to simulate the hardware device.

In the widget declaration, set the attribute refreshRate="xxx" with a value in Hertz. A zero or negative value disables the feature.

Note

This feature is only available when double buffering mode is enabled.

The application can substitute the platform’s value by setting the property -Dej.fp.widget.display.refreshRate=xxx in the application launcher.

Flush Time

On a hardware device, the time to send the frame data from the back buffer memory to the LCD is not null. According to the hardware device technology, this time varies between 3-4 ms to 10-15ms. In SPI mode, this time may be higher, around 50ms, even more. By default this widget copies the content of back buffer as faster as possible. It can be configured to reduce this time to simulate the hardware device.

In the widget declaration, set the attribute flushTime="xxx" with a value in milliseconds. A zero or negative value disables the feature.

Note

This feature is only available when double buffering mode is enabled.

The application can substitute the platform’s value by setting the property -Dej.fp.widget.display.flushTime=xxx in the application launcher.

Non-rectangular Display

The Front Panel can simulate using a filter (see Widget). This filter defines the pixels inside and outside the whole display area. The filter image must have the same size as the rectangular display area. A display pixel at a given position will not be rendered if the pixel at the same position in the mask is fully transparent.

Note

Usually the touch panel over the display uses the same filter to reduce the touch panel area.

Example of non-rectangular display and touch:

<ej.fp.widget.Display x="41" y="33" width="392" height="392" filter="mask_392.png" />
<ej.fp.widget.Pointer x="41" y="33" width="392" height="392" filter="mask_392.png" touch="true"/>

Custom Drawings

Custom algorithms like the embedded side can replace the Graphics Engine drawing algorithms. This chapter describes how to override the method fillRectangle().

1. Create an implementation of the interface ej.microui.display.UIDrawing.UiDrawingDefault: MyDrawer.
2. Create an empty widget to invoke the new implementation:

@WidgetDescription(attributes = { })
public class Init extends Widget{
   @Override
   public void start() {
      super.start();
      LLUIPainter.setDrawer(new MyDrawer());
   }
}

3. Invokes this widget in the .fp file:

<frontpanel xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="https://developer.microej.com"  xsi:schemaLocation="https://developer.microej.com .widget.xsd">
   <device name="STM32429IEVAL" skin="Board-480-272.png">
      <com.is2t.microej.fp.Init/>
      [...]
   </device>
</frontpanel>

4. Implements the fillRectangle method like this:

@Override
public void fillRectangle(MicroUIGraphicsContext gc, int x1, int y1, int x2, int y2) {

   // manage the clip
   if (gc.isClipEnabled() && !gc.isRectangleInClip(x1, y1, x2, y2)) {
      x1 = Math.max(x1, gc.getClipX1());
      x2 = Math.min(x2, gc.getClipX2());
      y1 = Math.max(y1, gc.getClipY1());
      y2 = Math.min(y2, gc.getClipY2());

      if (x2 < x1 || y2 < y1) {
         // nothing to render
         return;
      }
   }
   // clip disabled, or rectangle entirely fits the clip

   // get/create AWT objects
   Graphics2D src = (Graphics2D) ((BufferedImage) gc.getImage().getRAWImage()).getGraphics();
   Color color = new Color(gc.getRenderingColor());

   // fill rectangle
   src.setColor(color);
   src.fillRect(x1, y1, x2 - x1 + 1, y2 - y1 + 1);

   // update Graphics Engine's dirty area
   gc.setDrawingLimits(x1, y1, x2, y2);
}

Explanations:

• The Front Panel framework is running over AWT.
• The method gc.getImage() returns a ej.fp.Image. It is the representation of a MicroUI Image in the Front Panel framework. The method gc.getImage().getRAWImage() returns the implementation of the Front Panel image on the J2SE framework: a AWT BufferedImage. From this image, retrieve the AWT graphics.
• The MicroUI color (gc.getRenderingColor()) is converted in an AWT color.
• Before drawing, the MicroUI clip is checked (the MicroUI drawing can be outside the clip and/or the image itself).
• After drawing, the implementation updates the Graphics Engine dirty area by calling gc.setDrawingLimits().

Note

More details are available in LLUIPainter, UIDrawing, LLUIDisplay, and LLUIDisplayImpl classes.

New Drawings

The Front Panel framework allows the addition of custom drawing algorithms. This chapter describes how to implement the method drawSomething().

Note

To add a custom drawing algorithm, it is strongly recommended to follow the same rules as the MicroUI drawings: a class that synchronizes the drawings with the Graphics Engine and another class that performs the drawing itself (like Custom Drawings).

1. In the application, write the new native:

private static void drawSomething(byte[] gc, int x, int y);

2. Call this function from a render method:

@Override
public void render(GraphicsContext gc) {
   [...]
   drawSomething(gc.getSNIContext(), 15, 36);
   [...]
}

3. In the Front Panel project, write the equivalent of LLUIPainter class:

public static void drawSomething(byte[] target, int x, int y) {
   LLUIDisplay graphicalEngine = LLUIDisplay.Instance;
   synchronized (graphicalEngine) {
      MicroUIGraphicsContext gc = graphicalEngine.mapMicroUIGraphicsContext(target);
      if (gc.requestDrawing()) {
         MyDrawer.Instance.drawSomething(gc, x, y);
      }
   }
}

4. In the Front Panel project, write the equivalent of UIDrawing interface, see Custom Drawings:

public static void drawSomething(MicroUIGraphicsContext gc, int x, int y) {
   // 1- manage clip
   // 2- draw
   // 3- update Graphics Engine's dirty area
}

Note

More details are available in LLUIPainter, UIDrawing, LLUIDisplay, and LLUIDisplayImpl classes.

Inputs Extensions

The input device widgets (button, joystick, touch, etc.) require a listener to know how to react on input events (press, release, move, etc.). The aim of this listener is to generate an event compatible with MicroUI Event Generator. Thereby, a button press action can become a MicroUI Buttons press event or a Command event or anything else.

A MicroUI Event Generator is known by its name. This name is fixed during the MicroUI static initialization (see Static Initialization). To generate an event to a specific event generator, the widget has to use the event generator name as identifier.

A Front Panel widget can:

• Force the behavior of an input action: the associated MicroUI Event Generator type is hardcoded (Buttons, Pointer, etc.), the event is hardcoded (for instance: widget button press action may be hardcoded on event generator Buttons and on the event pressed). Only the event generator name (identifier) should be editable by the Front Panel extension project.
• Propose a default behavior of an input action: contrary to first point, the Front Panel extension project is able to change the default behavior. For instance a joystick can simulate a MicroUI Pointer.
• Do nothing: the widget requires the Front Panel extension project to give a listener. This listener will receive all widgets action (press, release, etc.) and will have to react on it. The action should be converted on a MicroUI Event Generator event or might be dropped.

This choice of behavior is widget dependant. Please refer to the widget documentation to have more information about the chosen behavior.

Heap Simulation

Graphics Engine is using two dedicated heaps: for the images (see Memory ) and the external fonts (see External Resources). Front Panel partly simulates the heaps usage.

• Images heap: Front Panel simulates the heap usage when the application is creating a BufferedImage, when it loads and decodes an image (PNG, BMP, etc.) which is not a raw resource and when it converts an image in MicroEJ format in another MicroEJ format. However it does not simulate the external image copy in heap (see External Resource).
• External fonts heap: Front Panel does not simulate this heap (see External Resources). There is no rendering limitation when application is using a font which is located outside CPU addresses ranges.

Image Decoders

Front Panel uses its own internal image decoders when the associated modules have been selected (see internal image decoders). Some additional decoders can be added like the C-side for the embedded platform (see external image decoders). Front Panel uses the Java AWT ImageIO API to load the encoded images.

Generic Image Decoders

The Java AWT ImageIO class holds a limited list of additional decoders. To be compliant with the embedded side, these decoders are disabled by default. To add an additional decoder, specify the property hardwareImageDecoders.list in Front Panel configuration properties file (see Installation) with one or several property values:

Front Panel Additional Image Decoders

Type	Property value
Graphics Interchange Format (GIF)	gif
Joint Photographic Experts Group (JPEG)	jpeg or jpg
Portable Network Graphics (PNG)	png
Windows bitmap (BMP)	bmp

The decoders list is comma (,) separated. Example:

hardwareImageDecoders.list=jpg,bmp

Custom Image Decoders

Additionally, the Java AWT ImageIO class offers the possibility to add some custom image decoders by using the service javax.imageio.spi.ImageReaderSpi.

Since UI Pack 13.2.0, Front Panel automatically includes new image decoders (new ImageIO services, see the method LLUIDisplayImpl.decode()), compiled in JAR files that follow this convention:

1. The JAR contains the service declaration /META-INF/services/javax.imageio.spi.ImageReaderSpi,
2. The JAR filename’s prefix is imageio-,
3. The JAR location is the platform configuration project’s dropins/tools/ directory.

Note

The same JAR is used by the Front Panel and by the Image Generator.

Classpath

A standard mock is running on the same JVM than the HIL Engine (see Mock chapter). It shares the same classpath. When the application is not using the MicroUI library (i.e., it is not an UI application, whether the platform holds the MicroEJ Graphics Engine or not), the Front Panel mock runs a standard mock. When the application is using the MicroUI library, the Front Panel _UI_ mock runs on the same JVM than the MicroEJ Simulator. In this case, the other mocks don’t share the same classpath than the Front Panel mock. As a consequence, an other mock than the Front Panel mock can not send input events to MicroUI, the object created in the standard mocks’s class loader are not available in the Front Panel _UI_’s class loader (and vice versa), etc.

Since the UI Pack 13.2.0, it is possible to force to run the Front Panel _UI_ mock in the same classpath than the HIL Engine by adding the property -Dej.fp.hil=true in the application JRE tab. Note that this option only works when the version of the MicroEJ Architecture used to build the Platform is 7.17.0 or higher.

Dependencies

• MicroUI module (see MicroUI),
• Display module (see Display): This module gives the characteristics of the graphical display that are useful for configuring the Front Panel.

Installation

Front Panel is an additional module for MicroUI library. When the MicroUI module is installed, install this module in order to be able to simulate UI drawings on the Simulator. See Installation to install the module.

The properties file can additional properties:

• hardwareImageDecoders.list [optional, default value is “” (empty)]: Defines the available list of additional image decoders provided by the hardware (see Image Decoders). Use comma (‘,’) to specify several decoders among this list: bmp, jpg, jpeg, gif, png. If empty or unspecified, no image decoder is added.

Use

Launch a MicroUI application on the Simulator to run the Front Panel.