Native Language Support

Introduction

Native Language Support (NLS) allows the application to facilitate internationalization. It provides support to manipulate messages and translate them in different languages. Each message to be internationalized is referenced by a key, which can be used in the application code instead of using the message directly.

Principle

NLS is distributed as an add-on library containing a single Java interface: NLS.

In addition to that, the binary-nls library provides a factory for implementations of this interface: it uses an add-on processor which processes, offboard, the Localization Source Files into one BON resource buffer file for compactness.

During the clinit phase, this resource file is opened and the list of locales is parsed. After that, the resource remains opened for the rest of the Application execution and is directly used to retrieve messages translations for the supported locales.

Usage of this binary-nls implementation is documented below (see current limitations).

Localization Source Files

Messages must be defined in localization source files, located in the Classpath of the application (i.e. in the src/main/resources folder).

Localization source files can be either PO files or Android String resources.

Here is an example of a PO file:

msgid "Label1"
msgstr "My label 1"

msgid "Label2"
msgstr "My label 2"

And here is an example of an Android String resource:

<resources>
   <string name="Label1">My label 1</string>
   <string name="Label2">My label 2</string>
</resources>

Note

When using Android String resources, string arrays are also supported. However, plurals are not supported.

NLS List Files

Localization source files are declared in Classpath *.nls.list files (and to *.externresources.list for an external resource, see Application Resources and Loading Translations as an External Resource).

The file format is a standard Java properties file, each line represents the Full Qualified Name of a Java interface that will be generated and used in the application. Example:

com.mycompany.myapp.Labels
com.mycompany.myapp.Messages

Usage

The binary-nls module must be added to the module.ivy of the Application project.

<dependency org="com.microej.library.runtime" name="binary-nls" rev="2.5.0"/>

This module includes an Add-On Processor which parses the localization source files. For each interface declared in the NLS list files, all the localization source files whose names start with the interface name are used to generate:

• a Java interface with the given FQN, containing a field for each message of the localization source files
• a NLS binary file containing the translations

So, in the example, the generated interface com.mycompany.myapp.Labels will gather all the translations from files named Labels* and located in any package of the Classpath. The names of the localization source files should be suffixed by their locale (for example Labels_en_US.po).

The generation is triggered when building the application or after a change done in any localization source file or *.nls.list files. This allows to always have the Java interfaces up-to-date with the translations and to use them immediately.

Besides the message fields, the generated interface declares an NLS instance which is automatically created in the clinit of the interface.

Once the generation is done, the application can use the Java interfaces to get internationalized messages, for example:

String label = Labels.NLS.getMessage(Labels.Label1);

For the application to know which language to use among those made available and when, you can set it and change it at any point using the setCurrentLocale(locale) method. If no locale has been set yet when getting a message, the translation for the first locale available in alphabetical order will be used by default. However, you can also pick this locale to default to yourself, by adding a com.microej.binarynls.defaultLocale property followed by a locale name in a .properties.list file.

Plural Forms

Starting with version 4.0.0 of the NLS module and version 3.0.0 of the binary-nls module is introduced support of GNU gettext’s plural form feature in PO files. This allows usage of Plural-Forms header entries and several msgstr ‘s per msgid (referred to as plural forms) as specified by gettext; you can then retrieve the correct message in a locale for a given count of things by using the ej.nls.NLS.getMessage() methods that take in this count value as an argument.

If a message for a given msgid has a msgid_plural and plural forms in a PO file for an interface declared in an NLS list file, it must also have plural forms in all other PO files for this interface.

Warning

Please note that one significant difference with gettext’s implementation is that the expression described in the plural field of the Plural-Forms header must be a valid Java expression returning an int, as opposed to a C expression. A usual case in which this makes a difference is for expressions that rely on boolean values being evaluated as zero or one in C, such as in:

"Plural-Forms: nplurals=2; plural=n != 1;\n"

This expression will not work with our implementation as Java does not interpret booleans as integers. An easy way to convert this expression would be:

"Plural-Forms: nplurals=2; plural=n != 1 ? 1 : 0;\n"

Also note that the validity of these provided expressions is not entirely checked. Providing an expression that is not valid Java or that would return an invalid plural form index would cause errors at runtime or even in the Java files generated by the Add-On Processor.

Dealing With Missing Translations

By default, if a translation is missing for a given msgid in a PO file in a given language, the message returned by the ej.nls.NLS.getMessage() method with the locale set to this language will simply be the msgid itself. In the case of an XML Android String resource, the name attribute of a missing string element will be returned. However if returning this identifier is not a suitable solution, you might want to set a fallback locale parameter for an interface. This parameter corresponds to a language to print the translation for a message in, in case it is not available in the current language.

Starting with version 2.5.0 of the binary-nls module, you can set this fallback locale by specifying a locale name in a .nls.list file, after the name of the interface you want this locale to be the fallback for, separated by a colon :. For example, with the following .nls.list file, if a translation is missing in a language for a message in the Labels and Messages PO/XML files, the message will be translated to en_US instead of just returning its msgid/name.

# Missing translations for Labels and Messages will fall back to en_US
com.mycompany.myapp.Labels:en_US
com.mycompany.myapp.Messages:en_US

As such, you can specify a different fallback locale for each interface in a .nls.list file. For example, with the following .nls.list file, the messages in Labels will not have a fallback language set and will only return the msgid/name if a translation is missing, while missing translations will default to en_US for the messages in Messages, and to ja_JP for the messages in Content :

# Missing translations for Labels will fall back to their msgid/name
com.mycompany.myapp.Labels

# Missing translations for Messages will fall back to en_US
com.mycompany.myapp.Messages:en_US

# Missing translations for Content will fall back to ja_JP
com.mycompany.myapp.Content:ja_JP

In the case of a message with plural forms in PO files, this works much the same way, using the messages and forms in the fallback locale if available. If no fallback locale is specified or if the requested message is not specified in it, then the msgid will be used for a count value of 1, and the msgid_plural will be used for any other value, as gettext would function.

BinaryNLS Resource Generation

If the classpath of the Application contains .po/.xml files and .nls.list files, the binary-nls Add-On Processor will generate the following source files for each NLS interface:

• a .resourcebuffer
• a .resourcebuffer.list which references the .resourcebuffer
• a .resources.list which references the resource (this resource does not exist yet but it will be generated later)

When building the Application or running it on Simulator, the Resource Buffer Generator is first executed. Based on the .resourcebuffer and the .resourcebuffer.list, it will generate a resource.

Since the generated resource is referenced by the .resources.list generated by the binary-nls ADP, the SOAR will embed the resource in the Application binary. Unless it is also referenced by an .externresources.list in which case the SOAR will output the resource in the External Resources Folder instead.

This resource is loaded as soon as the BinaryNLS instance is created, in the clinit of the generated NLS interface (see Principle).

Limitations

The latest BinaryNLS implementation does not support:

• to dynamicly add a new locale
• to dynamicly modify messages translations

even when the resource is external (see External resource loader).

For any addition / modification, the Application must be restarted and, typically, the full resource buffer must be updated (not only the part of the added/modified locale).

Also, there is no API to close the resource buffer. If it is external, the Application must be stopped to close this resource, before it can potentially be modified depending on the external resource loader.

NLS External Loader Tool

The NLS External Loader tool allows to update the PO files of an application executed on a Virtual Device without rebuilding it. PO files can be dropped in a given location in the Virtual Device folders to dynamically replace the language strings packaged in the application.

This is typically useful when testing or translating an application in order to have a quick feedback when changing the PO files. Once the PO files are updated, a simple restart of the Virtual Device allows to immediately see the result.

Installation

To enable the NLS External Loader in the Virtual Device, add the following dependency to the module.ivy file of the Firmware project:

<dependency org="com.microej.tool" name="nls-po-external-loader" rev="2.3.0" transitive="false"/>

Then rebuild the Firmware project to produce the Virtual Device.

Usage

Once the project built:

• unzip the Virtual Device and create a folder named translations in the root folder.

• copy all the PO files from the project into the translations folder. All PO files found in this folder are processed, no matter their folder level.

• start the Virtual Device with the launcher. The following logs should be printed if the NLS External Loader has been executed and has found the PO files:

  externalPoLoaderInit:init:
  
  externalPoLoaderInit:loadPo:
     [mkdir] Created dir: <PATH>\tmp\microejlaunch1307817858\resourcebuffer
  [po-to-nls] *.nls files found in <PATH>\output\<FIRMWARE>\resourceBuffer :
  [po-to-nls]   - com.mycompany.Messages1
  [po-to-nls]   - com.mycompany.Messages2
  [po-to-nls] Loading *.po files for NLS interface com.mycompany.Messages1
  [po-to-nls]   => loaded locales : fr_FR,de_DE,ja_JP,en_US
  [po-to-nls] Loading *.po files for NLS interface com.mycompany.Messages2
  [po-to-nls]   => loaded locales : fr_FR,de_DE,ja_JP,en_US
  

• update the languages strings in the PO files of the Virtual Device (the files in the translations/ folder).

• restart the Virtual Device and check the changes.

It is important to know the following rules about the NLS External Loader:

• the external PO files names must match with the default PO files names of the application to be processed.
• when PO files with a given name are loaded, the default translations for these PO files are replaced, there is no merge. It means that:
  • if messages are missing in the new PO files, they are not available anymore for the application and may very probably make it crash.
  • if languages are missing (the application has 3 PO files for English, French and Spanish, and only PO files for English and French are available in the translations folder), the messages of the missing languages are not available anymore for the application and may very probably make it crash.
  • if new messages are added in the PO files, it has no impact, they are ignored by the application.
• External PO files are loaded at Virtual Device startup, so any change requires a restart of the Virtual Device to be considered

Troubleshooting

java.io.IOException: NLS-PO:S=4

The following error occurs when at least 1 PO file is missing for a language:

[parallel2] NLS-PO:I=6
[parallel2] Exception in thread "main" java.io.IOException: NLS-PO:S=4 323463627 -1948548092
[parallel2]     at java.lang.Throwable.fillInStackTrace(Throwable.java:79)
[parallel2]     at java.lang.Throwable.<init>(Throwable.java:30)
[parallel2]     at java.lang.Exception.<init>(Exception.java:10)
[parallel2]     at java.io.IOException.<init>(IOException.java:16)
[parallel2]     at com.microej.nls.BinaryNLS.loadBinFile(BinaryNLS.java:310)
[parallel2]     at com.microej.nls.BinaryNLS.<init>(BinaryNLS.java:157)
[parallel2]     at com.microej.nls.BinaryNLS.newBinaryNLS(BinaryNLS.java:118)

Make sure that all PO files are copied in the translations folder.

Crowdin

Crowdin is a cloud-based localization platform which allows to manage multilingual content. The NLS External Loader can fetch translations directly from Crowdin to make the translation process even easier. Translators can then contribute and validate their translations in Crowdin and apply them automatically in the Virtual Device.

A new dependency must be added to the module.ivy file of the Firmware project to enable this integration:

<dependency org="com.microej.tool" name="nls-po-crowdin" rev="1.0.0" transitive="false"/>

Once the module has been built, edit the file platform/tools/crowdin/crowdin.properties to configure the Crowdin connection:

• set crowdin.token to the Crowdin API token. A token can be generated in the Crowdin in Settings > API > click on New Token.
• set crowdin.projectsIds to the id of the Crowdin project. The project id can be found in the Details section on a project page. Multiple projects can be set by separating their id with a comma (for example crowdin.projectsIds=12,586,874).

When the configuration is done, the fetch of the Crowdin translations can be done by executing the script crowdin.bat or crowdin.sh located in the folder platform/tools/crowdin/. The PO files retrieved from Crowdin are automatically pasted in the folder translations, therefore the new translations are applied after the next Virtual Device restart.