VEE Audio Framework

The VEE Audio Framework provides the building blocks to create an Audio Kernel executable and develop Audio Applications.

The Framework provides the following components:

  • the Audio Kernel

  • the Audio Service Library

These components expose the APIs used to build Audio Applications, of two kinds: a Voice Assistant and regular audio applications.

VEE Audio Kernel

The Audio Kernel is the core runtime of VEE Audio. It starts first at boot and remains resident for the lifetime of the device. Its responsibilities include:

  • Managing the lifecycle of all installed applications: starting, stopping, and uninstalling them on demand.

  • Exposing the Audio Service Library to applications through a controlled, versioned API.

  • Enforcing audio focus arbitration to ensure only one application produces sound at a time.

  • Acting as the trusted boundary between the hardware platform and the application layer.

The Kernel can run standalone, with a fixed set of applications bundled at firmware build time, or it can operate as an open runtime that accepts dynamically deployed applications at any point during device operation. This enables over-the-air application updates, A/B experimentation, and incremental feature rollout on shipped products.

The Kernel provides the following Foundation Libraries to the applications:

Library

Version

EDC

1.3

BON

1.4

KF

1.7

Audio

2.0

NET

1.1

SSL

2.2

Security

1.7

Trace

1.1

Event

2.1

VEE Audio Service Library

The Audio Service Library allows Audio Applications to access platform capabilities provided by the Kernel.

The KernelServiceProvider class gives access to every service instance.

The library provides the following services, implemented by the Kernel and consumed by the applications:

Service

Role

ApplicationService

Register applications and Voice Assistants with the Kernel

AudioService

Audio playback (AudioTrack), recording (AudioRecord), focus arbitration, volume, noise cancellation, and equalizer

ConnectivityService

Network connectivity and waitForConnectivity blocking call

PersistentStoreService

Typed key-value store for retaining configuration across device restarts

LedService

Programmatic control over the device’s status LEDs

EventService

Publish-subscribe mechanism for inter-application event dispatching

MdnsService

Zero-configuration local service discovery via Multicast DNS

The library also provides the following interfaces, implemented by the applications:

Interface

Role

Application

Declares the application lifecycle (start, stop) and the VoiceAssistantFunction instances it exposes

VoiceAssistant

Voice interface between the user and the installed applications; receives all Function registrations and dispatches voice commands

VoiceAssistantFunction

A discrete capability exposed by an application that the Voice Assistant can invoke on behalf of the user

Audio Focus

When multiple applications want to produce sound simultaneously, the AudioService arbitrates access through the Audio Focus mechanism. An application must request focus before opening an AudioTrack or activating an AudioRecord. At the system level, the audio sources are ordered by priority: Bluetooth music has the lowest priority, application audio preempts it, and a phone call preempts everything.

Each focus request carries three configurable properties:

  • Priority: determines the relative importance of the request.

  • canWait: when true, a request is queued until the current holder releases focus; when false, the denial is immediate.

  • canSuspend: allows a holder to be temporarily displaced by a same-priority request, resuming automatically when the displacing request releases focus.

The possible outcomes of a focus request are:

  • Granted: focus is available and assigned immediately.

  • Preemption: the new request has higher priority; the current holder loses focus.

  • Queued: the request is pending because a higher-priority holder is active and canWait is true.

  • Denied: the request cannot be satisfied and canWait is false.

  • Transient gain: the request temporarily displaces a same-priority holder with canSuspend set to true.

Pending requests are granted focus in priority order; requests at the same priority level are served in arrival order (FIFO).

Application Events

When a hardware event occurs (for example, a button press), the Kernel dispatches it in priority order:

  1. First to the application currently holding audio focus, through a registered ApplicationEventHandler on the focus request.

  2. If the focus holder does not consume the event, it is forwarded to the current application.

An event is considered consumed when its handler returns true, stopping further dispatch. This allows, for example, an alarm ringing at maximum audio priority to intercept a button press and dismiss itself before the Voice Assistant can act on it.

Audio Applications

An Audio Application is the deployable unit of functionality in VEE Audio. Applications are packaged as MicroEJ Features (lightweight binary modules that the Kernel can load, start, stop, and unload independently). A single Feature package can contain multiple applications, each registering independently with the Kernel at startup.

There are two kinds of Audio Applications:

Voice Assistant

A Voice Assistant is a special kind of Audio Application that acts as the voice interface between the user and all installed applications. It is the primary entry point for user interaction and has two roles:

  • Voice UI: it receives all VoiceAssistantFunction registrations from every installed application, matches incoming voice commands against them using an AI backend, and invokes the corresponding handler.

  • Conversational agent: it handles open-ended interactions that do not map to any registered Function, acting as a general-purpose assistant.

Note

Only one Voice Assistant can be active at a time. Registering a new one automatically replaces the previous one, with no state carried over from the replaced Voice Assistant.

Note

Three reference Voice Assistants are provided, each backed by a different AI service. See Reference Voice Assistants for their descriptions and specificities.

AI-Backend Agnostic Design

VEE Audio is Voice Assistant-agnostic: any realtime speech AI backend can be connected to the framework by implementing the VoiceAssistant interface. Swapping the AI backend does not affect the installed applications: their declared VoiceAssistantFunction instances remain unchanged and continue to be invoked by whichever Voice Assistant is active.

Function Calling

Applications expose their capabilities to the Voice Assistant through a Function Calling mechanism. Each VoiceAssistantFunction is described by:

  • a unique name that identifies it within the Voice Assistant’s function registry,

  • a human-readable description that the AI backend uses to match user intent,

  • a set of typed parameters, each with a name, type, and description, forming the input schema.

When the Voice Assistant determines that a user’s spoken intent matches a registered function, it extracts the relevant parameters and invokes the function’s handler with a structured argument payload. The handler performs the requested action and returns a JSON result. Functions that involve long-running operations can declare themselves as asynchronous, causing the framework to execute them in a dedicated thread.

The Kernel collects the VoiceAssistantFunction instances declared by each installed application and forwards them to the active Voice Assistant in real time, so adding a new application automatically extends the set of available voice commands, without any manual wiring.

AI Query API

Beyond dispatching voice commands to applications, the VoiceAssistant interface also exposes a processRequest() method that applications can call to leverage the underlying AI model directly. An application submits a text query and a set of instructions; the Voice Assistant forwards them to its AI backend and returns the result as a string.

This API is used by the Web Search application to have the AI model search the web and summarize the results, which are then spoken back to the user. After returning a function result that should be interpreted by the Voice Assistant without waiting for a new user prompt, an application can also call requestImmediateReply() to instruct the Voice Assistant to generate a follow-up response immediately, continuing the conversation.

Secure WebSocket

Voice Assistants communicate with their AI backends over a WebSocket API with SSL/TLS support, enabling persistent, bidirectional, full-duplex connections for real-time audio streaming and response reception.

A WebSocket instance is created by specifying the target URI, an event endpoint that handles incoming messages and connection lifecycle events, and an SSL context constructed from the platform’s certificate store.

Regular Audio Applications

Regular audio applications implement audio playback, recording, or any other user-facing audio feature. Any number of regular applications can be registered simultaneously.

Each application registers its capabilities as VoiceAssistantFunction instances with the Kernel at startup. The Kernel forwards these Functions to the active Voice Assistant in real time, so the user can invoke any application’s capabilities by voice, without any manual wiring between applications.

See Reference Applications for the sample applications shipped by VEE Audio.