@awarns/ml-kit

This module allows to execute TensorFlow Lite machine learning models embedded in the mobile device, which is useful to make predictions based on some input data.

Supported models architectures are Convolutional Neural Networks (CNN) and Multilayer Perceptrons (MLP), both for classification and regression problems.

Install the plugin using the following command line instruction:

ns plugin add @awarns/ml-kit

Usage

After installing this plugin, you will have access to an API and two tasks to perform classification or regression on the provided input data. But before using the plugin, you must meet the following requirements regarding the machine learning model:

• Have a TensorFlow Lite machine learning model (*.tflite) of the supported architectures (CNN or MLP). Classification models must have metadata (i.e., name, version, author, etc...) and an associated labels file with each label in a row of the file. While regression models don't have to include an associated labels file, it's recommended to add the metadata with the information of the model.
• Place your TensorFlow Lite models in a folder named ml-models inside your app's src folder (i.e., same level as {app|main}.ts).
  • The model file name must follow the next format: {model_name}-{cnn|mlp}-[version].tflite. The file name must contain a name (model_name), the model's architecture (cnn or mlp) and, optionally, the model's version (version). The file name elements must be splitted by a dash (-).

API

In order to do a regression or a classification, first you have to load a machine learning model. You can do that using the getModel(...) method provided by the ModelManager. It also provides the listModels method, useful known which models are available in the device.

When you load a model using the getModel(...) method, you obtain a BaseModel or a ClassificationModel to perform a regression or a classification, respectively. To do so, you can create a Regressor using a BaseModel, and a Classifier using a ClassificationModel. Finally, to perform the prediction, you just have to call to the predict method, which will return a RegressionResult or a ClassificationResult, depending on which predictor has been used.

Here's a complete example:

import {
  BaseModel,
  ClassificationModel,
  ClassificationResult,
  Classifier,
  DelegateType,
  getModelManager,
  InputData,
  ModelType, RegressionResult, Regressor
} from '@awarns/ml-kit';

async function doClassification(inputData: InputData /* number[] */) {
  const model: ClassificationModel = await getModelManager().getModel(
    'activity_classifier-cnn',
    ModelType.CLASSIFICATION,
    { acceleration: DelegateType.GPU } // Use GPU, if available.
  );

  const classifier = new Classifier(model);
  const result: ClassificationResult = classifier.predict(inputData);
}

async function doRegression(inputData: InputData /* number[] */) {
  const model: BaseModel = await getModelManager().getModel(
    'stress_regressor-cnn',
    ModelType.REGRESSION,
    { acceleration: 4 } // Use 4 threads.
  );

  const regressor = new Regressor(model);
  const result: RegressionResult = regressor.predict(inputData);
}

Note: the RegressionResult and ClassificationResult are not framework records. If you want to introduce these results into the framework for example, to persist them using the persistence package, you have to manually create a Regression and Classification records.

ModelManager

You can obtain the singleton instance of the ModelManager calling the getModelManager() function.

Model

ModelInfo

ModelType

ModelOptions

Regressor

Classifier

Tasks

Note: the input data provided through the invocation event's payload must be an array of numbers ready to be feed into the model. In other words, the main application is the one in charge of executing the required data preprocessing techniques (e.g., normalization, feature extraction, etc...) to prepare the data for the model, not this module.

To register these tasks for their use, you just need to import them and call their generator functions inside your application's task list:

import { Task } from '@awarns/core/tasks';
import {
  classificationTask,
  regressionTask,
  DelegateType,
} from '@awarns/ml-kit';
import { DelegateType } from './index';

export const demoTasks: Array<Task> = [
  classificationTask('human-activity', 'activity_classifier-mlp', '', { acceleration: 4 }),
  // humanActivityClassification

  classificationTask('human-activity', 'activity_classifier-cnn', 'UsingCNN', { acceleration: DelegateType.GPU }),
  // humanActivityClassificationUsingCNN
  
  regressionTask('stress-level', 'stress_regressor-cnn'),
  // stressLevelRegression
]

Task generator parameters:

Note: It's highly recommended to provide the {classification|regression}Aim in snake-case format. This string will be used as the type of the output record of the task. All the records have their type in snake-case, so providing the {classification|regression}Aim in snake-case will keep the consistency of the framework.

ModelNameResolver: () => string

Useful to change the model used by a task at runtime. You can use the ModelManager to obtain a list with the models that are available in the device.

ModelOptionsResolver: () => ModelOptions

Useful to change the options used by the model of a task at runtime.

Tasks output events:

• {classificationAim}Predicted
• {regressionAim}Predicted

Example usage in the application task graph:

on('inputDataForHumanActivity', run('humanActivityClassificationUsingCNN'));
on('humanActivityPredicted', run('writeRecords'));

on('inputDataStressLevel', run('stressLevelRegression'));
on('stressLevelPredicted', run('writeRecords'));

Note: To use the writeRecords task, the persistence package must be installed and configured. See persistence package docs.

Events

Records

Classification

ClassificationResult

ClassificationPrediction

Regression

RegressionResult

License

Apache License Version 2.0

Disclaimer

While we state that CNN models are supported, only 1D-CNN models have been tested. The code is general enough to support 2D and 3D CNN models with one input tensor, but they have not been tested. If you try 2D/3D-CNN models and something is not working as expected, contact us.