Cardboard Gesture Recognition with Embedded AI

An example showing how to easily add AI to an Arduino project using NanoEdge AI Studio

Mar 20, 2024

•

2081 views

•

3 respects

Arduino User Group

Data Collection

Embedded

Components and supplies

1

Arduino® UNO R4 WiFi

1

LIS3DH Triple-Axis Accelerometer

Apps and platforms

NanoEdge AI Studio

1

Arduino IDE 1.8.19

Project description

easily add AI to a project , without any knowledge in this field, using the software $ NanoEdge AI Studio $ and its Arduino compatible libraries!

This tutorial guides you through building a cardboard touchpad that relies on vibration analysis and an Embedded AI algorithm running on an Arduino UNO R4 . The UNO emulates a USB keyboard device.

Vibration data from the cardboard is captured using a basic accelerometer connected via the Qwiic connector. Within the UNO microcontroller, vibrations are classified using a NanoEdge AI library.

Based on the detected class, the touchpad triggers either a "PageUp" or "PageDown" keystroke.

Hardware:

1. Arduino UNO R4 Wifi

1. LIS3DH Triple-Axis Accelerometer (any accelerometer should work)

Software:

1. Arduino IDE (for Windows user, you may want to use the 1.8.19 desktop version)

1. NanoEdge AI Studio

NanoEdge AI Studio:

NanoEdge is a free machine learning software developed by STMicroelectronics which allow to easily create and integrate AI libraries to any cortex M microcontroller. Essentially, select a project type, import data locally, run a benchmark to find the best model automatically, test the model if you want and get an AI library.

In Nanoedge AI Studio, four kinds of projects are available, each serving a different purpose:

1. Anomaly detection (AD): to detect a nominal behavior and an abnormal one. Can be retrained directly on board.
2. 1 class classification (1c): Create a model to detect both nominal and abnormal behavior but with only nominal data. (In case you cannot collect abnormal examples)
3. N class classification (Nc): Create a model to classify data into multiple classes that you define
4. Extrapolation (Ex): Regression in short. To predict a value instead of a class from the input data (a speed or temperature for example).

Steps:

Open Arduino IDE and create a new project:

1. Copy data logger source code available below.
2. Click on Sketch > Include Library > Adafruit_LiS3DH to install the library.

Be careful : If your board is emulating a keyboard, you need to double press the reset button to be able to flash it.

Open the serial in Arduino IDE to check that the accelerometer data is correctly coming to your computer. (don't forget to close the serial after that)

Now we will automatically create an AI model able to recognize classes of gestures using vibration patterns.

Open NanoEdge AI Studio and create a new " N-Class Classification " project.

In project settings:

1. Set target to "UNO R4 Wifi"
2. Sensor to 3-axes accelerometer

In the Signal step , collect one dataset per class ("Nothing","Swipe","Multitap" or any gesture that you want) using serial port.

Collect one kind of gesture per dataset!

Here is how to proceed:

1. Make sure the board is connected to the pc with the data logger code flashed on it
2. Click ADD SIGNAL > FROM SERIAL
3. Make sure to select the right come port
4. Click START/STOP to collect data (100 buffers per signal should be enough)
5. Once finished click CONTINUE and then IMPORT

If everything is correct, you should see a new dataset added with plots and information.

Create the AI model:

Once you have all the classses that you want to recognize, go to the Benchmark step.

1. Click RUN NEW BENCHMARK.
2. Select all your datasets and click START.

NanoEdge AI Studio will take your data and look for a model that is able to classify them. (it also applies pretreatment on its own to your data)

You get the accuracy of the model and its RAM and Flash requirements.

You should reach around 99% pretty fast if you collected good data. You can stop the benchmark when it happens.

In the validation step, you can compare the libraries (model + preprocessing) if you want, look here for more info:

$ https://wiki.st.com/stm32mcu/wiki/AI:NanoEdge_AI_Studio#Validation $ 

Test the model:

The emulator step is more useful for us here, you can use the serial to test the model directly in the tool:

1. Click INITIALIZE EMULATOR
2. Click FROM SERIAL
3. Test your model with new real time data

The last step in NanoEdge is to get the AI library that we will use in Arduino IDE:

1. Click COMPILE LIBRARY
2. Get and extract the .zip file
3. The library is the .zip file in the folder Arduin , we will use it below:

Create the demo:

1. Open a new project in Arduino IDE
2. Get the main code below and paste it in your project
3. Click on Sketch > Include Library > Adafruit_LIS3DH
4. Add the Nanoedge AI Library (select the previously extracted zip):



5. Compile the code.
6. Flash the code.

It is finished, you can play with it.

Through this example, this tutorial shows how to add AI to an Arduino project using NanoEdge AI Studio. It is now up to you to think about other use case and do them on your own!

Code

Main code

c

The demo code

1/* If you want to use NEAI functions please, include NEAI library
2 * in your Arduino libraries then, uncomment NEAI parts in the following code
3 */
4
5/* Libraries part */
6#include <Wire.h>
7#include <Adafruit_LIS3DH.h>
8#include <Adafruit_Sensor.h>
9#include <Keyboard.h>
10#include <NanoEdgeAI.h>
11#include "knowledge.h"
12float input_user_buffer[DATA_INPUT_USER * AXIS_NUMBER]; // Buffer of input values
13float output_class_buffer[CLASS_NUMBER]; // Buffer of class probabilities
14uint16_t id_class = 0;
15  
16/* Macros definitions */
17#define SERIAL_BAUD_RATE  115200
18
19/* Default address is 0x18 but, if SDO is powered at 3v3,
20 *  address is set to 0x19, so you need to change it
21 *  depending on your current hardware configuration.
22 */
23#define SENSOR_I2C_ADDR 0x18
24
25/* Sensor data rates.
26 * You can choose from:
27 * LIS3DH_DATARATE_1_HZ
28 * LIS3DH_DATARATE_10_HZ
29 * LIS3DH_DATARATE_25_HZ
30 * LIS3DH_DATARATE_50_HZ
31 * LIS3DH_DATARATE_100_HZ
32 * LIS3DH_DATARATE_200_HZ
33 * LIS3DH_DATARATE_400_HZ
34 * LIS3DH_DATARATE_LOWPOWER_1K6HZ
35 * LIS3DH_DATARATE_LOWPOWER_5KHZ
36 */
37#define SENSOR_DATA_RATE	LIS3DH_DATARATE_400_HZ
38
39/* Sensor ranges.
40 * You can choose from:
41 * LIS3DH_RANGE_16_G
42 * LIS3DH_RANGE_8_G
43 * LIS3DH_RANGE_4_G
44 * LIS3DH_RANGE_2_G
45 */
46#define SENSOR_RANGE	LIS3DH_RANGE_2_G
47
48/* NanoEdgeAI defines part
49 * NEAI_MODE = 1: NanoEdgeAI functions = AI Mode.
50 * NEAI_MODE = 0: Datalogging mode.
51 */
52#define NEAI_MODE 1
53#define SENSOR_SAMPLES	256
54#define AXIS  3
55
56Adafruit_LIS3DH lis = Adafruit_LIS3DH();
57
58/* Global variables definitions */
59static uint16_t neai_ptr = 0;
60static float neai_buffer[SENSOR_SAMPLES * AXIS] = {0.0};
61
62
63/* Initialization function: In this function,
64 *  code runs only once at boot / reset.
65 */
66void setup() {
67  /* Init serial at baud rate 115200 */
68  Serial.begin(SERIAL_BAUD_RATE);
69
70  /* Init I2C connection between board & sensor */
71  if (!lis.begin(SENSOR_I2C_ADDR)) {
72    Serial.println("Can't initialize I2C comm with LIS3DH sensor...\n");
73    while(1);
74  }
75  Serial.println("OK");
76  /* Init LIS3DH with desired settings: odr & range */
77  lis.setRange(SENSOR_RANGE);
78  lis.setDataRate(SENSOR_DATA_RATE);
79
80  /* Initialize NanoEdgeAI AI */
81  enum neai_state error_code = neai_classification_init(knowledge);
82  if (error_code != NEAI_OK) {
83    Serial.println("Error starting NanoEdge AI lib");
84    /* This happens if the knowledge does not correspond to the library or if the library works into a not supported board. */
85  }
86
87  Keyboard.begin();
88  delay(1000);
89}
90
91/* Main function: Code run indefinitely */
92void loop() {
93  /* Get data in the neai buffer */
94  while (neai_ptr < SENSOR_SAMPLES) {
95     /* Check if new data if available */
96    if (lis.haveNewData()) {
97      /* If new data is available we read it ! */
98      lis.read();
99      /* Fill neai buffer with new accel data */
100      neai_buffer[AXIS * neai_ptr] = (float) lis.x;
101      neai_buffer[(AXIS * neai_ptr) + 1] = (float) lis.y;
102      neai_buffer[(AXIS * neai_ptr) + 2] = (float) lis.z;
103      /* Increment neai pointer */
104      neai_ptr++;
105    }
106  }
107  /* Reset pointer */
108  neai_ptr = 0;
109
110  /* Depending on NEAI_MODE value, run NanoEdge AI functions
111   * or print accelerometer data to the serial (datalogging)
112   */
113  
114  if (NEAI_MODE) {
115    neai_classification(neai_buffer, output_class_buffer, &id_class);
116
117    if (id_class == 1) {
118      Keyboard.write(KEY_PAGE_DOWN);
119      delay(100);  
120    } else if (id_class == 2) {
121      Keyboard.write(KEY_PAGE_UP);
122      delay(100); 
123    }
124  } else {
125    /* Print the whole buffer to the serial */
126    for (uint16_t i = 0; i < AXIS * SENSOR_SAMPLES; i++) {
127      Serial.print((String)neai_buffer[i] + " ");
128    }
129    Serial.print("\n");
130  }
131
132  /* Clean neai buffer */
133  memset(neai_buffer, 0.0, AXIS * SENSOR_SAMPLES * sizeof(float));
134}

Data logger

c

Data logger code to collect data

1/* If you want to use NEAI functions please, include NEAI library
2 * in your Arduino libraries then, uncomment NEAI parts in the following code
3 */
4
5/* Libraries part */
6#include <Wire.h>
7#include <Adafruit_LIS3DH.h>
8#include <Adafruit_Sensor.h>
9
10/* Macros definitions */
11#define SERIAL_BAUD_RATE  115200
12
13/* Default address is 0x18 but, if SDO is powered at 3v3,
14 *  address is set to 0x19, so you need to change it
15 *  depending on your current hardware configuration.
16 */
17#defin