The purpose of this project is to make a custom micropython firmware that installs tensorflow lite for micro controllers and allows for experimentation.
I want to incubate this project here but if it is successful at enabling tensorflow lite for microcontroller models to be run from micropython I will explore how to merge the microlite and audio_frontend modules into the tensorflow for microcontrollers upstream repository.
| Area | Are |
|---|---|
| tensorflow-microlite.a | This is the tensorflow lite micro library with all operations to start with |
| micropython-modules/microlite | Micropython 'microlite' module that interconnects micropython to tensorflow lite micropython c++ api |
| micropython-modules/audio_frontend | Micropython Wrapper for tensorflow experimental microfrontend audio processor needed by the micro_speech example to convert wav data into spectrogram data for processing by the model. |
| micropython-modules/ulab | We have a submodule dependency on ulab. |
At the moment all of the tensor operations are included in tensorflow-microlite.a but in the future we will try to externalize them so they can be brought on the file system with the model being run.
Because we have all of the operations there is a cost in the size of the image generated.
| Build Type | Status |
|---|---|
| ESP32 |  |
| UNIX |  |
The latest firmware can be downloaded from the applicable build above (in the Status section).
- Click on build status link.
- Click on the latest green build
- Review the available artifiacts for download
You do need to be careful to get the proper firmware for your board. If your board is not currently being built please file an issue and it can be added.
Give a model an x value and it will give a y value. The chart of such points is an approximate sine wave:
Status:
- Works on unix port and esp32 port.
Process:
- Sample Audio
- Convert to spectrogram
- Set 1960 bytes on input tensor corresponding to 1 second worth of spectrogram.
- Run inference on model 3-4 times per second and average scores.
- Scores over 200 indicate a match
Status:
- Works on unix port with files.
- Works on esp32 no-spi ram board with machine.I2S.
Process:
- Capture Images
- Convert to 96x96 pixel int8 greyscale images
- Set on input later of model
- Run inference on image
- if person > no person it thinks the image is a person
- if no person > person it thinks the image contains no person
Status:
- Works on unix port and esp32 port using files.
Person Detection Documentation
Almost all of the examples work now.
| # | Description | Issue |
|---|---|---|
| 1. | Adjust microlite api to match tflite python api (at least a subset). We don't really need the callback approach anymore. | mocleiri#10 |
| 2. | Cleanup micro_speech implementation | |
| 3. | Implement the magic wand example | mocleiri#5 |
| 4. | Investigate automatic tensor quantization | mocleiri#6 |
| 5. | Find way to externalize tensor op's from firmware | mocleiri#7 |
At the moment we are using the main branch in the tensorflow lite micro repository.
This is the c++ api version of tensorflow lite designed to run on microcontrollers.
At the moment we are building everything and linking it into the micropython firmware.
The next steps are to try and find ways to use build switches or native modules to support smaller firmware sizes.
We are building from micropython master.
Normally you can just download the precompiled firmware. If you want to know how to build it yourself then consult the Build Documentation.
ESP32D0WDQ6 4MB Flash
Download the firmware from the latest ci build.
The zip file contains:
- The bootloader
- The partition table
- The firmware
esptool.py -p COM5 -b 460800 --before default_reset --after hard_reset --chip esp32 write_flash --flash_mode dio --flash_size detect --flash_freq 40m 0x1000 bootloader/bootloader.bin 0x8000 partition_table/partition-table.bin 0x10000 micropython.bin
As far as I am aware OpenMV (https://openmv.io/) was the first micropython firmware to support tensorflow.
I copied extensively from their approach to get inference working in the hello world example and also for micro-speech example.
I started from their libtf code for how to interact with the Tensorflow C++ API from micropython:
https://github.com/openmv/tensorflow-lib/blob/343fe84c97f73d2fe17a0ed23540d06c782fafe7/libtf.cc and https://github.com/openmv/tensorflow-lib/blob/343fe84c97f73d2fe17a0ed23540d06c782fafe7/libtf.h
The audio-frontend module originated by looking at how openmv connected to the tensorflow microfrontend here: https://github.com/openmv/openmv/blob/3d9929eeae563c5b370ac86afa9216df50f0c079/src/omv/ports/stm32/modules/py_micro_speech.c