(60) A true infrared video camera: AMG8833 sensor, ESP32 microcontroller and 3.2 inch 320*240 TFT display

door Floris Wouterlood – The Netherlands – August 2, 2022

Summary
One of the attractive features of the AMG8833 8*8 pixel thermal infrared sensor is its 10Hz data refresh rate. Wired to an ESP32 microcontroller board such a fast datastream can be exploited to the max.
The software running on the ESP32 interpolates each of the sensor’s 8*8 pixel thermal ‘image frames’ into a 24*24 pixel false color image. The goal here is to scale these 24*24 pixel images to fill a 240*240 pixel checkerboard field in false colors, at the full 10 frames per second. The fastest drivers achieved on a 3.2 inch 320*240 pixel TFT display a cycle refresh rate of 10.3 Hz. A true infrared video camera!

figure 1. Live, 10 frames per second! Interpolated images acquired with a 10 Hz AMG8833 8×8 pixel infrared thermal imager and processed with an ESP32 microcontroller.

Introduction
For Arduino hobbyists the AMG8833 thermal sensor opens the door to the sometimes surrealistic world of infrared thermal imaging. This sensor is capable of recording images in the temperature range of 0 – 80 oC. Resolution is a modest 8*8 pixels. In previous papers I have discussed wiring of the AMG8833 to the Arduino Nano (*) and ESP8266 (**) microprocessor boards. With the exception of a configuration that includes an ESP8266 and a 130*130 TFT display (**) the image frame refresh rate was far below the maximum sensor refresh rate – claimed to be 10 Hz by the manufacturer. This difference is probably caused by the limited processing power of the Nano and the ESP8266s. The breathtaking processor power inside an ESP32 microcontroller might be exploited to achieve a 10 Hz display image refresh rate, maybe even with interpolated image frames and on big screens. The latter goal was pursued in this project by hooking the sensor up to a previously constructed ESP32 workbench: a board with on it an ESP32WROOM-32 microcontroller, a 3.2 inch color TFT display (320*240; ILI9341 controller) and a score of pin headers that suit experiments (***).

figure 2. AMG8833 Grid-EYE 8*8 pixel thermal scanning sensor mounted on an Arduino-compatible mini breakout board. The breakout has six pins of which four are essential: VIN, GND, SCL (clock) and SDA (data). Communication is via the I2C protocol.

About the sensor
Panasonic, the Japanese electronics giant, produces the Grid-EYE AMG8833, a low-cost, low resolution 8*8 pixel matrix sensor made to fit home appliances, energy saving devices, automatic door sensors and so forth (figure 2). Its specs are: temperature range 0-80 oC with an accuracy of 2.5 degrees; vertical and horizontal viewing angles 60 degrees; detection distance (say, detecting a human being) approximately 7 meters; frame rate 10 per second. I2C address 0x69. The breakout device is 3.3V-5V compliant.

Sensor data stream
As each ‘pixel’ in the sensor device registers temperature ten times per second the complete sensor data stream consists of 10 series of 64 temperatures per second. The microcontroller’s job is to compare each of the infrared ‘temperatures’ with a so-called color lookup table, determine the corresponding color and send that color to the display, with ‘cold’ as deep blue and ‘warm’ as hot red. The data from the thermal sensor is usually displayed in 8×8 color tiles where each tile is as high and wide as the display accommodates, for instance 16 pixels high and 16 pixels wide on a 128*128 screen. Because this is ‘live’ processing, microcontroller processing speed, program efficiency and display refresh rate become interesting issues. The display is at the end of the processing pipe and there is complex calculation all along. While an AMG8833 delivers at 10 Hz all downstream processing takes its toll and slows down the refresh rate of the images finally presented on the display.

figure 3. Left: scaled, tiled 8*8 false color image. Right: the same dataset interpolated to a 24*24 false color picture.

The basic sketch translates the sensor data into 64 screen pixels. On a 320*240 color display scaling can be done to 8×8 color tiles with each tile consisting of 30*30 pixels (figure 3, left; figure 4, upper). The disadvantage of ‘up-scaling’ to color tiles is that the final thermal image looks so ‘squary’, such as the image I took as ‘selfie’: my head consists here of one hot colored tile! (figure 3, left; figure 4, upper).

Interpolation
With the interpolation algorithm supplied in the Adafruit_AMG88xx library the temperature value of every sensor pixel is compared with that of the pixels surrounding it. Special is that the temperature value of each original pixel is transferred to a block of 9 ‘child’ pixels. The central pixel in this 3*3 block remains unaffected. The temperature values of all 8 pixels surrounding the central ‘child’ pixel are averaged with those of the apposing pixels belonging to the next block of ‘child’ pixels. Thus, from a first generation of 64 parent pixels a second generation consisting of 576 child pixels is calculated, with averaged temperature values. The second generation screen image has dimensions 24*24 pixels. The results can also be put on display as colored tiles that now can be kept smaller than in first-generation display images. The visual representation of the thermal data is in the interpolated image much more appreciable than the original 8*8 block image while there is no loss of information (figure 3, right; figure 4, lower screen half).

figure 4. Two selfies together! Test bench with a 3.2 inch, 320*240 SPI TFT with ILI9341 controller. Wiring provided in figure 5. For this particular illustration I wrote a special sketch that makes the ‘raw’ thermal image with 8*8 pixel color tiles appear on the upper half of the screen and the interpolated 24*24 color tiled on the lower half. Note that the thermal image is on screen 128 screen pixels high – 128 screen pixels wide while the interpolated image has dimensions 120*120 screen pixels. The difference in image size is because of color tile dimensions. Original 8*8 color tile ‘selfie’ image acquired with a AMG8833 thermal sensor.

figure 5. Wiring diagram for the AMG8833, ESP32 and the 3.2 inch SPI TFT

The display used here is a 3.2 inch 320*240 SPI TFT display with ILI9341 controller. It is attached to an assembly developed previously as a test bench (***) (Figure 4). Having a stable and proven test bench at hand is always very practical. The hardware I2C pins (D20, D22) have on the bench their own female socket header, ready to be used. The wires from the AMG8833 thermal sensor were plugged in (D22 = SCL and D21 = SDA in I2C communication) and the device worked immediately, without problems. Figure 5 presents the wiring diagram.

Testing refresh rates
In a video system, refresh rate is expressed in frames per second. Note that we are dealing here with two refresh rates. The first, further called ‘sensor refresh rate’, is the speed with which the sensor delivers 8*8 temperature measurements to the microcontroller. The second refresh rate – the rate that matters – is further called ‘display refresh rate’, is the refresh speed on the display of the interpolated thermal images. The displays used here are capable of much higher refresh rates than 10 Hz. As we are interested in the total performance of the system it is best to measure performance in cycles per second, using the appropriate unit, Hertz. A ‘cycle’ here is defined as one iteration of void loop () in the Arduino sketch.
The duration of one loop cycle can be determined by counting how many milliseconds it takes to go through 100 cycles. The resulting score is then sent to Serial Monitor. A few instruction lines were incorporated for this purpose in all experiments. These lines (cycle counter, with an ‘if’ statement to trigger Serial Monitor output) slow a cycle down, or better, prolong it, and therefore result in systematic underreporting. A small price to pay for monitoring!
The bench produced initially with this configuration a cycle refresh rate of 6.47 Hz.

Tricks to improve performance
There are several ways to increase the cycle refresh rate:
• Reduce the image frame size. This improved performance up to a 7.76 Hz cycle refresh rate.
• Select the most efficient combination of thermal sensor and display controller libraries (‘drivers’). In the previous experiments with an ESP8266 microcontroller (**) I noted that including the SparkFun_GridEYE driver gave slower cycle refresh rates than including the Adafruit_AMG88xx driver. In these experiments the best combination featured the Adafruit_AMG88xx driver in combination with the Adafruit_GFX and Adafruit_ILI9341 drivers.
• The Adafruit_GFX driver was in combination with an ESP32 very inefficient, producing video at a refresh rate of 1.06 Hz.

Streamlining the sketch, achieving 10 Hz
A considerable amount of time was invested to streamline the best performing sketch. The ‘winning’ sketch uses the GridEYE and TFT_eSPI libraries with as few instructions lines as possible. Cycle refresh rate was clocked at an astounding 10.1 Hz, or equal to the sensor’s output stream.

Other drivers and combination of drivers
The initial process of getting the experiments working were done with sketches in which I applied the SparkFun_GridEYE_Arduino_Library.h and TFT_eSPI.h as libraries supporting respectively the thermal sensor and the display. With this combination the display rate – as explained above – was 6.47 Hz.
Attempts to increase the frame rate by replacing TFT_eSPI.h by the Adafruit_ILI9341.h display controller library did not lead to spectacular results: an unimpressive 1.1 Hz.
The other way around: replacing the SparkFun_GridEYE.h library by the Adafruit_AMG88xx.h library in an attempt to speed up sensor data processing was even more disappointing: no coherent images were obtained.

The LovyanGFX.hpp driver
A relatively unknown driver that supports displays with the ILI9341 controller is LovyanGFX.hpp. This driver can be installed via the Library Manager of the Arduino IDE. The instructions seem strange since they point at ‘lcd’ instead of the familiar ‘tft’. Nevertheless a spectacular 11 Hz cycle refresh rate was obtained, with the LovyanGFX.hpp driver in combination with SparkFun_GridEYE.h supporting the thermal sensor.
Combination of the LovyanGFX driver with Adafruit’s AMG88xx driver whttp://www.zonnepanelen.wouterlood.com/arduinostuff/ESP32_AMG8833_interpol.zipas unsuccessful.

Discussion: Interpolated infrared video with an ESP32

figure 6. The end product: a fast video-speed 10 Hz big interpolated image stream on my ESP32 bench with 320*240 TFT display. The interpolated video frames are upscaled to 240*240 pixels.

The ingredients for building an infrared video camera are simple: an ESP32 microcontroller, AMG8833 thermal infrared sensor and a fast display supporting 240*240 images. The sketch produces software speed. The best performance was obtained with the SparkFun_GridEYE.h library for the thermal sensor, and with the LovyanGFX.hpp library supporting display control: 11 Hz. The Lovyan library has some instructions that at first sight look unfamiliar but otherwise the instruction set is very straightforward, and it follows Adafruit_GFX instruction conventions. The very versatile TFT_eSPI driver follows Adafruit_GFX instruction convention as well and moreover seems to be optimized for ESP microcontrollers. TFT_eSPI is a system that works with external display type configuration files instead of a display type-defining ‘constructor’ included in the sketch. It is my favorite display controller driver. With this driver, combined with the SparkFun_GridEYE.h library supporting the sensor I obtained the 10 Hz cycle refresh rate.
An amazing observation is that the original AMG8833 driver, the Adafruit_AMG88xx.h library, works best with microcontroller boards that use ATmega328 (Arduino Uno, Nano) or ESP8266 (NodeMCU, Wemos D1 mini) processors. Among these microcontrollers the combination Adafruit_AMG88xx.h and Adafruit_GFX.h performed best (*) (**).
At the viewer’s end of any infrared video camera it is the display refresh rate that is paramount. An ESP32 succeeds into bringing the full sensor’s published specification speed – 10 Hz – to the display, at a screen size of 240*240 pixels, and with interpolation implemented. In this configuration it is the sensor refresh rate that limits performance of the video camera system.

Sketch
In the course of this project I prepared numerous sketches. The winning sketch is published here: the 10 Hz cycle refresh speed sketch using the SparkFun_GridEYE and TFT_eSPI drivers, with the interpolation function supplied by Adafruit.

Sketch: ESP32_AMG8833_interpol.ino

Note on hardware
See figures 4, 5 and 6. Libraries that need to be installed in your Arduino IDE: SparkFun_GridEYE.h (sensor support), Adafruit_AMG88xx.h (interpolation example) and TFT_eSPI (support for the ILI9341 display controller). The User_setup.h file in the TFT_eSPI must match your display controller.

Downloadable
Sketch is packed in a ZIP file named ‘ESP32_AMG8833_interpol.zip’

References:

(*) 8*8 pixel AMG8833 GRID-EYE thermal scanner and the Arduino Nano. Thesolaruniverse.wordpress.com – April 4, 2022.

(**) 8*8 pixel AMG8833 thermal sensor, ESP8266 microcontroller board and a SSD1283 driven TFT display. Thesolaruniverse.wordpress.com – May 2, 2022.

(***) ESP32-WROOM-32 and ILI9341 TFT display – an interesting match. Thesolaruniverse.wordpress.com – March 1, 2021.