People-finding with a Lepton

This is a tutorial for implementing a basic people-finding algorithm with a FLIR Lepton camera using the open source computer vision library OpenCV.

The Goal

Thermal cameras are great at finding mammals in nearly any lighting conditions. As an exercise to explore what they can do, let’s try and find people in the Lepton’s field of view and put an outline around them with OpenCV.

The Tools

Hardware you’ll need:

1. A Lepton
2. A PureThermal board
3. A Python 2.7 environment with OpenCV bindings installed. This can be set up Windows, OSX or Linux.
4. PIL if you want to be able to save images.

The Setup

Follow a tutorial for your specific platform to get the Python environment set up, and OpenCV installed. Once you’re done, verify everything works by viewing a webcam stream.

import cv2
cv2.namedWindow("preview")
cameraID = 0
vc = cv2.VideoCapture(cameraID)

if vc.isOpened(): # try to get the first frame
rval, frame = vc.read()
else:
rval = False

while rval:
cv2.imshow("preview", frame)
rval, frame = vc.read()
key = cv2.waitKey(20)
if key == 27: # exit on ESC
break

This should show you a stream. If you have a webcam attached or integrated into your computer, you may need to change cameraID to a value other than 0. On my development machine, the PureThermal board’s ID is 1.

The Approach

OpenCV’s webcam capture code isn’t capable of capturing radiometric thermal data, which would be an ideal format for people-counting, but it does let you capture the colorized feed from a PureThermal board, which will be good enough for drawing outlines with a little preprocessing.

Specifically, humans tend to show up as very bright in the color default color palette, so converting the RGB images to HSV and looking at the V channel gives a pretty clear picture of where body-temperature objects are in the scene.

Try it out by swapping out cv2.imshow("preview", frame) with the following:

frame_hsv = cv2.cvtColor(frame, cv2.COLOR_RGB2HSV)
frame_v = frame_hsv[:,:,2]
cv2.imshow("preview", frame_v)

Now it’s pretty obvious where the humans are, and we have something that we can do computer vision with.

Enter OpenCV

OpenCV is a very popular computer vision library for C++ with bindings to Python. It offers a wide variety of common computer vision operations, which we’ll use to draw our outlines.

Canny edge detection is where we’ll start. It’s a robust technique for finding edges in an image.

You can view the edges that OpenCV detects with this code:

thresh = 50
edges = cv2.Canny(frame_v,thresh,thresh*2, L2gradient=True)
cv2.imshow("preview", edges)

This doesn’t look very good though. The edge detection is picking up too much high-frequency noise and mistaking it for edges. A little image smoothing should be able to sort that out. We’ll use an edge-preserving image smoothing method called a bilateral filter. This is like a gaussian blur but it has less of an impact on the edges we’re looking to find in the first place.

blurredBrightness = cv2.bilateralFilter(frame_v,9,150,150)
thresh = 70
edges = cv2.Canny(blurredBrightness,thresh,thresh*2, L2gradient=True)
cv2.imshow("preview", edges)

This looks way better, but there’s still room for improvement. Let’s try and cut down on things like lights being outlined as people. This is tricky, but OpenCV provides a way to do it. First, we’ll create a binary image by thresholding the original image and putting a 1 wherever the pixel is warm, and a 0 where it’s not. Then, we’ll use OpenCV to erode away at the blobs of 1’s created by that operation. After that, we’ll expand those blobs again to be roughly the same size as before.

_,mask = cv2.threshold(blurredBrightness,200,1,cv2.THRESH_BINARY)
erodeSize = 5
dilateSize = 7
import numpy as np
eroded = cv2.erode(mask, np.ones((erodeSize, erodeSize)))
mask = cv2.dilate(eroded, np.ones((dilateSize, dilateSize)))

After the erosion and dilation, the binary image is left essentially the same, but with all the small shapes removed. That’s exactly what we want. Now we can use it to mask away all the edges that belonged to small shapes.

Let’s see how this looks when applied to the detected edges.

Not bad! And it looks pretty good overlayed onto the source image too.

The final code looks something like this. You may need to tune the constants to your liking, and OpenCV provides a vast assortment of tools that you could use to improve results for your specific needs.

import cv2
import numpy as np
cv2.namedWindow("preview")
cameraID = 0
vc = cv2.VideoCapture(cameraID)

if vc.isOpened(): # try to get the first frame
rval, frame = vc.read()
else:
rval = False

while rval:
frame_v = cv2.cvtColor(frame, cv2.COLOR_RGB2HSV)[:,:,2]

blurredBrightness = cv2.bilateralFilter(frame_v,9,150,150)
thresh = 50
edges = cv2.Canny(blurredBrightness,thresh,thresh*2, L2gradient=True)

_,mask = cv2.threshold(blurredBrightness,200,1,cv2.THRESH_BINARY)
erodeSize = 5
dilateSize = 7
eroded = cv2.erode(mask, np.ones((erodeSize, erodeSize)))
mask = cv2.dilate(eroded, np.ones((dilateSize, dilateSize)))

cv2.imshow("preview", cv2.resize(cv2.cvtColor(mask*edges, cv2.COLOR_GRAY2RGB) | frame, (640, 480), interpolation = cv2.INTER_CUBIC))

rval, frame = vc.read()
key = cv2.waitKey(20)
if key == 27: # exit on ESC
break