Category Archives: Computer Vision

Architecture For DIY AI-Driven Home Security

The raspberry pi’s ecosystem makes it a tempting platform for building computer vision projects such as home security systems. For around $80, one can assemble a pi (with sd card, camera, case, and power) that can capture video and suppress dead scenes using motion detection – typically with motion or MotionEye. This low-effort, low-cost solution seems attractive until one considers some of its shortfalls:

  • False detection events. The algorithm used to detect motion is suseptible to false positives – tree branches waving in the wind, clouds, etc. A user works around this by tweeking motion parameters (how BIG must an object be) or masking out regions (don’t look at the sky, just the road)
  • Lack of high level understanding. Even in tweeking the motion parameters anything that is moving is deemed of concern. There is no way to discriminate between a moving dog and a moving person.

The net result of these flaws – which all stem from a lack of real understanding – is wasted time. At a minimum the user is annoyed. Worse they are fatigue and miss events or neglect responding entirely.

By applying current state of the art AI techniques such as object detection, facial detection/recognition, one can vastly reduce the load on the user. To do this at full frame rate one needs to add an accelerator, such as the Coral TPU.

In testing we’ve found fairly good accuracy at almost full frame rate. Although Coral claims “400 fps” of speed – this is inference, not the full cycle of loading the image, running inference, and then examining the results. In real-world testing we found the full-cycle results closer to 15fps. This is still significantly better than the 2-3 fps one obtains by running in software.

In terms of scalability, running inference on the pi means we can scale endlessly. The server’s job is simply to log the video and metadata (object information, motion masks, etc.).

Here’s a rough sketch of such a system:

This approach is currently working successfully to provide the following, per rpi camera:

  • moving / static object detection
  • facial recognition
  • 3d object mapping – speed / location determination

This is all done at around 75% CPU utilization on a 2GB Rpi 4B. The imagery and metadata are streamed to a central server which performs no processing other than to archive the data from the cameras and serve it to clients (connected via an app or web page).

Installing OpenCV 4.1.1 on Raspberry Pi 4

Recently I purchased a raspberry pi 4 to see how it performs basic computer vision tasks. I largely followed this guide on building opencv

However, in the guide there are several missing dependencies on a fresh version of raspbian buster. There are also some apparent errors in the CMakeLists.txt file which other users already discovered. After patching these fixes and adding the needed missing dependencies, I now have opencv running on my pi. Here’s my complete script below.

IMPORTANT: You must READ the script, don’t just run it! For example, you should check the version of python you have at the time you run this script. When I ran it i was at python 3.7. Also, feel free to bump to a later version (or master) of opencv.

Oh, also during the numpy step it hung and i was too lazy to look into it. It didn’t seem to affect my ability to use opencv – so i didn’t go back and dig. My bad.



sudo apt-get -y purge wolfram-engine
sudo apt-get -y purge libreoffice*
sudo apt-get -y clean
sudo apt-get -y autoremove

mkdir -p $opencvDirRoot
cd $opencvDirRoot

# Clean build directories
rm -rf opencv/build
rm -rf opencv_contrib/build

# Create directory for installation
rm -fr installation
mkdir -p installation
mkdir installation/OpenCV-"$cvVersion"

sudo apt -y update
sudo apt -y upgrade
sudo apt-get -y remove x264 libx264-dev
## Install dependencies
sudo apt-get install libblas-dev liblapack-dev
sudo apt-get install libeigen3-dev
sudo apt-get -y install qtbase5-dev qtdeclarative5-dev
sudo apt-get -y install build-essential checkinstall cmake pkg-config yasm
sudo apt-get -y install git gfortran
sudo apt-get -y install libjpeg8-dev libjasper-dev libpng12-dev

sudo apt-get -y install libtiff5-dev
sudo apt-get -y install libtiff-dev

sudo apt-get -y install libavcodec-dev libavformat-dev libswscale-dev libdc1394-22-dev
sudo apt-get -y install libxine2-dev libv4l-dev
cd /usr/include/linux
sudo ln -s -f ../libv4l1-videodev.h videodev.h
cd $opencvDirRoot

sudo apt-get -y install libgstreamer0.10-dev libgstreamer-plugins-base0.10-dev
sudo apt-get -y install libgtk2.0-dev libtbb-dev qt5-default
sudo apt-get -y install libatlas-base-dev
sudo apt-get -y install libmp3lame-dev libtheora-dev
sudo apt-get -y install libvorbis-dev libxvidcore-dev libx264-dev
sudo apt-get -y install libopencore-amrnb-dev libopencore-amrwb-dev
sudo apt-get -y install libavresample-dev
sudo apt-get -y install x264 v4l-utils
sudo apt-get -y install libmesa-dev
sudo apt-get -y install freeglut3-dev

# Optional dependencies
sudo apt-get -y install libprotobuf-dev protobuf-compiler
sudo apt-get -y install libgoogle-glog-dev libgflags-dev
sudo apt-get -y install libgphoto2-dev libeigen3-dev libhdf5-dev doxygen

sudo apt-get -y install python3-dev python3-pip
sudo -H pip3 install -U pip numpy
sudo apt-get -y install python3-testresources

cd $opencvDirRoot
# Install virtual environment
python3 -m venv OpenCV-"$cvVersion"-py3
echo "# Virtual Environment Wrapper" >> ~/.bashrc
echo "alias workoncv-$cvVersion=\"source $opencvDirRoot/OpenCV-$cvVersion-py3/bin/activate\"" >> ~/.bashrc
source "$opencvDirRoot"/OpenCV-"$cvVersion"-py3/bin/activate

############ For Python 3 ############
# now install python libraries within this virtual environment
sudo sed -i 's/CONF_SWAPSIZE=100/CONF_SWAPSIZE=1024/g' /etc/dphys-swapfile
sudo /etc/init.d/dphys-swapfile stop
sudo /etc/init.d/dphys-swapfile start
pip install numpy dlib
# quit virtual environment

git clone
cd opencv
git checkout $cvVersion
cd ..

git clone
cd opencv_contrib
git checkout $cvVersion
cd ..

cd opencv
mkdir build
cd build

# Eigen/Core to eigen3/Eigen/Core
sed -i s,Eigen/Core,eigen3/Eigen/Core/g ../modules/core/include/opencv2/core/private.hpp

# Add these to  opencv/samples/cpp/CMakeLists.txt 
find_package(OpenGL REQUIRED)
find_package(GLUT REQUIRED)

            -D CMAKE_INSTALL_PREFIX=$opencvDirRoot/installation/OpenCV-"$cvVersion" \
            -D INSTALL_C_EXAMPLES=ON \
            -D WITH_TBB=ON \
            -D WITH_V4L=ON \
            -D OPENCV_PYTHON3_INSTALL_PATH=$opencvDirRoot/OpenCV-$cvVersion-py3/lib/python$pythonVersion/site-packages \
        -D WITH_QT=ON \
        -D WITH_OPENGL=ON \
        -D OPENCV_EXTRA_MODULES_PATH=../../opencv_contrib/modules \

make -j$(nproc)
make install

sudo sed -i 's/CONF_SWAPSIZE=1024/CONF_SWAPSIZE=100/g' /etc/dphys-swapfile
sudo /etc/init.d/dphys-swapfile stop
sudo /etc/init.d/dphys-swapfile start

echo "sudo modprobe bcm2835-v4l2" >> ~/.profile

A few ways to quickly and automatically binarize an image

For my wife’s Spell To Write and Read (SWR) homeschooling we have a bunch of scanned worksheets.  A sample of the scanned image is shown below:


As you can see its entirely readable and fine for our purposes.  However it is not gentle on our laser printer toner budget.  What we really want is the background to be white, and the foreground to be black – nothing inbetween. This process is called binarization – and scanner software often has a feature that lets you do this during scantime.

We didn’t use that feature (or maybe our software didnt support it) at scantime. As such we need to resort to postprocessing. I have a Master’s in computer graphics and vision, and everytime I use something I learned the value of that degree goes up. It therefore behoves me to use it every chance I get.

As a good computer vision student, when I think binarization my mind jumps straight to Otsu!  He came up with a great way of automatically determining a good threshold value (meaning, when we look at each pixel in the image, everything below a value turns black, all else turns white).

My first thought is to check for an easy button somewhere. In gimp, for example, I found you can load the image, click on “Image -> Mode -> Indexed” then select “Use black and white (1 bit)”. Looks ok!

Now how to automate this, given I have 60+ images? Turns out there is a threshold option in imagemagick. I could go through each image in the directory and manually threshold, but I might get the threshold wrong, and I don’t really want to train my wife on picking a threshold value. Plus I know Otsu is better!

Turns out some guy named Fred has a bunch of ImageMagick scripts, including an Otsu one. I downloaded his script and ran it, yielding the following image:


Pretty nice – just black and white.  Thanks Fred… sorry I cannot call him “Fast Freddy” since it took around 18 seconds per image.  I know we can do better! Time to dust off those computer vision skills of Master.

Here’s what I came up with using python/opencv:

import cv2
import sys
ret,imgThresh=cv2.threshold(img, 0, 255, cv2.THRESH_OTSU)
cv2.imwrite(sys.argv[2], imgThresh)

Short and sweet! And performance is way better: about 3 seconds per image.  But it looks like most of the program runtime is spent loading cv2.  Based on that assumption I decided to add a bulk processing mode:

import cv2
import sys
if len(sys.argv) == 1 or "-h" in sys.argv:
    print "Usage: %s [-inplace] image1 [image2 [image 3 ...]]"
    print " %s inImage outImage"
if "-inplace" == sys.argv[1]:
    inOut = [ (arg, arg) for arg in sys.argv[2:] ]
    inOut = [ (sys.argv[1], sys.argv[2]) ]
for inImage, outImage in inOut:
    print "Converting %s to %s" % (inImage, outImage)
    ret,imgThresh=cv2.threshold(img, 0, 255, cv2.THRESH_OTSU)
    cv2.imwrite(outImage, imgThresh)

When I run this script on the whole directory it takes an average of 2 seconds per image. Better, but longer than needed. What gives? It turns out I have all my data on a QNAP and opening, reading, and writing lots of files is not its forte. When I copy the data to my local SSD on the MAC, the cost per image is now 140ms. Much better.

Since, as often happens, I have found my assumptions totally flawed, can I vindicate Freddy? After rerunning the test it appears he is still a “steady Freddy” at about 2.7 seconds when running straight on the hard drive. Sorry Fred; opencv just beat the pants off you.