A Novel Tracking Algorithm Using Thermal and Optical Cameras Fused with mmWave Radar Sensor Data

Tracking of moving objects in pan-tilt-zoom (PTZ) cameras is a critical feature for automated surveillance applications. The PTZ auto tracking function enables the camera to control its electric motor mechanisms to adjust the course of the optical sensor field of view (FOV) to track moving objects automatically. In this paper, a proposed novel object tracking method based on the fusion of thermal, optical sensor and millimeter-wave data is implemented. The implementation of the proposed method is performed on a pan-tilt setup, which ultimately tracks the object of interest based on thermal data while keeping the optical image resolution the same as the thermal image resolution and provides valuable distance measurements to the target based on the millimeter-wave radar readings. The proposed method offers improvements over conventional optical sensor tracking by adding an extra layer to object tracking through additional thermal and millimeter-wave sensor data, which in turn provides a surplus of information of the object of interest that is being tracked. The proposed method compares favorably with commercially available surveillance systems.