Background

Image-related tasks such as object recognition and tracking have been a very popular application of Machine Learning (ML). Most of this has been done either off-line using powerful GPUs, or (to a limited extent) using specialized accelerators built for specific devices (e.g., smartphones). In this project, we would like to explore the possibilities for using low-power reprogrammable computing (FPGAs) in independent mobile deployments (drones). We envision eventual applications to disaster response (e.g., looking for survivors), wildlife management (e.g., tracking wildlife), farming (e.g., tracking farm animals populations), etc.

Main Objectives

The overall objective is to assess the viability, and determine the constraints, of airborne

autonomous object tracking applications by developing a proof-of-concept drone-hosted FPGA-based video processing platform (see Deliverables). The project should promote knowledge and expertise in the design of FPGAs for object tracking in particular and machine learning applications in general. To this end, all code and designs produced during the project will be released using the open source MIT license.

Main Deliverables

The project will be expected to deliver a remotely-controlled flying drone that hosts a small video camera and an FPGA (as powerful as feasible given the power constraints), plus any software / hardware source code / scripts required to program / use the drone.

Within the drone, the video stream should be fed to the FPGA, which should host a deep learning model (likely aCNN) to identify and track objects in the camera's field of view, and

both the video stream and the FPGA output (bounding boxes, classification results, etc.) should be transmitted to a ground-based host (e.g., laptop or phone) and displayed to the user (as, e.g.bounding boxes superimposed on the image)