Computer Vision
Last updated
Last updated
Computer vision techniques enable computers to use visual data to make sense of their environment.
PX4 uses computer vision systems (primarily running on Companion Computers) in order to support the following features:
Optical Flow provides 2D velocity estimation (using a downward facing camera and a downward facing distance sensor).
Motion Capture provides 3D pose estimation using a vision system that is external to the vehicle. It is primarily used for indoor navigation.
Visual Inertial Odometry provides 3D pose and velocity estimation using an onboard vision system and IMU. It is used for navigation when global position information is absent or unreliable.
Obstacle Avoidance provides full navigation around obstacles when flying a planned path (currently missions are supported). This uses PX4/PX4-Avoidance running on a companion computer.
Collision Prevention is used to stop vehicles before they can crash into an obstacle (primarily when flying in manual modes).
:::tip The PX4 Vision Autonomy Development Kit (Holybro) is a robust and inexpensive kit for developers working with computer vision on PX4. It comes with no pre-installed software, but does include an example implementation of obstacle avoidance to demonstrate the capabilities of the platform. :::
Motion Capture (MoCap) is a technique for estimating the 3D pose (position and orientation) of a vehicle using a positioning mechanism that is external to the vehicle. MoCap systems most commonly detect motion using infrared cameras, but other types of cameras, Lidar, or Ultra Wideband (UWB) may also be used.
:::note MoCap is commonly used to navigate a vehicle in situations where GPS is absent (e.g. indoors), and provides position relative to a local coordinate system. :::
For information about MoCap see:
Visual Inertial Odometry (VIO) is used for estimating the 3D pose (position and orientation) and velocity of a moving vehicle relative to a local starting position. It is commonly used to navigate a vehicle in situations where GPS is absent (e.g. indoors) or unreliable (e.g. when flying under a bridge).
VIO uses Visual Odometry to estimate vehicle pose from visual information, combined with inertial measurements from an IMU (to correct for errors associated with rapid vehicle movement resulting in poor image capture).
:::note One difference between VIO and MoCap is that VIO cameras/IMU are vehicle-based, and additionally provide velocity information. :::
For information about configuring VIO on PX4 see:
Optical Flow provides 2D velocity estimation (using a downward facing camera and a downward facing distance sensor).
For information about optical flow see:
XTDrone - ROS + PX4 simulation environment for computer vision. The XTDrone Manual has everything you need to get started!