A vision-based strategy for autonomous aerial refueling tasks

•A monocular sensing solution for probe and drogue aerial refueling is presented.•Tracking the drogue without the need for modifications to the tanker hardware.•The core of the system is a tracking algorithm based on direct image registration.•Tested using images from real refueling hardware from a unique test environment.•Robust position estimations under rapid motions and occlusions of the drogue.

Autonomous aerial refueling is a key enabling technology for both manned and unmanned aircraft where extended flight duration or range are required. The results presented within this paper offer one potential vision-based sensing solution, together with a unique test environment. A hierarchical visual tracking algorithm based on direct methods is proposed and developed for the purposes of tracking a drogue during the capture stage of autonomous aerial refueling, and of estimating its 3D position. Intended to be applied in real time to a video stream from a single monocular camera mounted on the receiver aircraft, the algorithm is shown to be highly robust, and capable of tracking large, rapid drogue motions within the frame of reference. The proposed strategy has been tested using a complex robotic testbed and with actual flight hardware consisting of a full size probe and drogue. Results show that the vision tracking algorithm can detect and track the drogue at real-time frame rates of more than thirty frames per second, obtaining a robust position estimation even with strong motions and multiple occlusions of the drogue.