Efficient and high performance FPGA-based rectification architecture for stereo vision

This paper proposes an efficient and high performance rectification architecture to be used as a preprocessing module in a complete stereo vision system before the matching correspondence calculus. A complete rectification process is implemented in order to remove the radial and tangential distortion effects due to lenses and to align the left and right raw images acquired by a stereo camera for the epipolar constraining. Thus, the epipolar lines are made collinear with each other and with the image scanning lines in order to reduce the complexity of the matching problem to a one-dimension correspondence search.The image transformation operations required by the rectification process are computed as matrix calculus through a pipelined and efficient hardware design. Unlike the memory mapped rectification function implementations, the proposed solution does not require any external memory block for the storage of pre-computed rectification maps. Moreover, conforming to the camera model adopted by the Stereo MATLAB Calibration Toolbox which is renowned as the most widely used software toolset for estimating the calibration parameters of a stereo camera, the proposed rectification architecture is a ready-to-use hardware solution to be used in stereo vision real-time embedded systems after calibrating the employed stereo camera following the MATLAB Calibration Toolbox procedure.When implemented in a Xilinx XC4VLX60-12ff1148 FPGA chip, the proposed circuit rectifies 640 × 480 and 1280 × 720 stereo images at a frame rate of 367 fps and 120 fps, respectively. The proposed fully pipelined solution uses an efficient raw image buffer system which is opportunely sized in order to store the minimum number of image rows able to guarantee the synchronization between the image buffering and the rectification elaboration without any interruption of the pipelined processing flow. When the proposed rectification system was used for processing the stereo images acquired by the Point Grey Research Bumblebee BB2-03S2 stereo camera, just 32 BRAM blocks were necessary to implement the raw image buffer; thus, after a latency of 136 us (15,387 clock cycles), a continuous flow of left and right rectified image pixels is guaranteed in output, for each inputted left and right couple of raw image pixels, at each clock cycle.When compared to the other implementations present in literature, the proposed solution offers the advantage of not using any external memory with respect to the memory-mapped rectification solutions while offering a more efficient and complete solution reaching the highest speed performance with respect to the on-the-fly computed rectification implementations present in literature.