A Real-time Implementation of Rendering Light Field Imagery for Generating Point Clouds in Vision Navigation

Abstract

This dissertation develops a real-time implementation of rendering perspective and refocused imagery from a light field camera for the generation of sparse point clouds using a traditional stereo camera approach, a multiperspective approach, and two refocused ranging approaches. Unlike monocular cameras that have difficulty in extracting depth information without prior knowledge of a scene either in the form of previous images or object recognition, light field, or plenoptic, cameras can estimate depth with the introduction of additional hardware in the form of a microlens array to capture the 4-D radiance. The centers of these microlens are extracted with a calibration procedure and are used to build the radiance that contains the additional angular information that gives the plenoptic camera its ability to render multiple perspectives with a single main lens, thus improving upon the stereo or multiple camera setups that require more space and power as well as additional hardware for triggering synchronous images. This additional information for rendering multiple perspectives and refocused images greatly increases the plenoptic image size and thus the image processing time. With the end of dramatic growth in single processor performance, parallel processing has become the dominant method to decrease computing times. In response, this dissertation develops efficient methods for plenoptic image rendering algorithms for single core processor, multiple core processor, and graphics processing unit (GPU) architectures.

The multiple core processor architecture is realized through OpenMP with an Intel Core i7-4720HQ processor, and the GPU implementation uses Nvidia's CUDA API with a GeForce GTX 980M. For the perspective image rendering from the plenoptic case, OpenMP gives a 3.9x speedup from 21.8 ms with a single core to 5.6 ms on eight cores. The integral refocused image case improves its computation time with OpenMP from 699 ms for the serial architecture to 242.1 ms with eight cores for a 2.9x speedup. A 3.3x speedup is achieved with OpenMP for the FFT refocused imaging case from 717.3 ms for one core to 217.4 ms using eight cores. The perspective image rendering from the plenoptic image case has a GPU computation time of 5.6 ms and a speedup of 3.9x over the single core processing time. Computation with CUDA takes 17.3 ms to render an integral refocused image which amounts to a 40x speedup over the single core processing time. Finally, the FFT refocused imaging case results in a 14.6x speedup over the serial time with a total computation time of 49 ms.

Four methods are discussed in this dissertation for determining range using these efficiently rendered multiple perspective and refocused images: stereo ranging, multiperspective ranging, integral refocused ranging, and FFT refocused ranging. The stereo ranging approach follows similar techniques as traditional stereo algorithms through the ability of the plenoptic camera to render two virtual cameras. A multiperspective ranging approach uses an extrinsic property calibration model to rectify multiple virtual stereo pairs for a robust multiple stereo-pair ranging approach. Two depth-from-focus approaches use correlation to measure the depth through either an integral-based refocusing method or an FFT-based refocusing method to compare the alpha-refocused image with the in-focus perspective image. A ranging model then relates the alpha at the maximum correlation with the distance. During experimental testing, each of the four ranging algorithms estimates ranges to tracked features on a target using a 55 mm lens and a 135 mm lens. The 55 mm lens renders images with a field of view of 40 degrees at the cost of a maximum range of about 2 meters, while the 135 mm lens estimates improved ranges out to 6.5 meters at the cost of a field of view of only 16 degrees. Finally, this dissertation demonstrates plenoptic odometry with target tracking using each of the four ranging methods.