Sample selection and reconstruction for array-based multispectral imaging

Abstract

In this work we address the problem of acquisition of multispectral images in a sampled form and the subsequent processing of the acquired signal. The problem is relevant in the context of color imaging in digital cameras, and increasingly, in the field of hyperspectral imaging as applied to remote-sensing and target recognition. The scope of this work encompasses a broad swath across image processing problems and includes: image acquisition, in the problem of optimally selecting sampling rates and patterns of multiple channels; image reconstruction, in the reconstruction of the sparsely sampled data; image restoration, in obtaining an estimate of the true scene from noisy data; and finally, image enhancement and representation, in the problem of presenting the reconstructed image in a color-space that allows for transformations that achieve best perceived quality.

Acquisition of multispectral images in the simplest form entails either the use of multiple sensor arrays to sample separate spectral bands in a scene, or the use of a single sensor array with a mechanism that switches overlaying band-pass filters. Due to the nature of the acquisition process, both these methods suffer from shortcomings in terms of weight, cost, time of acquisition, etc. An alternative scheme widely in use only uses one sensor array to sample multiple bands. An array of filters, referred to as a \textit{mosaic}, is overlaid on the sensor array such that only one color is sampled at a given pixel location. The full color image is obtained during a subsequent reconstruction step commonly referred to as \textit{demosaicking}. This scheme offers advantages in terms of cost, weight, mechanical robustness and the elimination of the related post-processing step since registration in this case is exact.

Three main issues need to be addressed in such a scheme, viz., the shape and arrangement of the sampling pattern, selection of the sensitivities of the spectral filters, and the design of the reconstruction algorithm. Each of the above problems is contingent on multiple factors. Sensor sampling patterns are constrained by the limitations of electronic devices and manufacturing processes, spectral sensitivities are affected by the material properties of the colors painted on the array to form filters, and the reconstruction methods are limited by computational resources.

In this research, we address the above problems from a signal processing perspective and attempt to develop parametric algorithms that can accommodate external limitations and constraints. We have developed methodologies for the selection of optimal sampling patterns that will allow for ordered, repeated array blocks. In addition we have developed an algorithm for demosaicking of CFA data based on Bayesian techniques. We have also proposed a formulation for the selection of optimal spectral sensitivities for individual color filters.