Optical signal processing with illumination-encoded filters

Recently, computer vision researchers have shown that orthogonal functions and computational techniques from the signal processing framework can be mapped directly into the scene using projector–camera systems. These scene-space signal processing algorithms are achieved with illumination-encoded functions as primitives and computations derived from surface reflection models. Some examples of this new optical approach include convolution filtering and aliasing-canceling filter banks.In this paper we present computational techniques for realizing fundamental elements of the signal processing framework in the 3D scene domain. The motivation for optical computation directly in the scene is to avoid information loss when the rich 3D scene is reduced to an image. The computations are at subpixel resolution because they are performed within each camera sensor. Scene-space filtering applies 2D operators to locally planar 3D surfaces based on the optical coupling of the scene surface topology with projector–camera devices. Signal processing issues such as sampling geometry, dynamic range, mathematical operators, and resolution are addressed. We report a novel subpixel point correspondence technique for accurate camera sensor footprint localization in general scenes. It is a parallelizable optical clipping algorithm related to the polygon clipping and box filter anti-aliasing algorithms used in computer graphics. It replaces the regular coordinates of an image with a local surface parametrization in projector coordinates. The result is subpixel resolution filter responses. We provide experiments and results to evaluate the performance of our scene-space filtering techniques with both planar and non-planar objects.

Research highlights► Finite impulse response filters can be embedded in illumination. ► In projector–camera systems each sensor pixel footprint can be computed. ► Subpixel filtering of non-planar surface textures using dense light fields. ► Optically-based geometry clipping algorithm for projector–camera systems. ► Structured light patterns can be used for geometric and spectral computation.