RTLViT: real-time lensless reconstruction with a lightweight vision transformer

Abstract

Mask-based lensless imagers capture measurements using simple, compact hardware and recover images using a reconstruction algorithm. The reconstruction algorithm affects the image quality, inference speed, and computational requirements of the imaging system, often trading off image quality against computational efficiency. Advancing toward integrated lensless imagers, where encoding and reconstruction occur within a single device (e.g., a mobile phone), requires a fast, high-quality, and practical reconstruction method. We introduce the Real-Time Lensless Vision Transformer (RTLViT), a lightweight, purely data-driven reconstruction architecture for high-quality and real-time lensless reconstruction. With only 1.09 million learnable parameters, RTLViT provides higher-quality reconstructions than both real-time baselines and substantially larger attention-based architectures, including improvements of up to 3.46 dB in peak signal-to-noise ratio over architectures with $15 \times$ more learnable parameters. We demonstrate that RTLViT maintains high-quality reconstructions across a wide range of training dataset sizes and two different mask designs (lenslets and a diffuser). Finally, we implement real-time reconstruction with RTLViT on laptops and smartphones, supporting future integrated lensless imagers for real-time applications.