Model-Based Image Signal Processors via Learnable Dictionaries

by   Marcos V. Conde, et al.

Digital cameras transform sensor RAW readings into RGB images by means of their Image Signal Processor (ISP). Computational photography tasks such as image denoising and colour constancy are commonly performed in the RAW domain, in part due to the inherent hardware design, but also due to the appealing simplicity of noise statistics that result from the direct sensor readings. Despite this, the availability of RAW images is limited in comparison with the abundance and diversity of available RGB data. Recent approaches have attempted to bridge this gap by estimating the RGB to RAW mapping: handcrafted model-based methods that are interpretable and controllable usually require manual parameter fine-tuning, while end-to-end learnable neural networks require large amounts of training data, at times with complex training procedures, and generally lack interpretability and parametric control. Towards addressing these existing limitations, we present a novel hybrid model-based and data-driven ISP that builds on canonical ISP operations and is both learnable and interpretable. Our proposed invertible model, capable of bidirectional mapping between RAW and RGB domains, employs end-to-end learning of rich parameter representations, i.e. dictionaries, that are free from direct parametric supervision and additionally enable simple and plausible data augmentation. We evidence the value of our data generation process by extensive experiments under both RAW image reconstruction and RAW image denoising tasks, obtaining state-of-the-art performance in both. Additionally, we show that our ISP can learn meaningful mappings from few data samples, and that denoising models trained with our dictionary-based data augmentation are competitive despite having only few or zero ground-truth labels.


page 4

page 6

page 7


Learning Raw Image Denoising with Bayer Pattern Unification and Bayer Preserving Augmentation

In this paper, we present new data pre-processing and augmentation techn...

Modeling sRGB Camera Noise with Normalizing Flows

Noise modeling and reduction are fundamental tasks in low-level computer...

A Physics-based Noise Formation Model for Extreme Low-light Raw Denoising

Lacking rich and realistic data, learned single image denoising algorith...

Joint demosaicing and denoising by overfitting of bursts of raw images

Demosaicking and denoising are the first steps of any camera image proce...

Overexposure Mask Fusion: Generalizable Reverse ISP Multi-Step Refinement

With the advent of deep learning methods replacing the ISP in transformi...

Residual Contrastive Learning for Joint Demosaicking and Denoising

The breakthrough of contrastive learning (CL) has fueled the recent succ...

Pigmento: Pigment-Based Image Analysis and Editing

The colorful appearance of a physical painting is determined by the dist...

Please sign up or login with your details

Forgot password? Click here to reset