aiff22 / microisp Goto Github PK

This repository provides the implementation of the RAW-to-RGB mapping approach and MicroISP CNN presented in this paper. The model is trained to convert RAW Bayer data obtained directly from mobile camera sensor into photos captured with a professional medium format 102MP Fujifilm GFX100 camera, thus replacing the entire hand-crafted ISP camera pipeline. The provided pre-trained MicroISP model can be used to generate full-resolution 32MP photos from RAW (DNG) image files captured using the Sony Exmor IMX586 camera sensor directly on mobile devices.

• Python: scipy, numpy, imageio and pillow packages
• TensorFlow 2.X
• [Optioinal] Nvidia GPU + CUDA cuDNN

The model accepts the raw RGBG Bayer data coming directly from the camera sensor. The input is then grouped in 4 feature maps corresponding to each of the four RGBG color channels using the space-to-depth op. Next, this input is processed in parallel in 3 model branches corresponding to the R, G and B color channels and consisting of N residual building blocks. After applying the depth-to-space op at the end of each branch, their outputs are concatenated into the reconstructed RGB photo.

The proposed MicroISP model contains only layers supported by the Neural Networks API 1.2, and thus can run on any NNAPI-compliant AI accelerator (such as NPU, APU, DSP or GPU) available on mobile devices with Android 10 and above. The size of the MicroISP network is only 158 KB when exported for inference using the TFLite FP32 format. The model consumes around 90, 475 and 975MB of RAM when processing FullHD, 12MP and 32MP photos on mobile GPUs, respectively. Its GPU runtimes on various platforms for images of different resolutions are provided below:

python inference.py

The model will then process DNG images from the sample_RAW_photos directory and save the resulting RGB/PNG images to the sample_visual_results folder.

pretrained_weights/   -   the folder with the provided pre-trained MicroISP model
sample_RAW_photos/   -   the folder with sample RAW/DNG images from the Fujifilm UltraISP Dataset
model.py   -   TensorFlow MicroISP implementation
inference.py   -   applying the pre-trained model to full-resolution test images
export_to_tflite.py   -   model export to TensorFlow Lite format for on-device deployment

Copyright (C) 2024 Andrey Ignatov. All rights reserved.

Licensed under the CC BY-NC-SA 4.0 (Attribution-NonCommercial-ShareAlike 4.0 International).

The code is released for academic research use only.

@inproceedings{ignatov2022microisp,
  title={MicroISP: Processing 32MP Photos on Mobile Devices with Deep Learning},
  author={Ignatov, Andrey and Sycheva, Anastasia and Timofte, Radu and Tseng, Yu and Xu, Yu-Syuan and Yu, Po-Hsiang and Chiang, Cheng-Ming and Kuo, Hsien-Kai and Chen, Min-Hung and Cheng, Chia-Ming and others},
  booktitle={European Conference on Computer Vision},
  pages={729--746},
  year={2022},
  organization={Springer}
}

Please contact Andrey Ignatov ([email protected]) for more information