• About the Project
• Requirements
• Network structure
• Usage
  • Train
  • Test
• Acknowledgements
• Sources
• Citing this work
• Contact

This repository contains the code from our Light Field Microscopy project. LFMNet is a neural network that reconstructs a 3D confocal volume given a 4D LF image, it has been tested with the Mice Brain LFM-confocal public dataset. LFMNet is fully convolutional, it can be trained with LFs of any size (for example patches) and then tested on other sizes. In our case it takes 20ms to reconstruct a volume with 1287x1287x64 voxels.

The repo is based on Python 3.7.4 and Pytorch 1.4, see requirements.txt for more details. The dataset used for this network can be found here, but it works with any LF image that has a corresponding 3D volume.

The paradigm behind this network is that the input contains a group of microlenses and a neighborhood around them, and reconstructs the 3D volume behind the central microlenses. LFMNet has as an initial layer a conv4d, that ensures a fully convolutional network, this first layers traverses every lenslet, and grabs a neighborhood (9 lenses in our case) around. Then the output is converted to a 2D image with the number of channels equal to the number of depths to reconstruct. Lastly, this tensor goes into a U-net¹, which finishes up the feature extraction and 3D reconstution.

A tensor with shape 1,Ax,Ay,Sx,Sy, where A are the angular dimensions and S the spatial dimensions. In our case the input tensor is 1,33,33,39,39.

A tensor with shape nD,AxSx,AySy, where nD are the number of depths to reconstruct. In our case the output tensor is 64,1287,1287.

The training main file is mainTrain.py:

python3 mainTrain.py --epochs 1000 --valEvery 0.25 --imagesToUse 0 1 2 3 4 5 --GPUs 0 --batchSize 64 --validationSplit 0.1 --biasVal 0.1 --learningRate 0.005 --useBias True --useSkipCon False --fovInput 9 --neighShape 3 --useShallowUnet True --ths 0.03 --datasetPath "BrainLFMConfocalDataset/Brain_40x_64Depths_362imgs.h5" --outputPath, nargs='? "runs/" --outputPrefix "" --checkpointPath ""

And mainEval.py the testing file:

python3 mainEval.py --GPUs 0 --datasetPath "Brain_40x_64Depths_362imgs.h5" --outputPath "runs/" --outputPrefix "" --checkpointPath, "my_path/" --checkpointFileName, "checkpoint_" --writeVolsToH5 0 --writeToTB 1

• Computer Vision Group, University of Bern
• Theodor Kocher Institute, University of Bern
• Microscopy Imaging Center, University of Bern
• Computational Imaging and Inverse Problems, University of Munich

1. Ronneberger, Olaf and Fischer, Philipp and Brox, Thomas. "U-Net: Convolutional Networks for Biomedical Image Segmentation" MICCAI 2015

Josue Page - [email protected] Project Link: https://github.com/pvjosue/LFMNet

@article{9488315,
  author={Vizcaíno, Josué Page and Saltarin, Federico and Belyaev, Yury and Lyck, Ruth and Lasser, Tobias and Favaro, Paolo},
  journal={IEEE Transactions on Computational Imaging}, 
  title={Learning to Reconstruct Confocal Microscopy Stacks From Single Light Field Images}, 
  year={2021},
  volume={7},
  number={},
  pages={775-788},
  doi={10.1109/TCI.2021.3097611}}