mrgiovanni / modelsgenesis Goto Github PK

thanks for your excellent work! there are some question

1. we all know that voxel "Spacing" is important when sampling Medical Image Data, which spacing in your training dataset, and test dataset?
2. the scale (32, 64) per patient, such as "32 means I extracted 32 sub-volumes per patient, 64 means 64 sub-volumes per patient", why donnt you sampling data on-demand when pick one patient for much more sub-volumes.
3. what is means HU_thred and len_depth in function infinite_generator_from_one_volume.

Hi Zongwei,

Thanks for your great work. Your implemention is very clearly 👍
I'm trying to pretrain on mine own dataset, but i don't know how much training data been used when you pretrain (patch level). Could you tell me this detail ?

Thanks,
Jing

Hi,
can you please provide the test code which i can run for my own data to see the performance.

Thanks!

Thank you for all the hard work put into this project and for making the pretrained models and code available for public use! Would it be possible to also provide the code and pretrained weights for the 2D models, in particular Genesis Chest CT 2D, as mentioned in the paper?

Hello, I have two questions.
1.I would like to ask if I want to use your pre-training weight, do I have to use your model as my feature extraction model?
2.Can you provide a link to download your pre-training weights?

I think the role of skip-connection is rather disturbing in your paper where reconstruction tasks are usually performed, what do you think?

Using skip connection in reconstruction often transfers a lot of information intact in the initial layer. However, you put a strong distortion on input, probably which is why the cheating problem does not occur.

what do you think about it? Does skip-connection play an important role in your reconstruction task?

I want to try your code on 2D. Is the network I am training using UNET?

Hello,
I’m using some image transformations of Models Genesis (local pixel shuffling, non-linear transformation, in- and out-painting) to train an autoencodeur. As I’m working with 2D images, I adapted the code available to process 2D images. The transformations appear to work successfully.
However, they’re really time-consuming : on average 20 seconds per batch, with a batch size of 128 2D images. Because of this, training is impossible as it is far too slow. I tried decreasing the probability of each transformation (from 0.5 to 0.25), decreasing the number of blocks in pixel shuffling and using less transformations, which helped, but the process remains too slow.
Did you face the same issue during training? If yes, how did you overcome this issue?
Thank you !

hi, Giovanni, I just re-implemented the MG-2D on my dataset and met with some other bug for cpu explosion.

When I used MG for pre-training, I found that the utilization rate of the CPU would increase strangely, but the utilization rate of the GPU fluctuated greatly, and the buzzer of the machine PC would alarm soon. After several rounds of debugging, I found that it was this piece of code that caused the CPU utilization to sharply increase to 4000%+!!!!

Maybe this is a special case in my machine, and it has no reference value.
Thanks. :-)

Hello Zongwei, first of all thank you for your work. It appears to be a great idea and it is very clearly explained.
I was wondering if a version of your work will be developed for PET modality ?
Do you have some advises to train a model for PET modality using your approach (especially for the patch size) ?
I am working on a dataset with 128x128x(N slices) PET images.
Do you think that patches of size 64x64x64 would help a network (VNet for example) to learn from 128x128x(N slices) PET images ?
For how many epochs did you train your models ? (I think I saw 10 000 in one of your answers ?, before any early stop) ?
I also want to use your VNet Chest-CT implementation.
Do you have any additional advice regarding the fine-tuning ? (fine-tuning only the last layers, cubes of size 64 ?)

Thank you very much for your answers !

Hi!
Thank you for sharing this code!
I have a newbie question,
We are trying to run it with the weights: genesis_nnunet_luna16_006.model on the Task004_Hippocampus data,
but we get the print : "Training from scratch", meaning we are not using the weights.
I checked and the condition: (model_dict[key].shape == pretrained_dict[key].shape) is False.

It would be great if you are familiar with the reason and maybe can clarify it for us,
and tell us if what we are trying to do is even possible.

Thanks in advance!

Thanks for your excellent paper and repo,
As you mentioned in the infinite_generator_3D code, you used folds 7,8,9 as test data, did you compute your reported cpm (0.971 in the paper) just on these folds?
thanks a lot.

Tks for the great job. But from my side, i have serveral questions

1. In the paper, it doesn't compare result a simple auto-encoder without your transforms, so maybe an autoencoder can get comparable result ?
2. The proposed transforms can also be applied as data augmentations during training, so how does this perform compared with the pretrained model ?

I tried to download the LUNA16 dataset from zenodo.org, there are 10 files and I checked the md5. Everything is correct, but the Archive manager prompts "An error occurred while loading the archive". Is there anything I've done wrong? I have installed 7-zip extension.

Hi Zongwei,

This is a good work!
I have a question regarding the downstream tasks. Do you resize or rescale or resample the LUNA and MRI 3D volumes and their segmentation masks for the downstream tasks?

Thanks,
Ananya

Hi !
First, thank you for making your incredible work available. Your results are outstanding and the code is pretty easy to understand.
In order to use your work in another application that I am investigating I wanted first to replicate your results on the BraTs dataset.
I have downloaded and preprocessed the data : resizing the images to 64x64x64 and normalizing between 0 and 1.
However, when finetuning Genesis CT I don't manage to get anything close to your results.

Is there something I am missing ? The whole image has to be fed to the network or split in patches ? Also in the code example given in the keras folder num_classes is set to 2 for segmentation which I am not sure to understand as the convolution will create (2, 64, 64, 64) sized mask.

Thank you for your help,
Regards,
Camille

The size of the cube in your pre-trained code is 64X64. But I want to use your pre-trained model to segmentation the nodule , but the DICOM image size is 512X512, what should I do?

I am recreating your article, can the network input be changed to 128128128 without changing other parameters?

Hi Zongwei:
Thanks for the excellent job.
I met with some difficulties when trying to learn Models Genesis from my own unlabeled data.

1. Lines 165 - 181 in infinite_generator_3D.py, I guess the intent of the code is to determine whether the crop_window is lung part. But I don't understand why it can make it.
2. For my project, segmentations of gastrointestinal stromal tumors, i think i should modify this part ( including the value of config.HU_thred, config.lung_max and more, i guess), could you please give me some advise?

Best
lzp

Hi!

Do you have any pretrained models for 2D x-ray data you could possibly share?

Thanks in advance
Marian

Hi Zongwei:
Thank you for the excellent work.

I have some questions and any help would be appreciated.

How should I determine the input patch size? I mean, is there any particularity about whether the input patch is large or small. If my target is lung tissue such as pulmonary blood vessels, should I take a smaller patch size?

I am working on the segmentation of pulmonary vessels. Some previous work tried the Gaussian pyramid for multi-scale processing, and then used a 5x5x5 patch size into a double-layer network. Here, since your model has multiple layers of downsampling, does this mean that my patch can be correspondingly larger than 5 and no longer require multi-scale preprocessing ?

Sincerely looking forward to your reply.

Best Wishes!
zlin

Thanks for the great repo. One question about the pretrained model. What is the config of the pretrained model here. Especially, is the train_fold the same as this?

Hello,

First thanks for your amazing work. I am training my own genesis network Using lung CT scans. The patches generated includes a reasonable amount of Lung part while when training with Genesis_Chest_CT, the output of the network will learn one single value for all pixels, any idea why this happened?

Thanks,

Tian

hello, how do you think if i add GAN model into this framework, will I have chance to get more discrimitive feature from this autoencoder?

demo in file "ModelsGenesis/tree/master/pytorch"

self.out_glb_avg_pool = F.avg_pool3d(self.base_out, kernel_size=self.base_out.size()[2:]).view(self.base_out.size( [0],-1)

view(self.base_out.size( [0],-1) ???

Hello! So interested in your work.@MrGiovanni
I don't quite understand what the parameter “len_depth” is for in infinite_generator_3D.py.
Could you please tell me more about that?

    for d in range(config.input_deps):
        for i in range(config.input_rows):
            for j in range(config.input_cols):
                for k in range(config.len_depth):
                    if crop_window[i, j, d+k] >= config.HU_thred:
                        t_img[i, j, d] = crop_window[i, j, d+k]
                        d_img[i, j, d] = k
                        break
                    if k == config.len_depth-1:
                        d_img[i, j, d] = k
                        
    d_img = d_img.astype('float32')
    d_img /= (config.len_depth - 1)
    d_img = 1.0 - d_img

Hi Zongwei,

Thanks for sharing such a great work.
I have tried to reproduce the results shown in the Tab. 2 of the paper, however, I can't get the comparable scores.
Could you release the experimental setting (preprocessing, loss functions, optimizer ...) of those experiments (NCC, NCS, ECC, LCS, BMS)?

Thanks for the help.

Best,
Yu-Cheng

error happens with:
output = activate(final_convolution, axis=1)
cause its output is not a keras layer.
suggest to replace with this line:
output = keras.layers.Softmax(axis = 1)(final_convolution)
it works!!!

By the way, good job 纵苇! Your pretrained model improves my project performance.

This is a wonderful work and has great impact on 3D medical imaging deep learning.

I am trying to fine-tune Models Genesis to classify HCC pathological biomark (negative OR positive) based on liver CT data.

I'm fairly new in this field and there is a question about data pre-processing. I have abdomen CT image datas (DICOM files) and HCC ROI segmentation result (3D nii file, which value is 0 or 1 to identify the ROI region ), i don't konw how to process them to put into the model？ Merge them into one npy file?

Could you please list the network architecture of 3D Models Genesis? I plan to employ the encoder as a fixed feature extractor.

Thank you for your help,
Best Regards.

Shanhu

I don't find the implement of permute_dimensions() function in BilinearUpSampling.py

Thank you for your amazing work. I am implementing your method on my own dataset. I have questions regarding the PyTorch version of in-painting and out-painting. It looks like you replace the pixels within the window with random values in both transformations. For example, in in-painting, you use:

x[:, 
   noise_x:noise_x+block_noise_size_x, 
   noise_y:noise_y+block_noise_size_y, 
   noise_z:noise_z+block_noise_size_z] = np.random.rand(block_noise_size_x, 
                                                        block_noise_size_y, 
                                                        block_noise_size_z, ) * 1.0

which assigns random values to the block, instead of:

x[:, 
   noise_x:noise_x+block_noise_size_x, 
   noise_y:noise_y+block_noise_size_y, 
   noise_z:noise_z+block_noise_size_z] = np.random.rand()

In this case the block is replaced with a single random value, which is the case according to your paper:

We then assign a random value to all pixels outside the window while retaining the original intensities for the pixels within. As for in-painting, we retain the original intensities outside the window and replace the intensity values of the inner pixels with a constant value.

This is also the case suggested by fig.1 in the paper. Which one is correct? Also I don't quite understand why the random noise is multiplied with 1.0 in both in-painting and out-painting. Since the np.random.rand function generates float numbers, the multiplication seems unnecessary.

Hi there,
I would really like to read some additional details about nnU-Net and models genesis. So far you seem to have taken the first place in Task03, but it is difficult so see whether that is a significant result (given that I am getting some variation when running the same training several times which will also translate into different test set performances). Overall, your submission on the Decathlon is below ours, indicating that the pretraining may not be beneficial on all tasks. This makes is difficult to really estimate the impact of your pre-training strategy.

Specifically, I would be interested in how much your pretrained models help in semi-supervised learning. Say you take all non-LiTS/Task03 (Task03 is essentially LiTS) datasets with livers in them (BCV Abdomen, KiTS, Pancreas (?), ...) and run models genesis on them for pretraining, how well does your pretrained nnU-Net perform when fine tuned on 10, 20, 50 etc LiTS cases for the LiTS task? Can you beat the nnU-Net baseline by a significant margin if you use all these additional datasets for pretraining?

Best,
Fabian

Thanks so much for sharing the code , and I have some questions.
1.could you give more information about the version of packages(requirements.txt),I cant reconstruct
the same env with these info.
2. When I using tf=1.14 and keras=2.3.1 . There occurs an error "“Tensor” object hs no attribute "assign""

Thanks for your work. Is it possible to give pytorch version in the future?

Hello @MrGiovanni, thanks a lot for your work! It is very cool and interesting.

I am working on image classification (disease positive vs disease negative) in CT of head and neck cancer. The problem is that, as you state, the genesis chest model was trained on the -1000 HU to 1000 HU, and my interesting range would be much less (approx -50 HU to 200 HU). My dataset size is rather limited (less than 600 patients) so that is why I think a pre-trained model is a good approach but I was wondering if you have any advice on this problem or any data normalization technique. Thanks a lot in advance,

Best,

Agustina

Hello, I want to use my own data to refine your trained model according to the instructions you gave on GitHub, but Training from scratch appears when it is running. What is the reason for this? Looking forward to your reply!

Hi,

Thank you for releasing your code, I very much appreciate the extensive experiments in your paper.

I'd like to compare your self-supervision training procedure against a different self-supervision task on some different 2D medical imaging data. Would you be able to release the code you used to transform your 2D data when making comparisons against ImageNet (just the transformation code would be enough for me, not necessarily the whole 2D training procedure)? I imagine the code is pretty similar to your 3D code, but it would be nice to get the right parameters from you.

Since training on 3D is rather resource intensive, do you think that 2D pretraining performance is a good indicator to compare among different self-supervision tasks?

Hello
The results showed Models Genesis top any 2D approaches, including ImageNet models. So I‘d like to try your pretrained weight of 2D.

Hi,

I find your work very interesting. I have tried to open a request for the pretrained weights, but it seems the link is down.

Is there any h5 files which I can download? Or I must pre-train the model by myself.

when I load the pre-trained weights, I got the ----ValueError: axes don't match array
anyone has the same problem? it's the problem of the version of keras?

If I do classification task.

in TargetNet
self.base_model(x)
self.base_out = self.base_model.out512

Why not modify UNet3D.py

    if self.do_seg:
        self.out_up_256 = self.up_tr256(self.out512,self.skip_out256)
        self.out_up_128 = self.up_tr128(self.out_up_256, self.skip_out128)
        self.out_up_64 = self.up_tr64(self.out_up_128, self.skip_out64)
        self.out = self.out_tr(self.out_up_64)

        return self.out
    else:
        return self.out512

Doing so will save a lot of memory and FLOPS.

This is a wonderful work, and there are some questions about data perocessing.
Question 1: When preparing our own image data, we always scale the data between -1 and 1. And I see the author process the data into 0 and 1. Is it necessary to scale the input data to 0 and 1?

Question 2: The author provide a UNet3D based model, and the number of the model class is 1. If we want use the author's pretrained model with 4 class (or other class), is it worked by loading the network layer without "module.out_tr.final_conv.weight", and just fine-tune the model? Do we need fixing the model layer and with a two stage training?

Thanks and looking forward your reply!

Hi Zongwei,

Thanks for the release of the Pytorch version.

However, there is some difference between the weights of Keras and Pytorch.
The network utilizes the UpSampling3D as the up convolution layer in the Keras version, therefore, there is no trainable weight in those layers.
In contrast, the Pytorch model adopts the ConvTranspose3d, which contains trainable weights.

Which version is recommended for us if we would like to reproduce the excellent performance in your accepted paper?

Best,
Yu-Cheng

Such as mrbrains18 dataset

Hello! When looking at your Readme.MD you mention that the infinite_cube_generator.py generates N cubes from each patient, N is the scale correct?

However, when using your own 3D extraced cubes from luna, when opening the file from a single patient, there are a lot more than 32 cubes. For example bat_32_s_64x64x32_0.npy has dimensions of (2848, 64, 64, 32) which are 2848 cubes. Could you explain to me what scale represents then in your code and what you use scale for when generating cubes? Did you extract the cubes you provided with a much larger scale?

Also, could you please explain the difference between the purpose of input_rows and crop_rows?

Hi! Thank you so much for this repository. This looks very promising and interesting.
When I tried running the sample code I got the following error

Would you be able to please help me with it?

Appreciate the help!

thanks!

It seems that the pretrained weight .h5 is for Keras. Could you please provide .pth pretrained weight for Pytorch?

Thank you