marco-rudolph / differnet Goto Github PK

Hi, there,

I found in the code, the evaluation takes two steps

1. calculate z = model(inputs), loss and anomaly_score
2. use export_gradient_maps to get the gradient map for locationzation.

It seem that the inputs used in the two cases are slightly different:

1. in the first case, the inputs are just the original images after reizing
2. in the second case, there is an additional step to add fixed rotations according to code in https://github.com/marco-rudolph/differnet/blob/master/multi_transform_loader.py#L49 since we set testloader.dataset.get_fixed = True in https://github.com/marco-rudolph/differnet/blob/master/localization.py#L38

The consequence is that the data used to calculate anomaly_socre are not averaged out among different rotations while the gradients are. Is this a problem?
Shouldn't we be consistent and add the rotation tranformation to even the first step, ie, set testloader.dataset.get_fixed = True always? Maybe there is a reason to do this differently by design?

Thank you very much

when I trained the net on my own dataset, the test_loss is super large, I do not know is right or not.
Can you give me some advice?

Compute loss and scores on test set: Epoch:1 test_loss:30636857344.0000 AUROC: last:0.9935 max:0.9935 epoch_max:1

How to judge whether the prediction result is OK or ng，and where is the threshold set？

@marco-rudolph Thank you for your work.
I would like to take the model then testing with an image, and show the score. How can I do that?

@marco-rudolph
Hi, Thank you for your nice work!
I want to know that, can your work of differnet or cs-flow can be used to find small exceptional points in X-ray images, which have 3 channels with same pixel data?
First, I want to introduce the backwark. I want to find anomalies of food images which were taken by X-ray tools. These images have 3 channels, but the pixel data in 3 channels are same. And the anomalies in images, such as stainless steel/glass/Al/wire, will block the passage of light produces by X-ray, and produces different texture with normal food.
But the anomalies points in images are very small, because the radius of anomalies materials are very small, some only about 7mm. And I wonder, these small points can be found totally by the differnet or cs-flow?
Thanks for your attention!

When I run the program, get following issue
\functional.py:942: UserWarning: Argument interpolation should be of type InterpolationMode instead of int. Please, use InterpolationMode enum.
warnings.warn(

Hi there,
Can you share the train and test split for the magnetic tile dataset?
The dataset from [13] does not provide this.
Evaluation on different split generates different results.

Hi,

 Can you submit the test code to predict the image result according to the trained model? 
 Thanks!

Hey, thanks for a great job, but this is new to me. So what I want to ask is how do I test the weights that I've trained.

Hi, dear Marco. I have a quesion: when I use this code to train , the train losses from positive to negative , is it right ? I'm confused. Waiting for your reply.

Best Regards!

Is it normal for the training loss to be less than 0?

Hello,

Do you have some information about trained model inference FPS on MVTec AD dataset? Can you provide information about your GPU?

As mentioned in the paper, the performance is also very good on small sample data.
In the experiment, used the 16-shots setting. Did it only use 16 good samples during training? Or is there only 16 good samples in the test set?

Hi @marco-rudolph , thank you for sharing your work.
I have a couple questions while trying to replicate your work on the MVTec-AD dataset.
On the paper is claimed to train during 192 epochs. Is that 24 epochs x 8 subepochs?
Also do you have a code snippet to generate the Figure 6 histogram?

Best regards,
Bruno

Hi,
Thank you very much for your work.
Can it be possible for this code to process only one channel of images?

Hi, Marco!

I want to train a model for defect detection on several kinds of surfaces/objects (one model - several surfaces).
Instruction from the docs:

Set the variables dataset_path and class_name in config.py to run experiments on a dataset of your choice.

1. Can I set class_name equal to a list of classes or something else?
2. Would you tell me how to train a model for several classes of surfaces/objects?

From the code I don't see any module that make inference by feeding the model with new images.
Could I have some sample code?

when set grad_map_viz to be True, I got a mistake: rotate() got an unexpected keyword argument 'reshape'.
could you please fix it!!! thanks a lot for you work!!!

I have read the article and wanted to try this approach with the MVTec dataset. However, I could not be able to understand how to test it with the pre-trained model. I tried to run main.py and it started training initially but what I wanted is testing with the pre-trained model. How can I do this?

During training, the loss might become negative. What does this situation mean?

Hi，
How is θ given? Is it artificially set or calculated? If it is calculated, how is it calculated?
Looking forward to your reply.

Hi!
When I ran main.py I got runtime error:

py main.py
Traceback (most recent call last):
File "C:\Projects\bricky\defect_detection\differnet\main.py", line 10, in
train_set, test_set = load_datasets(c.dataset_path, c.class_name)
File "C:\Projects\bricky\defect_detection\differnet\utils.py", line 104, in load_datasets
trainset = ImageFolderMultiTransform(data_dir_train, transform=transform_train, n_transforms=c.n_transforms)
File "C:\Projects\bricky\defect_detection\differnet\multi_transform_loader.py", line 67, in init
super(ImageFolderMultiTransform, self).init(root, loader, IMG_EXTENSIONS,
File "C:\Projects\bricky\defect_detection\differnet\multi_transform_loader.py", line 34, in init
classes, class_to_idx = self._find_classes(self.root)
AttributeError: 'ImageFolderMultiTransform' object has no attribute '_find_classes'

I think that this error causes due the typo in "differnet\multi_transform_loader.py" (line 34). DatasetFolder object haven't _find_classes method, instead of it has find_classes method.

Hi

I face an error when I run python main.py

thanks

Output is below:

Traceback (most recent call last):
File "main.py", line 10, in
train_set, test_set = load_datasets(c.dataset_path, c.class_name)
File "/home/ozan/differnet/differnet-master/utils.py", line 104, in load_datasets
trainset = ImageFolderMultiTransform(data_dir_train, transform=transform_train, n_transforms=c.n_transforms)
File "/home/ozan/differnet/differnet-master/multi_transform_loader.py", line 70, in init
is_valid_file=is_valid_file, n_transforms=n_transforms)
File "/home/ozan/differnet/differnet-master/multi_transform_loader.py", line 34, in init
classes, class_to_idx = self._find_classes(self.root)
AttributeError: 'ImageFolderMultiTransform' object has no attribute '_find_classes'

Hello, I'm interested in your method.

While reading your papers(DifferNet, CS-Flow), I got a question.

The *1-NN performance in the DifferNet paper is recorded as 80.0 but in the CS-Flow paper, it is recorded as 97.7.
(*Method for "Are pretrained cnns good feature extractors for anomaly detection in surveillance videos?")

May I know the difference between the two results?
And, Is there any difference in the experimental environment?

Thank you :)

Hi,
In train.py, line 89, you have the code:
is_anomaly = np.array([0 if l == 0 else 1 for l in test_labels])
The thing is, that it seems that non-defective (or "good") data gets the label 1 in your code (your first class index in utils.py line 66: class_idx = 1)

Shouldn't the correct code be: [0 if l == 1 else 1 for l in test_labels]?

Thanks,
Omer

Just run python main.py, I have the following issue:

"z_grouped = torch.cat(test_z, dim=0).view(-1, c.n_transforms_test, c.n_feat)
RuntimeError: shape '[-1, 64, 768]' is invalid for input of size 262144"

could you help to check?

Hi, Marco!

When I visualise the gradient map, I get incorrect localisation of anomalies (see examples below).
For me, the ideal is when gradients repeat the contour of the anomaly. Is it possible?

? How can I improve the accuracy of the localisation of anomalies on gradient map?

I use the evaluate.py script to scores some images.
How can I get the gradient map from the scored images?

Furthermore, it seems that the train loss here is only seeking the parameters of the normalizing flow (NF) structure that can most likely transform the features y into a standard normal distribution z. How do we determine whether the feature extraction network is good or not?

As per the screenshot, is it normal that anomaly_score is super large? I was using the following config.py (just change to cpu and meta_epochs = 2) to train the dummy_dataset. Have the same huge anomaly_score in the second epochs.

'''This file configures the training procedure because handling arguments in every single function is so exhaustive for
research purposes. Don't try this code if you are a software engineer.'''

# device settings
device = 'cpu' #'cuda' # or 'cpu'
import torch
torch.cuda.set_device(0)

# data settings
dataset_path = "dummy_dataset"
class_name = "dummy_class"
modelname = "dummy_test"

img_size = (448, 448)
img_dims = [3] + list(img_size)
add_img_noise = 0.01

# transformation settings
transf_rotations = True
transf_brightness = 0.0
transf_contrast = 0.0
transf_saturation = 0.0
norm_mean, norm_std = [0.485, 0.456, 0.406], [0.229, 0.224, 0.225]

# network hyperparameters
n_scales = 3 # number of scales at which features are extracted, img_size is the highest - others are //2, //4,...
clamp_alpha = 3 # see paper equation 2 for explanation
n_coupling_blocks = 8
fc_internal = 2048 # number of neurons in hidden layers of s-t-networks
dropout = 0.0 # dropout in s-t-networks
lr_init = 2e-4
n_feat = 256 * n_scales # do not change except you change the feature extractor

# dataloader parameters
n_transforms = 4 # number of transformations per sample in training
n_transforms_test = 64 # number of transformations per sample in testing
batch_size = 24 # actual batch size is this value multiplied by n_transforms(_test)
batch_size_test = batch_size * n_transforms // n_transforms_test

# total epochs = meta_epochs * sub_epochs
# evaluation after <sub_epochs> epochs
meta_epochs = 2
sub_epochs = 8

# output settings
verbose = True
grad_map_viz = True
hide_tqdm_bar = True
save_model = True

Hi, very thanks for your contribution！

I find one tools called FrEIA. FrEIA only transform half input in one block. However, you team choose to transform the hole input in your block. What are the advantages of doing this？

Looking forward to your reply！

When training Alexnet and VGG on my dataset, the AUROC can reach 0.95 +, but when using Resnet,the AUROC can only reach 0.5 +.
Is this normal? If not, how can I modify the code?

        # Alexnet  /  VGG
        # feat_s = self.feature_extractor.features(x_scaled)

`

        # Resnet
        feat_s = self.feature_extractor.conv1(x_scaled)
        feat_s = self.feature_extractor.bn1(feat_s)
        feat_s = self.feature_extractor.relu(feat_s)
        feat_s = self.feature_extractor.layer1(feat_s)
        feat_s = self.feature_extractor.layer2(feat_s)
        feat_s = self.feature_extractor.layer3(feat_s)
        feat_s = self.feature_extractor.layer4(feat_s)
        y_cat.append(torch.mean(feat_s, dim=(2, 3)))

`

Is there any way to train anomaly images such as to modify Loss Function get_loss() ?
Assuming that I got anomaly images after DifferNet trainning done.

I use the default configuration with MVTec AD data, but the train loss I got is negative.
Thanks for your answer!

Train epoch 0
THCudaCheck FAIL file=/pytorch/aten/src/THC/THCGeneral.cpp line=405 error=11 : invalid argument
Epoch: 0.0 train loss: -0.0312
Epoch: 0.1 train loss: -0.9794
Epoch: 0.2 train loss: -1.1155
Epoch: 0.3 train loss: -1.2642
Epoch: 0.4 train loss: -1.3475
Epoch: 0.5 train loss: -1.4397
Epoch: 0.6 train loss: -1.5100
Epoch: 0.7 train loss: -1.5529

Compute loss and scores on test set:
Epoch: 0 test_loss: -1.5800
AUROC: last: 0.9106 max: 0.9106 epoch_max: 0

How can i see classification result without localization? Such as, good and different. Is this possible?

Hello, I noticed that the final output of your code is the anomaly score. How can I locate the defects on the test image?

Hello!

First of all, thank you very much for publishing your code as well as writing the paper.

I wonder how to robustly reproduce the results stated in the paper. Namely, what setup is used for the grid category from the MVTec Anomaly Detection dataset? I have tried to use the default setup and could not reach the value of 0.84, only 0.8.

Also, my training process was quite unstable (maybe due to the aggressive data augmentations by default).

I use compatible with the requirements.txt virtual environment which I do not describe for brevity but let me know if it is important.

Thank you very much for your answer in advance.

P.S. Whilst a difference by 0.04 may seem insignificant, e.g. it is the difference between your method and ideal AUC. Hence, I am quite interested is the results stated in the paper are really robust :)

Tried to test the false alarm by using the good sample. It lable the square as anomaly as well. Is it normal as expected?