In https://github.com/facebookresearch/odin/blob/master/code/calMetric.py#L180 precision and recall is computed as:

    tp = np.sum(np.sum(X1 >= delta)) / np.float(len(X1))
    fp = np.sum(np.sum(Y1 >= delta)) / np.float(len(Y1))
    if tp + fp == 0: 
        continue
    precision = tp / (tp + fp)
    recall = tp

But, values tp and fp are TPR and FPR which means that precision is computed as TPR/(TPR + FPR) which seems to be not the same as TP/(TP + FP).
Although, those values will coincide when the number of negatives equals the number of positive samples, which is true for all reported datasets, except for iSUN, am I right?

There are comparisons in the paper for OOD detection performance VS MMD of the datasets. Have you provided the code for calculating the MMD somewhere or is it possible to provide?

Hi,

Pytorch 0.4 seems not to work with this code. I've downgraded pytorch to 0.3.1 and now it works.

Thanks!

Would you consider releasing the code for the MMD and Energy distance calculation between datasets?

I see the optimizer1 has been defined in cal.py here

But I didn't see it ever being used! Am I missing something? please let me know !

Thanks

the download links of the datasets are not available,can you supply new links,thx!

Hi, thank you so much for your work. I would like to understand better on your tpr95 method.

Q1. For obtaining the results of FPR at 95% TPR, I would like to understand why the values of start and end were different when comparing your algorithm vs the baseline method, which uses an end of 0.1 and 0.01 for CIFAR10 and CIFAR100 respectively?

def tpr95(name):
...
    # calculate baseline
    if name == "CIFAR-10": 
	start = 0.1
	end = 1 
    if name == "CIFAR-100": 
	start = 0.01
	end = 1    
...
    # calculate our algorithm
...
    if name == "CIFAR-10": 
	start = 0.1
	end = 0.12 
    if name == "CIFAR-100": 
	start = 0.01
	end = 0.0104    

Q2. For the same function, how are the values "if tpr <= 0.9505 and tpr >= 0.9495:" chosen for 95% TPR? As in is it based on 4 decimal places or some other approach. I experimented with other datasets and models, and found that there were zerodivision errors errors if the tpr was slightly above 0.9505 (like 0.9545 for example)

Thank you so much!

How exactly is the value of 'start ' and 'end' chosen after performing temperature scaling?

• Is it dependent on Temperature?
• Is it dependent on the number of classes?

Is this the average value of all pixels for each channel for all images? Or something else?

Unfortunately, Python 3 treats tab errors as syntax errors...

flake8 testing of https://github.com/facebookresearch/odin on Python 3.6.3

$ _flake8 . --count --select=E901,E999,F821,F822,F823 --show-source --statistics

./code/cal.py:71:104: E999 TabError: inconsistent use of tabs and spaces in indentation
	testset = torchvision.datasets.CIFAR10(root='../data', train=False, download=True, transform=transform)
                                                                                                       ^
./code/calMetric.py:37:12: E999 TabError: inconsistent use of tabs and spaces in indentation
	start = 0.1
           ^
2     E999 TabError: inconsistent use of tabs and spaces in indentation
2

It seems you neglect first 1000 items in datasets

code/calData.py line:39 and line:93
if j<1000: continue

Why?

Thanks

I am able to reproduce the results reported in the paper when I use the pretrained models provided in the repo.
But when I train a densenet from scratch the results are poorer compared to the report. Did anyone face a similar problem?

Hi, @YixuanLi
I have a question about x˜ = x − εsign(-∇x log Syˆ(x; T)).
The loss (-∇x log Syˆ(x; T)) after sign function will get number (-1, 0, 1).

If all the image(OOD or ID) will get the same perturbation?

Thank you.
Jacky

In the paper, the equation of input preprocessing( eq(2) in paper) is x˜ = x − εsign(−∇x log Syˆ(x; T)).

This equation is introduced in your repository, https://github.com/facebookresearch/odin/blob/master/code/calData.py#L72

tempInputs = torch.add(inputs.data,  -noiseMagnitude1, gradient)

However, the gradient = torch.ge(inputs.grad.data, 0) in your line 65 represents sign(∇x log Syˆ(x; T))
In fact, I think that your code is to implement x˜ = x − εsign(∇x log Syˆ(x; T)) not x˜ = x − εsign(-∇x log Syˆ(x; T))

What's your opinion?

Hi,

Any plans releasing Single GPU version of Wide ResNet?

Thank you!

Hi,

I don't get this line - is nnOutputs[0] the first sample in the batch?

Hi, The parameters are tuned by using ood val data. Can we generalize the tuned parameters for the other ood data? Or when we have new ood samples do we have to tune the parameters again?

Also, I am curious about the possible usage of this technique on signal data like radar signals.

Thanks

You can download the WideResNet pretrained model: https://www.dropbox.com/s/elfw7e3uofpydg5/wideresnet100.pth.tar.gz