niamhbelton / fewsome Goto Github PK

In my opinion, the computation of F1 is wrong.

The output with "outputs > thres" are first set to be 1. Then, these "1" are also used for the following "outputs <= thres".

To my complete shock, the threshold is chosen by "the outputs of testing" by 10% proportion. In this way, your testing accuracy is fixed and can be predicted in advance to be 90% (F1 can be a bit biased, but it is also fixed and can be predicted in advance). Then, due to the above fault, your F1 float around 90%, as shown in your paper.

Beacause your test dataset including anomalies, 90% may be not very accurate and even higher. For example, CIFAR10 testing cosists of 10% targets and 90% anomalies. Targets always have lower/higher distances/scores than anomalies due to your model trained only using targets. Ideally, you get 100% accuracy by your threshold setting.

But, the threshold is also a part of your model. It can't be derived from the testing. In other words, your high accuracy is caused by the testing information leakage.

What you do will mislead the people who will work in this field in the future.

Hi, in the following snippet the image (x) is being passed as is. However, the base model would expect normalized image. The image was not normalized elsewhere in the code. Is it supposed to be like this?

class RESNET_pre(nn.Module):
  def __init__(self):
      super(RESNET_pre, self).__init__()

      self.features = models.resnet18(pretrained=True)


  def forward(self, x):
      x = torch.unsqueeze(x, dim =0)
      x= self.features(x)
      x=nn.Sigmoid()(x)

      return x #output

In my opinion, the computation of F1 is wrong.

The output with "outputs > thres" are first set to be 1. Then, these "1" are also used for the following "outputs <= thres".

To my complete shock, the threshold is chosen by "the outputs of testing" by 10% proportion. In this way, your testing accuracy is fixed and can be predicted in advance to be 90% (F1 can be a bit biased, but it is also fixed and can be predicted in advance). Then, due to the above fault, your F1 float around 90%, as shown in your paper.