Low val mIoU about pytorch-segmentation HOT 6 CLOSED

Hi, what dataset are you using it on? I see that it is pascal voc, does the mIoU stays this low after more that 10 iterations (I see you're using batches of two, so 20 images), because at the beginning, it is probable that some classes weren't seen, so the IoU for them is 0 and it reduces the mean.

from pytorch-segmentation.

Hi, glad to receive your reply! Yes, I'm using pascal voc dataset. This is the result after 1237 its, but I can't see it will show any increment...

from pytorch-segmentation.

Oh, I changed the 'split' from 'train_aug' to 'trainval_aug' in config.json just now. The number of total samples is changed, but the mIoU is still at a low level.

This is the result after 1 epoch.

The following is my config.json, which is modified from the origin to match the pretrained settings.

{
    "name": "PSPNet",
    "n_gpu": 1,
    "arch": {
        "type": "PSPNet",
        "args": {
            "backbone": "resnet50",
            "freeze_bn": false,
            "freeze_backbone": false
        }
    },

    "train_loader": {
        "type": "VOC",
        "args":{
            "data_dir": "/home/qiming/path/to/datasets",
            "batch_size": 2,
            "base_size": 420,
            "crop_size": 380,
            "augment": true,
            "shuffle": true,
            "scale": true,
            "flip": true,
            "rotate": true,
            "blur": false,
            "split": "trainval_aug",
            "num_workers": 1
        }
    },

    "val_loader": {
        "type": "VOC",
        "args":{
            "data_dir": "/home/qiming/path/to/datasets",
            "crop_size": 480,
            "batch_size": 2,
            "val": true,
            "split": "val",
            "num_workers": 1
        }
    },

    "optimizer": {
        "type": "SGD",
        "differential_lr": true,
        "args":{
            "lr": 0.01,
            "weight_decay": 1e-4,
            "momentum": 0.9
        }
    },

    "loss": "CrossEntropyLoss2d",
    "ignore_index": 255,
    "lr_scheduler": {
        "type": "Poly",
        "args": {}
    },

    "trainer": {
        "epochs": 150,
        "save_dir": "saved/",
        "save_period": 10,
  
        "monitor": "max Mean_IoU",
        "early_stop": 10,
        
        "tensorboard": true,
        "log_dir": "saved/runs",
        "log_per_iter": 20,

        "val": true,
        "val_per_epochs": 1
    }
}

from pytorch-segmentation.

After some test, the problem is the low batch size, with only a batch of 2 the loss is a lot more noisy, and the batch norm's running params also change.

for example here are the results with a batch of 8:

Possible solutions, is to use a very low learning rate, set the batch norms training to false or of course a bigger batchsize.

from pytorch-segmentation.

After some test, the problem is the low batch size, with only a batch of 2 the loss is a lot more noisy, and the batch norm's running params also change.

for example here are the results with a batch of 8:

Possible solutions, is to use a very low learning rate, set the batch norms training to false or of course a bigger batchsize.

Thanks a lot. I've changed the batch size to 8 and run it on my lab's server. Now it works well.

It occurs to me that the papers about norm methods I recently read, saying that the batch norm will show bad performance with small batch size especially with batch size of exactly 2.

from pytorch-segmentation.

Glad it worked, yes you're write, with low batch size, the batch norm running mean and var will change significantly.

from pytorch-segmentation.