Results on subset about pytorch-mac-network HOT 4 CLOSED

Hi,
I actually run my code three times while using only 10% of the dataset (7000 images + all questions belonging to those 7000 images, so roughly 70,000 questions) for 25 epochs.
I get accuracies of 85.41%, 85.82%, and 85.05% on the validation set, which seems to be roughly the same as in the paper (at least from what I can tell from Fig. 9 (2) in the paper).
When running on 1% of the data I get accuracies of 45.79%, 46.20%, and 46.05% which is also similar to the paper (I think).
What kind of accuracies do you get?

from pytorch-mac-network.

Hi,
Thank you for your response.
I got only 74.2% after 25 epochs and about 78.4% after 40 epochs. I also took 10% of images (7,000 images and their associated questions so roughly 70,000) whilst using full validation set (15,000 images + 150,000 questions).

Can you please share your code you used to validate this? I could not seem to figure out what's wrong with my experiments.

from pytorch-mac-network.

Hi, sure. The difference might be in the sampling of the dataset. E.g., to train on 10% I randomly chose 7,000 images from the training set (10% of the images) and then use all available questions for these images (usually around 70,000) for training.
For this, you only need to modify the ClevrDataset class in datasets.py. I first choose the 7,000 images and then find their corresponding questions. Then I repeat the dataset as often as necessary so that it is as large as the original one (to make it easier to compare accuracies after trained epochs). Training on this achieves an accuracy of about 85% (for me).

This is what it looks like for me (for "num_labeled_images"=7000):

`

class ClevrDataset(data.Dataset):

    def __init__(self, data_dir, split='train', **kwargs):
        with open(os.path.join(data_dir, '{}.pkl'.format(split)), 'rb') as f:
            self.data = pickle.load(f)
        self.original_length = len(self.data)
        self.real_length = self.original_length
        self.img = os.path.join(data_dir, 'images/{}_features.h5'.format(split))
        self.img_dir = os.path.join(data_dir, 'images/{}'.format(split))

        if "num_labeled_images" in kwargs and kwargs["num_labeled_images"] is not None:
            self.num_labeled_images = kwargs["num_labeled_images"]
            img_list = glob.glob(self.img_dir + "/*.png")

            idxs = random.sample(range(len(img_list)), self.num_labeled_images)

            data_new = []
            sampled_imgs = []
            for idx in idxs:
                sampled_imgs.append(img_list[idx].split("/")[-1])
            for question in self.data:
                if question[0] in sampled_imgs:
                    data_new.append(question)
            self.data = data_new
            self.real_length = len(data_new)
            while len(self.data) < self.original_length:
                self.data += data_new
            self.data = self.data[:self.original_length]

    def __getitem__(self, index):
        imgfile, question, answer, family = self.data[index]
        id = int(imgfile.rsplit('_', 1)[1][:-4])

        imageh5 = h5py.File(self.img, 'r')
        img = torch.from_numpy(imageh5['features'][[id]]).squeeze()

        return img, question, len(question), answer, family

    def __len__(self):
        return len(self.data)`

from pytorch-mac-network.

Hi,
Thank you for that. I am now able to achieve around 82% after 30 epoch, which is acceptable I think. The difference might be in the sampling process as you mentioned.

Best regards,

from pytorch-mac-network.