Comments (9)
Hello @gmberton, thank you for the question,
We did no pre-processing nor any post-processing, we only resize images to 320x320.
We are using the exact same procedure that the original authors published on their GitHub (see demo: https://github.com/mapillary/mapillary_sls/blob/main/demo.ipynb).
Note that in order to evaluate on the validation dataset, you need to initialize SAMPLE_CITIES = "val". and also use mode = 'val' when instantiating MSLS object. (mode='test' works just fine), but only mode='val' that will return you the ground_truth (the positives for each query).
There are 750 query images in msls_val from which they use 740 (3 are panoramas and the others are simply ignored (https://github.com/mapillary/mapillary_sls/blob/b6c4e43fcd6d5a8a7f40295ee240f8cf0448891d/evaluate.py#L116).
I rewrote the code (https://github.com/amaralibey/MixVPR/blob/main/dataloaders/MapillaryDataset.py) for MSLS to avoid listing all images into a list and generate the ground each time. I simply saved into a numpy array the qImages, the dbImages and the pIdx (which is the groud truth with threshold of 25meters for a correct match). This made it way faster for the im2im task (image to image recall@k). Also this allows us to use the same evaluation function I developed for GSV-Cities framework (https://github.com/amaralibey/MixVPR/blob/main/utils/validation.py).
Finally, I just re-evaluated our model on the demo shared by the authors of MSLS and got the exact same results:
all_recall@1: 0.882
all_recall@2: 0.903
all_recall@3: 0.922
all_recall@4: 0.928
all_recall@5: 0.931
all_recall@10: 0.943
all_recall@15: 0.955
all_recall@20: 0.955
all_recall@25: 0.957
all_map@1: 0.882
all_map@2: 0.824
all_map@3: 0.782
all_map@4: 0.747
all_map@5: 0.709
all_map@10: 0.644
all_map@15: 0.625
all_map@20: 0.619
all_map@25: 0.620
Please let me know if there's something I missed or misunderstood.
Thank you,
from mixvpr.
Thank you both @amaralibey and @hit2sjtu for your thorough answers!
I was confused because I always followed the procedure from the MSLS paper, where images from Copenhagen and San Francisco are used for validation (excluding the panoramas), so I always used all their queries (11k). I see that however in the code they select a much smaller subset (also from Copenhagen and San Francisco) of 740 images according to the fields of the files subtask_index.csv
, so I should use those to reproduce your results :)
from mixvpr.
@amaralibey Fairness of comparison can be tricky: I'm not sure that training all methods on GSV-Cities is the best way to make a fair comparison, given that some methods might not be suitable for it.
For example CosPlace is designed for training on large-scale dense datasets, and in the paper it is shown to perform poorly when trained on smaller datasets, whereas other methods (like NetVLAD) might be more suitable for smaller datasets.
Also, using the same backbone for all methods is generally a good idea, but in some cases a given backbone can give an advantage to some methods: for example, NetVLAD (and probably MixVPR) performs best when a ResNet backbone is cropped at the second last layer, whereas GeM performs best when cropping the ResNet at the last layer.
In my opinion there are very few things that bring benefits to any method (bigger training datasets not being one of them), for example using a bigger batch size or better data augmentation.
from mixvpr.
I totally agree with you that not using a fixed split between queries and database can help to increase the effective training data (we also didn't use a fixed split for CosPlace), and the mining method that you use clearly improves the results of GeM and NetVLAD, congrats on that!
On the other hand CosPlace is designed to be trained on a dense and large scale (>1M images) dataset, so training it on a sparse and smaller dataset (as is GSV-Cities) might not be fair, given that we clearly show such limitation of CosPlace in the paper.
But again, it is very difficult (if not impossible) to design completely fair experiments in VPR, because different methods might benefit from different training choices.
Also, higher resolution does not always mean better results: you can see in this benchmark that sometimes lower resolutions can improve the recall, although in CosPlace we used similar resolutions to all previous works.
from mixvpr.
@gmberton
Here is my reproduction result using the checkpoint"resnet50_MixVPR_4096_channels(1024)_rows(4).ckpt'.
For your reference I use my previous script which loads MSLS images and performs image resizing to 320x320 (cubic sampling) and same normalization as here.
Then I use faiss to perform nearest-search to get top 5 match and finally get recall&map using MSLS official evaluation script. The metrics are computed using a total of 740 images as @amaralibey said.
all_recall@1: 0.877
all_recall@5: 0.931
all_map@1: 0.877
all_map@5: 0.706
from mixvpr.
@hit2sjtu, thank you for trying our models :) your results are correct when using BICUBIC interpolation to resizeimages to 320x320 (I've tried BICUBIC interpolation when testing and got the same results as you.). However, since our models have been trained on images resized with BILINEAR interpolation, it's preferable to resize the test/valid sets with the same interpolation used during the training. For instance, if you want the best performance (as those I shared) then you just need to use BILINEAR interpolation when resizing :)
@gmberton now I understand, thanks for clarifying. Sometimes (and most of the time in VPR), the code doesn't reflect exactly what was presented in the paper. In our comparison in Table 1 and Figure 3, we trained all networks on the same data and used the same script for validation and testing, it was always about fairness of comparison.
Thanks to both of your,
Best regards.
from mixvpr.
I did just try some simple training on single 4080 GPU with no param tuning. I have to limit the batch size to 100 due to memory constraints. After only 6 hours (40 epochs), the results on MSLS looks great though. Really impressive @amaralibey
all_recall@1: 0.858
all_recall@5: 0.923
from mixvpr.
@gmberton In our paper we chose the best configuration for CosPlace (we cropped ResNet at the last residual block, not the second last, because that's how it was configured in your paper). We used the same image size (320x320) and batch size as other techniques.
As for the dataset, I think that GSV-Cities is not a small dataset for VPR, because instead of fixing queries and forming pairs and triplets around them, we approached the training as a metric learning problem, where every image in the dataset is used as query and as reference (depending on how it was selected in the online mining step), that makes the number of informative triplets and pairs exponentially higher than when fixing the number of queries. That's one reason why both GeM (which you showed that it benefits from bigger datasets) and NetVLAD obtain a new sota performance when trained on GSV-Cities (which is ~80x smaller than SF-XL). I believe that Metric Learning on GSV-Cities has the potential to bring a lot to VPR.
Finally, when training CosPlace on GSV-Cities, I think we could've achieved better results if we used images of size 480x640 or 512x512 as used in your paper. However, due to the limited resolution of images in GSV-Cities, we opted for 320x320 resolution and obtained 84.5 recall@1 on MSLS (compared to 85.2 when trained on SF-XL with resolution 512x512 as in your paper).
from mixvpr.
What does a fixed split mean?
#5 (comment)
from mixvpr.
Related Issues (20)
- why is the skip connection executed inside the FeatureMixerLayer here? HOT 2
- Really a good work with simple but effective approach!
- Training loss and generalization during test HOT 4
- Just for testing HOT 2
- A singleGPU will run the results, but multiple GPUs will make an error!! HOT 3
- can you provide more detailed comparative data? HOT 2
- optimizer step HOT 2
- question about backbone ‘Swin’ HOT 5
- Multi-similarity mining on Pittsburgh30k training set HOT 6
- how to change query image shape? HOT 1
- The error samples are due to issues with the ground truth annotations rather than errors in the model predictions. HOT 6
- Dataset HOT 4
- About the specific number of images of the Mapillary Challenge dataset
- would you release the resnet18 pretrained model?
- How to evaluate this model?
- Some questions
- About pitts30k_val.mat. I can not find it in PittsburgDataset HOT 1
- License File
- Releasing the model on torch.hub?
- On releasing the trained weights of ResNet50 with 2048 dimensionality.
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from mixvpr.