relja / netvlad Goto Github PK

hello!
I followed the instructions of "readme" and used the database "Tokyo247". However, a problem occurred to me. What is relationship between query images, database and test images? After finishing reading your paper, I assumed the test image play the role of query image.
By the way, if I want to train my own network using Pytorch , the length of triplet is fixed? and how can I choose the potential positive images and how can I choose the firm negative ones. The details are confusing to me.

How can I use multiple GPUs (several K20m) to extract features on large dataset? Thanks in advance.

Apologies if my question sounds dumb!

I would like to know if it makes sense to use netvlad for place recognition of problems dealing with sequences of images (SLAM) over seasonal changes (no specific viewpoint changes)?

What I have in mind is Nordland dataset in which images rather suffer from illumination changes over seasons.

Cheers,

i want to construct the off-the-shelf network:
caffe_offtheshelf_conv5_max.mat, vd16_offtheshelf_conv5_3_max.mat.
I try to do that as you suggest, but one question is that how to define the options since i find the example code in trainweakly.m:
% ----- Network setup

net= loadNet(opts.netID, opts.layerName);

% --- Add my layers
net= addLayers(net, opts, dbTrain);

.
I do not know how to define the opts, dbTrain for addLayers.
Could you give me additional advice or upload the copy of source code about how to obtain the off-the-shelf networks?

Hi @Relja ,I am a student and I am learning about your netvlad.

After browsing the relevant home page, I have completed the basic configuration, but the next work lacks the database file. I learned from others that I need to send an application to you. Can you provide it to me? My email address is [email protected] .

Thank you very much.

At the same time, there are some configuration problems. I run the code on win10 system, but I see Yael_ matlab_ linux64_ V438 doesn't fit with win. If I don't download matlab from Ubuntu to reconfigure, is the path set to Yael_ The path to the dummy file?

The feature is small in my experiments which is [1, 1024]. This shape is far small than that in your experiments. So how can I choose my cluster number and dimension number?

wrong use testCoreRank (line 45)
No positives in the test
wrong testFromFn (line 34)
rankloss= testCoreRank(db, qFeat, dbFeat, opts.margin, opts.nNegChoice,
'nTestSample', opts.nTestRankSample);
wrong trainWeakly (line 191)
testFromFn(dbVal, opts.dbCheckpoint0val, opts.qCheckpoint0val,
opts);

Hi Sir,
When we train this netvlad by using our own database (about 340 pictures) we find this problem. And we don't konw how to deal with it or change some variables.

• When I run this demo, I meet a error.

serialAllFeats: Start
serialAllFeats 09-Nov-2022 11:07:26 1 / 7599
73 if iBatch==1
serialAllFeats 09-Nov-2022 11:31:06 2 / 7599; time 00:23:40; left 2997:29:09; avg 1420.2355 s
serialAllFeats 09-Nov-2022 11:31:19 3 / 7599; time 00:23:52; left 1511:52:57; avg 716.4378 s
serialAllFeats 09-Nov-2022 11:31:51 4 / 7599; time 00:24:24; left 1030:00:57; avg 488.1592 s
serialAllFeats 09-Nov-2022 11:32:05 8 / 7599; time 00:24:38; left 445:34:30; avg 211.2844 s
serialAllFeats 09-Nov-2022 11:32:08 16 / 7599; time 00:24:42; left 208:10:14; avg 98.8152 s
serialAllFeats 09-Nov-2022 11:32:15 32 / 7599; time 00:24:48; left 100:57:08; avg 48.0217 s
serialAllFeats 09-Nov-2022 11:32:28 64 / 7599; time 00:25:01; left 49:53:44; avg 23.8356 s
serialAllFeats 09-Nov-2022 11:32:54 128 / 7599; time 00:25:27; left 24:57:55; avg 12.0283 s
serialAllFeats 09-Nov-2022 11:33:46 256 / 7599; time 00:26:20; left 12:38:29; avg 6.1968 s
serialAllFeats 09-Nov-2022 11:35:31 512 / 7599; time 00:28:05; left 06:29:33; avg 3.2976 s
serialAllFeats 09-Nov-2022 11:37:12 759 / 7599; time 00:29:46; left 04:28:39; avg 2.3564 s
serialAllFeats 09-Nov-2022 11:39:02 1024 / 7599; time 00:31:36; left 03:23:09; avg 1.8537 s
错误使用 fwrite
文件标识符无效。使用 fopen 生成有效的文件标识符。

出错 serialAllFeats (line 79)
fwrite(fout, buffer(:, 1:numInBuffer), class(buffer));

出错 demo (line 93)
serialAllFeats(net, dbTest.dbPath, dbTest.dbImageFns, dbFeatFn, 'batchSize', 10); % adjust batchSize depending on your GPU /
network size

• I find that the reason for this error is there is no file called vd16_tokyoTM_conv5_3_vlad_preL2_intra_white_tokyo247_db.bin.

In demo.m
dbFeatFn= sprintf('%s%s_%s_db.bin', paths.outPrefix, netID, dbTest.name);

But I don't find the introduction about this file. So should I create this file myself.
Thanks.

Hello, please send me the database, thank you。
[email protected]

Hello, I would like to do only inference on Tokyo247 using AlexNet + Max (off-the-shelf).
After running the following code I get rec@10= 0.0762, but I see in the paper that the rec@10 for AlexNet + Max off-the-shelf should be around 0.32. Am I doing something wrong in building the net? How can I load one of your trained nets from the project page? I don't think it could be useful as the weights of the net are fixed, but I'm using matlab R2020a, matconvnet-1.0-beta25, cuda 10.1

`cd my_path/netvlad;
setup;
paths = localPaths();
opts.netID = 'caffe';
opts.layerName = 'conv5';
opts.method = 'max';
net = loadNet(opts.netID, opts.layerName);
net = addLayers(net, opts, []);

dbTest= dbTokyo247();
qFeatFn = sprintf('%s%s_%s_q.bin', paths.outPrefix, opts.netID, dbTest.name);
dbFeatFn= sprintf('%s%s_%s_db.bin', paths.outPrefix, opts.netID, dbTest.name);
serialAllFeats(net, dbTest.qPath, dbTest.qImageFns, qFeatFn, 'batchSize', 1);
serialAllFeats(net, dbTest.dbPath, dbTest.dbImageFns, dbFeatFn, 'batchSize', 64);
qFeat = fread( fopen(qFeatFn, 'rb'), inf, 'float32=>single');
dbFeat= fread( fopen(dbFeatFn, 'rb'), inf, 'float32=>single');
[recall, ~, ~, opts]= testFromFn(dbTest, dbFeatFn, qFeatFn);
`

Hope you are fine. Thanks for your amazing work. Sorry to bother you here , I have sent you an email for the dataset ,but I haven't received your reply, have you received any email from me? my email name is zhuxiaozx. Sorry to bother you here again! Looking forward to your reply,thanks a lot!

This is hard for me to test as I don't have Matlab and didn't find an answer in the paper. Could anyone help? I'm essentially wondering what kind of throughput this can achieve on a GPU in terms of images/s.

MatConvNet beta17 broke some backward compatibility so the code doesn't currently work with this version. Fix is coming soon, until then use previous versions.

I try to export weights from netvlad to use in tensorflow model, but have an issue, weights for 31 and 33 layers are empty.

net = load('data/vd16_pitts30k_conv5_3_vlad_preL2_intra_white.mat')
Warning: While loading an object of class 'layerVLAD':
Method 'loadobj' is not defined for class 'layerVLAD' or is removed from MATLAB's search path. 

net.net.layers{31}

ans = 

  layerVLAD with properties:

            type: 'custom'
            name: 'VLAD'
               K: []
               D: []
         vladDim: []
         weights: []
        momentum: []
    learningRate: []
     weightDecay: []
        precious: 0

What am I doing wrong?

PS:
I use matlab 2015b for mac and installed matconvnet and add relja_matlab into matlab path.

Do you guys also have deploy.prototxt and model.caffemodel files compatible with python?

Or is there any workaround to convert those .mat model files into deploy.prototxt and model.caffemodel to apply for feature extraction in python?

Cheers,

Hi @Relja ,I find that I need the TokyoTM database to train, but in your project I don't find the download link. Can you provide it to me？
Thanks a lot!

HI, I'm studying netvlad. I think pitts30 datasets is necceray for beginers. Please tell me some link? Thank you very much.
My email : [email protected]

whether the imPath is the folder which include images？ but how to set the imageFns？

Is there any other way to get those datasets, such as Tokyo 24/7? I have sent an email to Akihiko Torii before, however, There is no reply......

hi, I am implementing it in torch. I have some confusion. Kindly, guide me.

1. suppose last conv layer has maps with W x H size and depth of layer (number of feature maps) is D. according to your paper, they are interpreted as N X D descriptor. Is it imply that we made a single long vector on size N=(W*H) and there are total D of them?

2. second, each feature map of conv layer is treated as single descriptor of image?

3. please explain dimension written on 1 X 1 conv layer. when we apply 1x1 conv then depth of feature maps changes, right. What is D x K?

4. final output (vlad vector) is telling us scores for each class/cluster?

Kindly, consider.
thanks

Dear Relja,
I have download all the datasets correctly on the project_web and your email link. when I was running the project with tinyTimeMachine or RootTokyoTM, it occured the error as followed.

opts = 

  struct with fields:

                  netID: 'caffe'
              layerName: 'conv5'
                 method: 'max'
              batchSize: 4
           learningRate: 0.0100
             lrDownFreq: 3
           lrDownFactor: 2
            weightDecay: 0.1000
               momentum: 0.9000
        backPropToLayer: 13
              fixLayers: {}
             nNegChoice: 30
                nNegCap: 10
              nNegCache: 10
                 nEpoch: 10
                 margin: 0.1000
        excludeVeryHard: 0
             jitterFlip: 0
            jitterScale: []
              sessionID: 'fdfa'
              outPrefix: 'G:/yk/NetVLAD_out/'
          dbCheckpoint0: 'G:/yk/NetVLAD_out/tokyoTinyTM_train_caffe_conv5_max_db.bin'
           qCheckpoint0: 'G:/yk/NetVLAD_out/tokyoTinyTM_train_caffe_conv5_max_q.bin'
       dbCheckpoint0val: 'G:/yk/NetVLAD_out/tokyoTinyTM_val_caffe_conv5_max_db.bin'
        qCheckpoint0val: 'G:/yk/NetVLAD_out/tokyoTinyTM_val_caffe_conv5_max_q.bin'
      checkpoint0suffix: ''
                   info: 'tiny test'
                  test0: 1
          saveFrequency: 15
     compFeatsFrequency: 10
       computeBatchSize: 10
     epochTestFrequency: 1
                 doDraw: 1
              printLoss: 0
         printBatchLoss: 0
            nTestSample: Inf
        nTestRankSample: 40
               recallNs: [1 2 3 4 5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 80 85 90 95 100]
                 useGPU: 1
             numThreads: 12
             startEpoch: 1
            dbTrainName: 'tokyoTinyTM_train'
              dbValName: 'tokyoTinyTM_val'
    backPropToLayerName: 'conv5'
          backPropDepth: 3

testFromFn:
G:/yk/NetVLAD_out/tokyoTinyTM_val_caffe_conv5_max_db.bin
G:/yk/NetVLAD_out/tokyoTinyTM_val_caffe_conv5_max_q.bin
NaN 13-Aug-2019 14:22:51 1 / 40
0.0434 13-Aug-2019 14:22:52 2 / 40; time 00:00:00; left 00:00:06; avg 0.1603 s
0.0273 13-Aug-2019 14:22:52 3 / 40; time 00:00:00; left 00:00:03; avg 0.0904 s
0.0386 13-Aug-2019 14:22:52 4 / 40; time 00:00:00; left 00:00:02; avg 0.0646 s
0.0362 13-Aug-2019 14:22:52 8 / 40; time 00:00:00; left 00:00:01; avg 0.0319 s
0.0370 13-Aug-2019 14:22:52 12 / 40; time 00:00:00; left 00:00:00; avg 0.0228 s
0.0321 13-Aug-2019 14:22:52 16 / 40; time 00:00:00; left 00:00:00; avg 0.0169 s
0.0333 13-Aug-2019 14:22:52 20 / 40; time 00:00:00; left 00:00:00; avg 0.0135 s
0.0350 13-Aug-2019 14:22:52 24 / 40; time 00:00:00; left 00:00:00; avg 0.0113 s
0.0372 13-Aug-2019 14:22:52 28 / 40; time 00:00:00; left 00:00:00; avg 0.0097 s
0.0369 13-Aug-2019 14:22:52 32 / 40; time 00:00:00; left 00:00:00; avg 0.0085 s
0.0360 13-Aug-2019 14:22:52 36 / 40; time 00:00:00; left 00:00:00; avg 0.0076 s
0.0364 13-Aug-2019 14:22:52 40 / 40; time 00:00:00; left 00:00:00; avg 0.0069 s

	loss= 0.0365, margin= 0.1000, time= 0.2709 s, avgTime= 6.7723 ms

NaN 13-Aug-2019 14:22:52 1 / 60
1.0000 13-Aug-2019 14:22:52 2 / 60; time 00:00:00; left 00:00:01; avg 0.0190 s
1.0000 13-Aug-2019 14:22:52 3 / 60; time 00:00:00; left 00:00:00; avg 0.0118 s
1.0000 13-Aug-2019 14:22:52 4 / 60; time 00:00:00; left 00:00:00; avg 0.0092 s
1.0000 13-Aug-2019 14:22:52 6 / 60; time 00:00:00; left 00:00:00; avg 0.0080 s
0.8571 13-Aug-2019 14:22:52 8 / 60; time 00:00:00; left 00:00:00; avg 0.0060 s
0.9091 13-Aug-2019 14:22:52 12 / 60; time 00:00:00; left 00:00:00; avg 0.0042 s
0.8667 13-Aug-2019 14:22:52 16 / 60; time 00:00:00; left 00:00:00; avg 0.0032 s
0.8235 13-Aug-2019 14:22:52 18 / 60; time 00:00:00; left 00:00:00; avg 0.0029 s
0.8696 13-Aug-2019 14:22:52 24 / 60; time 00:00:00; left 00:00:00; avg 0.0023 s
0.8621 13-Aug-2019 14:22:52 30 / 60; time 00:00:00; left 00:00:00; avg 0.0019 s
0.8710 13-Aug-2019 14:22:52 32 / 60; time 00:00:00; left 00:00:00; avg 0.0018 s
0.8857 13-Aug-2019 14:22:52 36 / 60; time 00:00:00; left 00:00:00; avg 0.0017 s
0.9024 13-Aug-2019 14:22:52 42 / 60; time 00:00:00; left 00:00:00; avg 0.0015 s
0.9149 13-Aug-2019 14:22:52 48 / 60; time 00:00:00; left 00:00:00; avg 0.0014 s
0.9245 13-Aug-2019 14:22:52 54 / 60; time 00:00:00; left 00:00:00; avg 0.0013 s
0.9322 13-Aug-2019 14:22:52 60 / 60; time 00:00:00; left 00:00:00; avg 0.0012 s

	rec@10= 0.9333, time= 0.0708 s, avgTime= 1.1805 ms

001 0.4333
002 0.6167
003 0.7167
004 0.7667
005 0.7833
010 0.9333
015 0.9500
020 1.0000
025 1.0000
030 1.0000
035 1.0000
040 1.0000
045 1.0000
050 1.0000
055 1.0000
060 1.0000
065 1.0000
070 1.0000
075 1.0000
080 1.0000
085 1.0000
090 1.0000
095 1.0000
100 1.0000

testFromFn:
G:/yk/NetVLAD_out/tokyoTinyTM_train_caffe_conv5_max_db.bin
G:/yk/NetVLAD_out/tokyoTinyTM_train_caffe_conv5_max_q.bin
Error using testFromFn (line 22)
Assertion failed.

Error in trainWeakly (line 193)
            testFromFn(dbTrain, opts.dbCheckpoint0, opts.qCheckpoint0, opts);

Error in test (line 6)
trainWeakly(dbTrain, dbVal, ...

I was only running the code with these.

dbTrain= dbTiny('train'); dbVal= dbTiny('val'); 

trainWeakly(dbTrain, dbVal, 'netID', 'caffe', 'layerName', 'conv5', ...

    'method', 'max', 'backPropToLayer', 'conv5', ...

    'margin', 0.1, ...

    'batchSize', 4, 'learningRate', 0.01, 'lrDownFreq', 3, 'momentum', 
0.9, 'weightDecay', 0.1, 'compFeatsFrequency', 10, ...

    'nNegChoice', 30, 'nNegCap', 10, 'nNegCache', 10, ...

    'nEpoch', 10, ...

    'epochTestFrequency', 1, 'test0', true, ...

    'nTestSample', inf, 'nTestRankSample', 40, ...

    'saveFrequency', 15, 'doDraw', true, ...

    'useGPU', true, 'numThreads', 12, ...

    'info', 'tiny test');

The files were stored as followed.

When I changed the paths.dsetRootTokyoTM with I:\datasets\NetVLAD\tokyoTimeMachine. It occured the same error.

Error using testFromFn (line 22)
Assertion failed.

Error in trainWeakly (line 193)
            testFromFn(dbTrain, opts.dbCheckpoint0, opts.qCheckpoint0, opts);

Error in test (line 6)
trainWeakly(dbTrain, dbVal, ...
```

When I run this code， I meet this problem， I dont konw why，hope your answer。thank you。

demoRetrieval
警告: 名称不存在或不是目录: E:\SLAM相关\回环检测\Code\matconvnet\matlab\mex
In path (line 109)
In addpath (line 96)
In vl_setupnn (line 13)
In run (line 91)
In setup (line 3)
In demoRetrieval (line 11)
警告: MatConvNet is not compiled. Consider running vl_compilenn.
In vl_setupnn (line 24)
In run (line 91)
In setup (line 3)
In demoRetrieval (line 11)
serialAllFeats: Start
serialAllFeats: Done
serialAllFeats: Start
serialAllFeats: Done
错误使用 fread
文件标识符无效。使用 fopen 生成有效的文件标识符。
出错 demoRetrieval (line 79)
dbFeat= fread( fopen(dbFeatFn, 'rb'), inf, 'float32=>single');

Hello, thanks for sharing code.

I have dataset from simulator. Each of image has txt file with cartesian coordinates. I don't have GPS data, therefore I don't have UTM data.
What should I change/add in your code so that I can make dbStruct and class of dataset by use cartesian coordinates?

RT

Hello, thank you for your work on the VPR task.:clap: I have some questions about pitts250k dataset

1. One panoramic image is cut into 24 perspective images according to two yaw Directions [4, 26.5] and 12 pitch [0, 30,..., 360] directions. How is it done? Is there a code?

2. I found that some objects have strange problems after cutting, such as: the lamp on the roadside lamp pole is separated from the pole, and there is a lateral dislocation in the middle. Is this caused by cutting the panoramic image?

3. My current research needs the original panorama (before cutting), but collecting data by yourself is a labor-intensive work. Can you share it with me if it is convenient? Or can you tell me the focal length of the perspective image after cutting, so that the cut images can be restored on the cylinder by projection?

Do you have any better suggestions? Looking forward to your early reply!:smiley:

Hello, I use demo.m to try to get some results, but I find there is no result in figure of recall@N . Here is my code.

1. netID='vd16_tokyoTM_conv5_3_vlad_preL2_intra_white';%%%%%按照github上更换成这个网络
2. 
3. paths= localPaths();
4. load( sprintf('%s%s.mat', paths.ourCNNs, netID), 'net' );
5. net= relja_simplenn_tidy(net);
6. im= vl_imreadjpeg({which('football.jpg')}); im= im{1}; 
7. feats= computeRepresentation(net, im);
8.  dbTest= dbTokyo247();
9. paths= localPaths();
10. dbFeatFn= sprintf('%s%s_%s_db.bin', paths.outPrefix, netID, dbTest.name);
11. 
12. qFeatFn = sprintf('%s%s_%s_q.bin', paths.outPrefix, netID, dbTest.name);
13. serialAllFeats(net, dbTest.dbPath, dbTest.dbImageFns, dbFeatFn, 'batchSize', 10); 
14. serialAllFeats(net, dbTest.qPath, dbTest.qImageFns, qFeatFn, 'batchSize', 1); 
15. [recall, ~, ~, opts]= testFromFn(dbTest, dbFeatFn, qFeatFn);
16. plot(opts.recallNs, recall, 'ro-'); grid on; xlabel('N'); ylabel('Recall@N');

And I have acquired two .bin files of query picture and database picture by running code. But the result of retrieval doesn't look so good.

Dear Relja,
In the context of my Master's thesis I am working on visual place recognition and I am using NetVlad as a baseline. In order to make the comparison a bit more fair, I would like to fine-tune your best performing (compact-descriptor) model on my dataset, which is a subset of the Oxford Robotcar dataset.

I have two questions and one issue that I would very much appreciate your help with:
Question 1) For using NetVlad as baseline, I want to fine-tune your best performing compact-descriptor model, which according to my understanding would be that one contained in vd16_pitts30k_conv5_3_vlad_preL2_intra_white plus cropping the resulting descriptors at 256-Dims and L2-normalizing them. Could you just quickly confirm this?

Question 2) Is there an easy way to adjust the code such that I can take exactly that model downloaded from your website and train it further for a couple of epochs on my dataset?
What I tried so far is summarized as follows:

1. I defined the dataset exactly as described in dbBase.m.
2. I am 100% sure that there are positives in all the train / val / test sets.
3. I modiefied lines 4-26 in loadNet.m to be able to load the model downloaded from your project website:

    switch netID
        case 'vd16'
            netname= 'imagenet-vgg-verydeep-16.mat';
        case 'vd19'
            netname= 'imagenet-vgg-verydeep-19.mat';
        case 'caffe'
            netname= 'imagenet-caffe-ref.mat';
        case 'places'
            netname= 'places-caffe.mat';
        case 'vd16_pitts30k'
            netname= 'vd16_pitts30k_conv5_3_vlad_preL2_intra_white.mat';
        otherwise
            error( 'Unknown network ID', netID );
    end
    
    paths= localPaths();
    if strcmp(netID, 'vd16_pitts30k')
        net= load( sprintf('%s%s', paths.pretrainedCNNs, netname), 'net');
        net= net.net;
        %net= load( fullfile(paths.pretrainedCNNs, netname), 'net');
    else
        net= load( fullfile(paths.pretrainedCNNs, netname));
    end

4. My train_robotcar.m looks like this:

%%
dbTrain= dbOxfordRobotCar('train');
dbVal= dbOxfordRobotCar('val');
lr= 0.0001;

%%
sessionID= trainWeakly(dbTrain, dbVal, ...
    'netID', 'vd16_pitts30k', 'layerName', 'conv5_3', 'backPropToLayer', 'conv5_3', ...
    'method', 'vlad_preL2_intra', 'learningRate', lr, ...
    'batchSize', 20, ...
    'computeBatchSize', 40, ...
    'nEpoch', 20, ...
    'sessionID', [], ...
    'useGPU', true, 'numThreads', 12, ...
    'doDraw', false, 'startEpoch', 1);

As I understand it all weights belonging to the VGG part are fixed and only the NetVlad layer is trained on the dataset.
Here is the list of options output when I start the training:

opts =                                                                                     
                                                                                          
  struct with fields:                                                                                      
                                                                                                     
                  netID: 'vd16_pitts30k'                                                      
              layerName: 'conv5_3'                                                         
                 method: 'vlad_preL2_intra'                                                
              batchSize: 20                                                                
           learningRate: 1.0000e-04                                                        
             lrDownFreq: 5                                                                 
           lrDownFactor: 2                                                                
            weightDecay: 1.0000e-03                                                                                 
               momentum: 0.9000                                                           
        backPropToLayer: 29                                                               
              fixLayers: {}                                                               
             nNegChoice: 1000                                                              
                nNegCap: 10                                                                                      
              nNegCache: 10                                                                
                 nEpoch: 20                                                               
                 margin: 0.1000                                                               
        excludeVeryHard: 0                                                                    
             jitterFlip: 0                                                                
            jitterScale: []                                                               
              sessionID: '3dca'                                                           
              outPrefix: '~/netvlad/output/'                                               
          dbCheckpoint0: '~/netvlad/output/robotcar_train_vd16_pitts30k_conv5_3_vlad_preL2_intra_db.bin'
           qCheckpoint0: '~/netvlad/output/robotcar_train_vd16_pitts30k_conv5_3_vlad_preL2_intra_q.bin'
       dbCheckpoint0val: '~/netvlad/output/robotcar_val_vd16_pitts30k_conv5_3_vlad_preL2_intra_db.bin'
        qCheckpoint0val: '~/netvlad/output/robotcar_val_vd16_pitts30k_conv5_3_vlad_preL2_intra_q.bin'
      checkpoint0suffix: ''                                                                                             
                   info: ''                                                                                                                      
                  test0: 1                                                                            
          saveFrequency: 2000                                                                                            
     compFeatsFrequency: 1000                                                                                               
       computeBatchSize: 40                                                                                                     
     epochTestFrequency: 1                                                                                                       
                 doDraw: 0                                                                                                   
              printLoss: 0                                                                                                
         printBatchLoss: 0                                                                                 
            nTestSample: 1000                                                             
        nTestRankSample: 5000                                                             
               recallNs: [1 2 3 4 5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 80 85 90 95 100]  
                 useGPU: 1                                                                       
             numThreads: 12                                                                   
             startEpoch: 1                                                                                                                                   
            dbTrainName: 'robotcar_train'                                                                          
              dbValName: 'robotcar_val'                                                                    
    backPropToLayerName: 'conv5_3'                                                                                                                                                           
          backPropDepth: 5  

After training for a couple of hours I get the following error:

serialAllFeats: Done
testNet: 3dca ep000001_latest_train
NaN 07-Jan-2019 09:45:07 1 / 5000
Index in position 2 exceeds array bounds (must not exceed 33935).

Error in testCoreRank (line 37)
        dsSq= sum( bsxfun(@minus, qFeat(:, qID), dbFeat(:, potPosIDs)) .^2, 1 );

Error in testNet (line 5)
    rankloss= testCoreRank(db, qFeat, dbFeat, opts.margin, opts.nNegChoice, 'nTestSample', opts.nTestRankSample);

Error in trainWeakly (line 481)
            [obj.train.recall(:, end+1), obj.train.rankloss(:, end+1) ...

Error in train_robotcar (line 16)
sessionID= trainWeakly(dbTrain, dbVal, ...

Can you make anything of this?
Your help is much appreciated.

Can I use GPS coordinates in my dataset to replace the utm coordinates to generate .mat files ? Do I need to modify the code which is used to find closePosition?（db.cp= closePosition(db.utmDb, db.posDistThr); in dbBase.m)
If I use GPS coordinates, do I need to modify the code to use a different radius, and a suitable distance measure?

Hi,
I want to do seoul landmark recognition referencing your project.
The problem is that I cannot figure out how I can get street view time machine data.
If I see google map, then there are just few photos in street view time machine.
Are there other methods to get a lot of time machine data?

Thank you for reading this.

Hi,
I am stuck on an issue to use '.mat' trained model files inside PyTorch module. I want to use these as baseline models for a new dataset and somehow test and find my recall values. The problem with PyTorch is that the weight checkpoints (.pth.tar) are usually saved in the form of Ordered dict with different keys and in .mat files, the weights are save in the form of numpy.ndarray. I am not able to load the model inside my script and use them for inference.

As a normal procedure, I am using scipy.io module and then 'loadmat' function to load the NetVLAD trained model '.mat' files. I suppose we need to check the shape and also possible flipped kernels, since some frameworks flip the conv kernels to use a convolution instead of the correlation. This is suggested inside this thread:
https://discuss.pytorch.org/t/how-to-transfer-a-trained-model-from-matlab-to-pytorch/15903

The other option is also to convert to ONNX format and then convert to PyTorch model. The link for that is here:
https://stackoverflow.com/questions/66223768/how-to-import-deep-learning-models-from-matlab-to-pytorch
The problem with this approach is that you need to have MATLAB first for converting it every time to '.onnx' format and then via script to PyTorch.

Anyone with the better solution and a script to help with this would be really beneficial for future work.

pitts30k datasets
Hello, please send me the database, thank you.

Hello.

I ran the code with Pitts30K and TokyoTM dataset.

When I ran the code with "vd16_pitts30k_conv5_3_vlad_preL2_intra_white" or
"vd16_tokyoTM_conv5_3_vlad_preL2_intra_white" models with Pitts30K test set,
I got following recall values.

Recall@
1 0.0117
2 0.0239
3 0.0333
4 0.0427
5 0.0543

If I run the same code with TokyoTM dataset, it gives perfect result (above 95%).
Although I utilized provided code examples from demo.m, the result looks strange.
Following is my evaluation code:

netID= 'vd16_pitts30k_conv5_3_vlad_preL2_intra_white';
paths= localPaths();
load( sprintf('%s%s.mat', paths.ourCNNs, netID), 'net' );
net= relja_simplenn_tidy(net);
dbTest = dbPitts(false, 'test');
dbFeatFn= sprintf('%s%s_%s_db.bin', paths.outPrefix, netID, dbTest.name);
qFeatFn = sprintf('%s%s_%s_q.bin', paths.outPrefix, netID, dbTest.name);
batchSize = 32;
serialAllFeats(net, dbTest.dbPath, dbTest.dbImageFns, dbFeatFn, 'batchSize', batchSize, 'useGPU', true);
serialAllFeats(net, dbTest.qPath, dbTest.qImageFns, qFeatFn, 'batchSize', batchSize, 'useGPU', true);
N = 50;
listN = [1:N];
opts= struct(...
'nTestRankSample', 0, ...
'nTestSample', inf, ...
'recallNs', listN, ...
'margin', 0.1, ...
'nNegChoice', 1000, ...
'cropToDim', 0 ...
);
[recall, ~, ~, opts]= testFromFn(dbTest, dbFeatFn, qFeatFn, opts);

Is there any suggestion for possible fault from my side?
In addition, is it possible to get exact values in Fig. 5 of the paper?

Hi, thanks for your great contribution to place recognition. Here I want to consult some questions about indoor usage.

In the paper, all experiments are conducted in outdoor datasets. I wander what would be the difference if applying
it into a real indoor environment to help re-localization and loop detection in a SLAM system, such as home, official and even some room with repeat region(such as a Server Room where many server computers with the same appearance are placed)? What should I care about when collecting training data without GPS?
Thanks for your attention and I am always looking forward to your kind response and any advice.

Best , Slamer

when the input is (N, W, H, D), the output of NetVLAD is (N, DK), the output of GhostVLAD is also (N, DK), is it correct? It is indicated in GhostVLAD that Take any number of images as input, and output a fixed-length template descriptor to represent the input image set, it should be N vectors aggregated into 1 vector, i.e. input is (N, W, H, D), output is (1, D*K). How can I understand this sentence?

Thanks for sharing netvlad code with us.
Now we plan to learn a light weighted deep network with netvlad algorithm.
How can we get the trainning DB?
Thanks in advance~

Hello

  When I ran demo.m, I just can not find the 'relja_simplenn_move' function. It is  really confusing.

Hello, I just imported the .mat trained network, but I don't know how to pass it an image to obtain the depth estimation, would you please give me an explante?
Thanks!

Attached is the link for dataset http://www.ok.ctrl.titech.ac.jp/~torii/project/repttile/. can you send me the dataset link

Hi,
I am trying to look at the VLAD parameters like w, b, c, ie. the centroid of the
cluster etc. in as in the NetVLAD paper.

I downloaded the best model:

load('vd16_pitts30k_conv5_3_vlad_preL2_intra_white.mat')

display( sprintf( '# layers = %d', size(net.layers,2) ) )

for i=1:size(net.layers,2)
    mat = cell2mat(net.layers(i));
    
    if isfield( mat, 'name' )
        ty = getfield( mat, 'type' ) ;   
        name = getfield( mat, 'name' ) ;
        display( sprintf('%d: %s : %s', i, ty, name ) );
    end
end

Running this script gave:

# layers = 35
1: conv : conv1_1
2: relu : relu1_1
3: conv : conv1_2
4: pool : pool1
5: relu : relu1_2
6: conv : conv2_1
7: relu : relu2_1
8: conv : conv2_2
9: pool : pool2
10: relu : relu2_2
11: conv : conv3_1
12: relu : relu3_1
13: conv : conv3_2
14: relu : relu3_2
15: conv : conv3_3
16: pool : pool3
17: relu : relu3_3
18: conv : conv4_1
19: relu : relu4_1
20: conv : conv4_2
21: relu : relu4_2
22: conv : conv4_3
23: pool : pool4
24: relu : relu4_3
25: conv : conv5_1
26: relu : relu5_1
27: conv : conv5_2
28: relu : relu5_2
29: conv : conv5_3
30: normalize : preL2
32: normalize : vlad:intranorm
34: conv : WPCA

So, I tried to look at layer 30 and 32, but I fail to see the learned weights.
I do see the weights for other layers though.
struct with fields:

>>  cell2mat( net.layers(30) )
        type: 'normalize'
        name: 'preL2'
       param: [1024 1.0000e-12 1 0.5000]
    precious: 0

>>  cell2mat( net.layers(32) )
        type: 'normalize'
        name: 'vlad:intranorm'
       param: [1024 1.0000e-12 1 0.5000]
    precious: 0

Am I missing something?

hello,when i run tiny tokyo time machine datasets,it show assert error in yael_nn file at line 4 and my plateform is window

   Segmentation violation detected at Tue Mar 13 09:44:53 2018

Configuration:
Crash Decoding : Disabled - No sandbox or build area path
Crash Mode : continue (default)
Current Graphics Driver: Unknown software
Current Visual : 0x21 (class 4, depth 24)
Default Encoding : UTF-8
Deployed : false
GNU C Library : 2.19 stable
Host Name : sawei
MATLAB Architecture : glnxa64
MATLAB Entitlement ID: Unknown
MATLAB Root : /usr/local/MATLAB/R2016b
MATLAB Version : 9.1.0.441655 (R2016b)
OpenGL : software
Operating System : Linux 4.4.0-31-generic #50~14.04.1-Ubuntu SMP Wed Jul 13 01:07:32 UTC 2016 x86_64
Processor ID : x86 Family 6 Model 94 Stepping 3, GenuineIntel
Virtual Machine : Java 1.7.0_60-b19 with Oracle Corporation Java HotSpot(TM) 64-Bit Server VM mixed mode
Window System : The X.Org Foundation (11803000), display :0.0

Fault Count: 1

Abnormal termination:
Segmentation violation

Register State (from fault):
RAX = 0000000000000000 RBX = 0000000100000000
RCX = 00007f87aa46d320 RDX = 0000000000000000
RSP = 00007f878aa58830 RBP = 00007f878aa588e0
RSI = 00007f878aa58420 RDI = 0000000000000000

R8 = 0000000000000000 R9 = 00007f86e3a85450
R10 = 00007f86b983a140 R11 = 00000000ab8d61a2
R12 = 00007f86e3382d18 R13 = 0000000000000001
R14 = 00007f86e36dc8a0 R15 = 0000000000000001

RIP = 00007f86d0c82cdc EFL = 0000000000010206

CS = 0033 FS = 0000 GS = 0000

Stack Trace (from fault):
[ 0] 0x00007f86d0c82cdc /usr/local/MATLAB/R2016b/bin/glnxa64/libmwgpu.so+08363228
[ 1] 0x00007f86d0c68b0a /usr/local/MATLAB/R2016b/bin/glnxa64/libmwgpu.so+08256266
[ 2] 0x00007f86d0c66873 /usr/local/MATLAB/R2016b/bin/glnxa64/libmwgpu.so+08247411
[ 3] 0x00007f86d0c669f2 /usr/local/MATLAB/R2016b/bin/glnxa64/libmwgpu.so+08247794
[ 4] 0x00007f86d0c55c94 /usr/local/MATLAB/R2016b/bin/glnxa64/libmwgpu.so+08178836
[ 5] 0x00007f86d3fc979f /home/sawei/matconvnet/matlab/mex/vl_nnconv.mexa64+00034719
[ 6] 0x00007f86d3fc9f99 /home/sawei/matconvnet/matlab/mex/vl_nnconv.mexa64+00036761
[ 7] 0x00007f86d3fc64d8 /home/sawei/matconvnet/matlab/mex/vl_nnconv.mexa64+00021720 mexFunction+00001412
[ 8] 0x00007f879ba66caa /usr/local/MATLAB/R2016b/bin/glnxa64/libmex.so+00175274 mexRunMexFile+00000106
[ 9] 0x00007f879ba5f1a3 /usr/local/MATLAB/R2016b/bin/glnxa64/libmex.so+00143779
[ 10] 0x00007f879ba60345 /usr/local/MATLAB/R2016b/bin/glnxa64/libmex.so+00148293
[ 11] 0x00007f879ad5f8a3 /usr/local/MATLAB/R2016b/bin/glnxa64/libmwm_dispatcher.so+00768163 ZN8Mfh_file16dispatch_fh_implEMS_FviPP11mxArray_tagiS2_EiS2_iS2+00000947
[ 12] 0x00007f879ad6016e /usr/local/MATLAB/R2016b/bin/glnxa64/libmwm_dispatcher.so+00770414 ZN8Mfh_file11dispatch_fhEiPP11mxArray_tagiS2+00000030
[ 13] 0x00007f8797b9a847 /usr/local/MATLAB/R2016b/bin/glnxa64/libmwm_lxe.so+11675719
[ 14] 0x00007f8797b9aaab /usr/local/MATLAB/R2016b/bin/glnxa64/libmwm_lxe.so+11676331
[ 15] 0x00007f8797c00411 /usr/local/MATLAB/R2016b/bin/glnxa64/libmwm_lxe.so+12092433
[ 16] 0x00007f8797526930 /usr/local/MATLAB/R2016b/bin/glnxa64/libmwm_lxe.so+04909360
[ 17] 0x00007f8797528c3c /usr/local/MATLAB/R2016b/bin/glnxa64/libmwm_lxe.so+04918332
[ 18] 0x00007f8797525410 /usr/local/MATLAB/R2016b/bin/glnxa64/libmwm_lxe.so+04903952
[ 19] 0x00007f8797520855 /usr/local/MATLAB/R2016b/bin/glnxa64/libmwm_lxe.so+04884565
[ 20] 0x00007f8797520b69 /usr/local/MATLAB/R2016b/bin/glnxa64/libmwm_lxe.so+04885353
[ 21] 0x00007f879752520d /usr/local/MATLAB/R2016b/bin/glnxa64/libmwm_lxe.so+04903437
[ 22] 0x00007f87975252e2 /usr/local/MATLAB/R2016b/bin/glnxa64/libmwm_lxe.so+04903650
[ 23] 0x00007f879761c688 /usr/local/MATLAB/R2016b/bin/glnxa64/libmwm_lxe.so+05916296
[ 24] 0x00007f879761eb2f /usr/local/MATLAB/R2016b/bin/glnxa64/libmwm_lxe.so+05925679
[ 25] 0x00007f8797a9d10e /usr/local/MATLAB/R2016b/bin/glnxa64/libmwm_lxe.so+10637582
[ 26] 0x00007f8797a64eab /usr/local/MATLAB/R2016b/bin/glnxa64/libmwm_lxe.so+10407595
[ 27] 0x00007f8797a64fb3 /usr/local/MATLAB/R2016b/bin/glnxa64/libmwm_lxe.so+10407859
[ 28] 0x00007f8797a670d9 /usr/local/MATLAB/R2016b/bin/glnxa64/libmwm_lxe.so+10416345
[ 29] 0x00007f8797adfbbe /usr/local/MATLAB/R2016b/bin/glnxa64/libmwm_lxe.so+10910654
[ 30] 0x00007f8797adff4a /usr/local/MATLAB/R2016b/bin/glnxa64/libmwm_lxe.so+10911562
[ 31] 0x00007f879a0043db /usr/local/MATLAB/R2016b/bin/glnxa64/libmwbridge.so+00189403
[ 32] 0x00007f879a00507d /usr/local/MATLAB/R2016b/bin/glnxa64/libmwbridge.so+00192637 _Z8mnParserv+00000829
[ 33] 0x00007f879b021243 /usr/local/MATLAB/R2016b/bin/glnxa64/libmwmcr.so+00537155
[ 34] 0x00007f879b0231ce /usr/local/MATLAB/R2016b/bin/glnxa64/libmwmcr.so+00545230
[ 35] 0x00007f879b023849 /usr/local/MATLAB/R2016b/bin/glnxa64/libmwmcr.so+00546889 _ZN5boost6detail17task_shared_stateINS_3_bi6bind_tIvPFvRKNS_8functionIFvvEEEENS2_5list1INS2_5valueIS6_EEEEEEvE6do_runEv+00000025
[ 36] 0x00007f879b022236 /usr/local/MATLAB/R2016b/bin/glnxa64/libmwmcr.so+00541238
[ 37] 0x00007f879b7e9b49 /usr/local/MATLAB/R2016b/bin/glnxa64/libmwiqm.so+00863049
[ 38] 0x00007f879b7d651c /usr/local/MATLAB/R2016b/bin/glnxa64/libmwiqm.so+00783644 _ZN5boost6detail8function21function_obj_invoker0ISt8functionIFNS_3anyEvEES4_E6invokeERNS1_15function_bufferE+00000028
[ 39] 0x00007f879b7d61fc /usr/local/MATLAB/R2016b/bin/glnxa64/libmwiqm.so+00782844 _ZN3iqm18PackagedTaskPlugin7executeEP15inWorkSpace_tagRN5boost10shared_ptrIN14cmddistributor17IIPCompletedEventEEE+00000428
[ 40] 0x00007f879b7b5a0a /usr/local/MATLAB/R2016b/bin/glnxa64/libmwiqm.so+00649738
[ 41] 0x00007f879b7a1690 /usr/local/MATLAB/R2016b/bin/glnxa64/libmwiqm.so+00566928
[ 42] 0x00007f879b7a4048 /usr/local/MATLAB/R2016b/bin/glnxa64/libmwiqm.so+00577608
[ 43] 0x00007f87ab95240a /usr/local/MATLAB/R2016b/bin/glnxa64/libmwservices.so+02634762
[ 44] 0x00007f87ab9539af /usr/local/MATLAB/R2016b/bin/glnxa64/libmwservices.so+02640303
[ 45] 0x00007f87ab9540e6 /usr/local/MATLAB/R2016b/bin/glnxa64/libmwservices.so+02642150 _Z25svWS_ProcessPendingEventsiib+00000102
[ 46] 0x00007f879b0218c6 /usr/local/MATLAB/R2016b/bin/glnxa64/libmwmcr.so+00538822
[ 47] 0x00007f879b021c42 /usr/local/MATLAB/R2016b/bin/glnxa64/libmwmcr.so+00539714
[ 48] 0x00007f879b00f8d6 /usr/local/MATLAB/R2016b/bin/glnxa64/libmwmcr.so+00465110
[ 49] 0x00007f87aa25c184 /lib/x86_64-linux-gnu/libpthread.so.0+00033156
[ 50] 0x00007f87a9f88ffd /lib/x86_64-linux-gnu/libc.so.6+01040381 clone+00000109
[ 51] 0x0000000000000000 +00000000

This error was detected while a MEX-file was running. If the MEX-file
is not an official MathWorks function, please examine its source code
for errors. Please consult the External Interfaces Guide for information
on debugging MEX-files.

If this problem is reproducible, please submit a Service Request via:
http://www.mathworks.com/support/contact_us/

A technical support engineer might contact you with further information.

Thank you for your help.

Thanks a lot!

error on this sentence:
ims= cat(4, ims_{:});

for images have different dimensions, there will be an error.
In my experiments, the size of ims_{1} is 1024x683 while the size of ims_{2} is 683x1024
Could you give me some suggestions? Thanks a lot.

In file: netvlad/yael_dummy/yael_nn.m
ds= sum( bsxfun(@minus, v, q(:,1)).^2, 1 );
Should it be as following?
ds= sum( bsxfun(@minus, v, q(:,iVec)).^2, 1 );

hi Relja,
I want to run this Project, I use Centos, so I want to know your proj can run on the octave ?

thx !

It seems that the latest MatConvNet versions introduced some minor breaks in this code (see issue raised vlfeat/matconvnet#552 ), so I suggest using versions from 1.0-beta18 to 1.0-beta20 until this issue is resolved (either by MatConvNet or by myself). I'm quite sure I used 1.0-beta18 and 1.0-beta19, so those should work.

Thanks for your amazing work. I have sent you an email for the dataset.

Hi @Relja,

NetVLAD is trained on real-world images, but I was wondering how effective your pre-trained models would work in simulation camera images (like in a Gazebo environment). To be more specific, my Gazebo world has simple obstacles like bookshelves, garbage cans and walls. A sample camera image taken from my robot in the simulation is shown here:

To your knowledge, has anyone tried this out? If so, what do you think are the best pre-trained models to try out? I'm trying to avoid training a custom model since building a dataset is super tedious in my instance.

Thanks for your response!
Quest2GM

Hi
Can we use the trained model in the real-time application? and which GPU can be used in real-time?

The input images in your codebase are normalised as follows:
im(:,:,1)= im(:,:,1) - net.meta.normalization.averageImage(1,1,1);
im(:,:,2)= im(:,:,2) - net.meta.normalization.averageImage(1,1,2);
im(:,:,3)= im(:,:,3) - net.meta.normalization.averageImage(1,1,3);

and while averageImage is a matrix (of size [224,224,3] for vgg16 and [227,227,3] for AlexNet) only the first pixel's RGB values are used. Luckily, for the models using vgg16 this is not a problem, because all pixels have the same RGB value. However, for the AlexNet models this is not the case:

metadata = load('caffe_pitts30k_conv5_vlad_preL2_intra.mat');
mean(mean(metadata.net.meta.normalization.averageImage, 1), 2)

ans(:,:,1) = 122.6769
ans(:,:,2) = 116.6700
ans(:,:,3) = 104.0102

metadata.net.meta.normalization.averageImage(1,1,:)

ans(:,:,1) = 117.3785
ans(:,:,2) = 117.6438
ans(:,:,3) = 110.1771

If done consistently between train and test its probably not a major issue, but it might affect generalisation performance. I didn't have a chance to test it (nor a proper setup to do it), but it seems worthwhile (if I didn't miss anything) to retrain and reevaluate the AlexNet models with correct normalization.