inference guide for retrieval model

• Add competition metric (MRR)
• Create abstract metric class for retrieval purpose (as good as possible)
• Add new auto tests

An visualize tool could explain extracted srl meta data

The implement metric only calculates the results on 1 batch and averages.

We need to call metric.calculate at the on_eval_end step to perform the search on the entire dataset. Currently, the similarity search method only works in 1 batch, so the top 5 is quite high (compared to the random percent of 5/8). Please correct by append embedding to the list on on_val_step model hook

• The uuids might be changed when resuming training, which can hurt the performance of model (affect the instance loss)

• Part of code that produces this error:

class CityFlowNLDataset(Dataset):
      ....
      self.list_of_uuids = list(tracks.keys())  # the order can be different when resuming
      self.list_of_tracks = list(tracks.values())
      ...
      self.all_indexs = list(range(len(self.list_of_uuids)))
def __getitem__(self, index):
      tmp_index = self.all_indexs[index] # then this will be used as target for instance loss
      ...

• Adapt module (tracking, detect veh) from hcmus 2021 source code
• Add document
• Add tests

Merge srl module, add missing follow category (follow by)

display multiple loss for training process analyst purpose

• This module extracts relation graph between tracks, the outputs determine which tracks follows or is_followed by other tracks.

• This module is for post process, helps filter out retrieval prediction
• Should be used on test tracks to determine which track has turn or stop actions

remove (or comment) # data[key] += json_data[key]['nl_other_views'] in external/extraction/textual/data_helper.py

Improve baseline model using more data

automatic testing

Add dockerfile to running on server

ModuleNotFoundError when running scripts/srl/extract_postproc.py script

Suggestions:

• Add 2 dataset/dataloader for query text features and visual features (motion + crop).

• Add a new class for inference (nn.Module or pl.LightningModule), which can load in checkpoint and has two functions: encoding texts and encoding visual

• Finally, use Faiss to calculate similarity, save results to file for submission and visualization

• Small example snippet

@torch.no_grad()
def inference(self):
    self.model.eval()

    # Extract lang feats
    lang_results = {}
    for idx, batch in enumerate(self.lang_dataloader):
        batch = move_to(batch, self.device)
        lang_feats = self.model.encode_nlang_feats(batch)
        lang_feats = move_to(lang_feats, torch.device('cpu')).detach().numpy()
        ids = batch['ids']
        for lang_id, lang_feat in zip(ids, lang_feats):
            lang_results[lang_id] = lang_feat.tolist()
        
    with open(osp.join(self.savedir, 'text_embeds.json'), 'w') as f:
        json.dump(lang_results, f)

    # Extract visual feats
    visual_results = {}
    for idx, batch in enumerate(self.visual_dataloader):
        batch = move_to(batch, self.device)
        visual_feats = self.model.encode_visual_feats(batch, inference=True)
        visual_feats = move_to(visual_feats, torch.device('cpu'))
        visual_feats = visual_feats.detach().numpy()
        ids = batch['ids']
        for visual_id, visual_feat in zip(ids, visual_feats):
            visual_results[visual_id] = visual_feat.tolist()
        
    with open(osp.join(self.savedir, 'visual_embeds.json'), 'w') as f:
        json.dump(visual_results, f)

    # Faiss retrieval
    retriever = FaissRetrieval(dimension=self.dimension)
    query_embeddings = np.stack(lang_results.values(), axis=0).astype(np.float32)
    gallery_embeddings = np.stack(visual_results.values(), axis=0).astype(np.float32)
    query_ids = list(lang_results.keys())
    gallery_ids = list(visual_results.keys())

    retriever.similarity_search(
        query_embeddings,
        gallery_embeddings,
        query_ids,
        gallery_ids,
        top_k=self.top_k,
        save_results=osp.join(self.savedir, 'retrieval_results.json'))

• Adding new version of hcmus visualize tool
• Update document (input format, convert scripts)

query_dm = WrapperDataModule(self.query_ds, batch_size=1)
track_dm = WrapperDataModule(self.track_ds, batch_size=1)

change to batch_size=self.batchsize

• Add logging callback (support visualize query - tracklet/image)
• Usage of logging callback