Git Product home page Git Product logo

junweizheng93 / materobot Goto Github PK

View Code? Open in Web Editor NEW
12.0 4.0 0.0 309 KB

Official repository for paper "MATERobot: Material Recognition in Wearable Robotics for People with Visual Impairments" at ICRA 2024, Best Paper Finalist on Human-Robot Interaction

Home Page: https://arxiv.org/pdf/2302.14595.pdf

License: Apache License 2.0

Python 99.83% Shell 0.17%
material-recognition wearable-robot moe multi-task-learning real-time-vit

materobot's Introduction

MATERobot: Material Recognition in Wearable Robotics for People with Visual Impairments

Project Homepage

This project has been selected as the Best Paper Finalist on Human-Robot Interaction at ICRA 2024! For more information about the project, please refer to our project homepage.

Setup

# create virtual environment
conda create -n materobot python=3.8 -y
conda activate materobot
# install PyTorch
conda install pytorch==1.12.1 torchvision==0.13.1 cudatoolkit=11.6 -c pytorch -c conda-forge -y
# install other packages
pip install -U openmim
mim install mmengine==0.4.0
mim install mmcv==2.0.0rc3
pip install -r requirements.txt

Prepare datasets

The overall folder structure is shown below:

MATERobot
├── mmseg
├── materobot
├── pretrain
├── requirements
├── tools
├── data
│   ├── DMS
│   │   ├── images
│   │   │   ├── training
│   │   │   ├── validation
│   │   │   ├── test
│   │   ├── annotations
│   │   │   ├── training
│   │   │   ├── validation
│   │   │   ├── test
│   ├── coco_stuff10k
│   │   ├── images
│   │   │   ├── train2014
│   │   │   ├── test2014
│   │   ├── annotations
│   │   │   ├── train2014
│   │   │   ├── test2014
│   │   ├── imagesLists
│   │   │   ├── train.txt
│   │   │   ├── test.txt
│   │   │   ├── all.txt

DMS

Download DMS dataset using tools/dms_download_tools/download_DMS.py:

python tools/dms_download_tools/download_DMS.py /home/usr_name/project_root/data

After downloading, prepare the DMS dataset according to its official GitHub repository. You need to solve all problems by yourself according to the preparation_outcomes.json and image_issues.json. Make sure every label has its corresponding image (the number of labels should be equal to the number of images).

Finally, prepare DMS labels in order to get the desired DMS46 dataset:

python tools/dms_download_tools/prepare_DMS_labels.py /home/usr_name/project_root/data/DMS_v1

COCO-Stuff10k

Please prepare coco-stuff10k according to this page.

Prepare pretrained backbone

Please download the pretrained backbone model here and place the model to pretrain folder under the project root.

Usage

Train

Single-task Model

# Command: bash tools/dist_train.sh config/file/path num_gpus
bash tools/dist_train.sh materobot/configs/matevit_vit-t_single-task_dms.py 4

Multi-task Model

# Command: bash tools/dist_train.sh config/file/path num_gpus
bash tools/dist_train.sh materobot/configs/matevit_vit-t_multi-task.py 4

Test

Single-task Model

# Command: bash tools/dist_test.sh config/file/path checkpoint/file/path num_gpus
bash tools/dist_test.sh materobot/configs/matevit_vit-t_single-task_dms.py work_dirs/matevit_vit-t_single-task_dms/best_mIoU_epoch_100.pth 4

Multi-task Model

Since there is only one kind of gt label for each input data, you need to modify the dataset before running the test script:

# dataset before modification in materobot/configs/matevit_vit-t_multi-task.py:
_base_ = [
    './_base_/models/matevit_multi-task.py',
    './_base_/datasets/dms_coco.py', './_base_/default_runtime.py',
    './_base_/schedules/schedule_200epochs.py'
]

# if you want to test on DMS dataset:
_base_ = [
    './_base_/models/matevit_multi-task.py',
    './_base_/datasets/dms.py', './_base_/default_runtime.py',
    './_base_/schedules/schedule_200epochs.py'
]

# if you want to test on COCO-Stuff10k dataset:
_base_ = [
    './_base_/models/matevit_multi-task.py',
    './_base_/datasets/coco-stuff10k.py', './_base_/default_runtime.py',
    './_base_/schedules/schedule_200epochs.py'
]

After the modification, you can run the following command:

# Command: bash tools/dist_test.sh config/file/path checkpoint/file/path num_gpus
bash tools/dist_test.sh materobot/configs/matevit_vit-t_multi-task.py work_dirs/matevit_vit-t_multi-task/best_mIoU_epoch_200.pth 4

Inference

Please refer to inference_demo.py

Checkpoints

Download from here

Citation

If you are interested in this work, please cite as below:

@inproceedings{zheng2024materobot,
title={MATERobot: Material Recognition in Wearable Robotics for People with Visual Impairments},
author={Zheng, Junwei and Zhang, Jiaming and Yang, Kailun and Peng, Kunyu and Stiefelhagen, Rainer},
booktitle={2024 IEEE International Conference on Robotics and Automation (ICRA)},
year={2024}
}

materobot's People

Contributors

junweizheng93 avatar

Stargazers

avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar

Watchers

avatar avatar avatar avatar

materobot's Issues

Pretrained Models

Hi

Do you have any link for the pretrained model(s) to run the inference using them? I wanted to test some images with material segmentation before running any training.

Best
Don

Welcome update to OpenMMLab 2.0

Welcome update to OpenMMLab 2.0

I am Vansin, the technical operator of OpenMMLab. In September of last year, we announced the release of OpenMMLab 2.0 at the World Artificial Intelligence Conference in Shanghai. We invite you to upgrade your algorithm library to OpenMMLab 2.0 using MMEngine, which can be used for both research and commercial purposes. If you have any questions, please feel free to join us on the OpenMMLab Discord at https://discord.gg/A9dCpjHPfE or add me on WeChat (ID: van-sin) and I will invite you to the OpenMMLab WeChat group.

Here are the OpenMMLab 2.0 repos branches:

OpenMMLab 1.0 branch OpenMMLab 2.0 branch
MMEngine 0.x
MMCV 1.x 2.x
MMDetection 0.x 、1.x、2.x 3.x
MMAction2 0.x 1.x
MMClassification 0.x 1.x
MMSegmentation 0.x 1.x
MMDetection3D 0.x 1.x
MMEditing 0.x 1.x
MMPose 0.x 1.x
MMDeploy 0.x 1.x
MMTracking 0.x 1.x
MMOCR 0.x 1.x
MMRazor 0.x 1.x
MMSelfSup 0.x 1.x
MMRotate 0.x 1.x
MMYOLO 0.x

Attention: please create a new virtual environment for OpenMMLab 2.0.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.