Git Product home page Git Product logo

intelligentcarpet's Introduction

IntelligentCarpet: Inferring 3D Human Pose from Tactile Signals

alt text

Yiyue Luo, Yunzhu Li, Michael Foshey, Wan Shou, Pratyusha Sharma, Tomás Palacios, Antonio Torralba, and Wojciech Matusik

CVPR 2021 [Project Page] [Paper] [Video]

Quick Start

git clone https://github.com/yiyueluo/intelligentCarpet  
cd IntelligentCarpet   
conda env create -f environment.yml   
conda activate p36   

Dataset

Raw dataset on 10 people recorded on three days (including camera calibration, which happens once per day): https://www.dropbox.com/sh/g3l4jdablczffj3/AACuFy9E2YonQdNjUu4beClta?dl=0

In each dataset folder, there are:

  1. raw tactile signal (including timestamp) on 9 pieces of tiles: touch#.hdf5
  2. aligned tactile signal as 96x96 matrix: touch_aligned.p
  3. normalized tactile signal : touch_normalized.p
  4. video recorded by 2 calibrated cameras: webcam#.mp4, and corresponding timestamps: webcam#.txt
  5. visualization with Openpose output (only in selective folder for demonstration): webcam#_openpose.avi
  6. 21 keypoints output by Openpose from each camera: pt_webcam#.mat
  7. triangulated 21 keypoints: keypoint3D.mat
  8. optimized 3D keypoints: keypoint_refined.p
  9. transformed 3D keypoints corresponding to the carpet perspective: keypoint_transform.p, and the positions of the carpet: tile_transform.p

In each calibration folder, there are:

  1. calibration parameters for each camera: webcam#_intrinsic.mat, webcam#_extrinsic.mat, webcam#_dis.mat
  2. carpet positions corresponding to each camera

The important files are: touch_normalized.p, keypoint_refined.p (they are aligned with timestamps). Use heatmap_from_keypoint3D.py to generate GT 3D heatmap (20x20x18).

Demo

Checkpoints and test dataset can be found here: https://www.dropbox.com/sh/5l0lm4po64xf6jd/AACuMt_oGy99Beyz_IMeknQ6a?dl=0

  • ckpts.zip contains the trained model
  • singlePerson_test.zip contains the test set for single person pose estimation
  • singlePerson_test_diffTask.zip contains the test set for single person pose estimation, which is arranged by individual tasks (Note: use sample_data_diffTask to load data)

Download the checkpoint and desired test set to "./train/" and unzip to its correspondng folder.

python ./train/threeD_train_final.py

To visualize predictions, set --exp_image or --exp_video.
To export L2 distance between predicted skeleton and groundtruth , set --exp_L2.
To export data on tactile input, groundtruth keypoint, grondtruth heatmap, predicted keypoint, predicted heatmap, set --exp_data.

Note: pay attention to the used checkpoints --ckpts, experiment path --exp_dir and test data path --test_dir.

Contact

If you have any questions about the paper or the codebase, please feel free to contact [email protected].

intelligentcarpet's People

Contributors

yiyueluo avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.