Light

facebookresearch / you2me Goto Github PK

View Code? Open in Web Editor NEW

49.0 9.0 9.0 26.29 MB

Inferring Body Pose in Egocentric Video via First and Second Person Interactions

License: Other

MATLAB 6.95% Shell 2.45% C++ 16.68% Python 73.92%

you2me's Introduction

You2Me: Inferring Body Pose in Egocentric Video via First and Second Person Interactions (CVPR 2020)

Install

Download dataset

Original training done with CUDA 10.2

Install basic dependencies with pip install -r requirements.txt

Test

Please generate:

directory of homographies (see calc_homgraphy/README.md)
directory of openpose predictions
vocab.pkl (see vocab/build_vocab.py)

for your sample sequence.

Then run the following command:

python sample.py --vocab_path <path/to/sample_vocab.pkl> --output <path/to/output_dir> --encoder_path <path/to/trained/encoder.pth> --decoder_path <path/to/trained/decoder.pth> --upp

Change flag --upp to --low to test the lower body model.

Include flag --visualize to plot the predicted stick figures.

Train

Please generate

directory of homographies (see calc_homgraphy/README.md)
directory of openpose predictions
vocab.pkl (see vocab/build_vocab.py)
annotation.pkl (see vocab/build_annotation.py)

for your each of your training sequences.

Then run the following command:

python train.py --model_path <path/to/save/models> --vocab_path <path/to/train_vocab.pkl> --annotation_path <path/to/annotation.pkl> -upp

Change flag --upp to --low to train the lower body model.

License

CC-BY-NC 4.0. See the LICENSE file.

Citation

@article{ng2019you2me,
  title={You2Me: Inferring Body Pose in Egocentric Video via First and Second Person Interactions},
  author={Ng, Evonne and Xiang, Donglai and Joo, Hanbyul and Grauman, Kristen},
  journal={CVPR},
  year={2020}
}

you2me's People

Contributors

Stargazers

Watchers

Forkers

sanyam-git daydreamer2023 nudlesoup peterzs peterzhousz qimaqi wenjunjiang jackzhousz

you2me's Issues

Wrong egopose estimation

Hi, I would like to use your code to estimate egopose in egocentric videos, but estimations aren't accurate and I don't know what I'm doing wrong. Firstly I used KMeans to cluster lower and upper body egoposes gt in kinect/*/synchronized/gt-egopose and used them in build_vocab.py (by keeping the order of the sorted kmeans clusters). Then I used build_annotation.py to choose the training set and I chose every folder except sport56, sport57, sport58. So I trained the network for upper and lower body pose, but when I test it, I get a very strange estimation:

Do you have any idea about why I am facing this problem? Or which step should I check? Every tip would be very appreciated! Thank you.

How can we arrange the dataset to train the model?

Hi, thank you for the code and contribution.

Could you please tell me how we can arrange the datasets (kinect, panoptic)? I am not understanding well how we should create the directories and subdirectories of homographies, openpose, and images, because, when I run train.py it asks me for sequences that do not exist in the dataset. I am trying with the panoptic dataset.

Thank you.

Where can we find the weights of the trained model?

Hi, thanks for the code and paper. Is it possible to get the weights of the trained model?

Thank you

How to generate directory of openpose predictions

In the Test and Train the readme.md file asks to generate directory of openpose predictions. Can you please tell how to do that ?

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.