Training data about stylegestures HOT 53 CLOSED

@evgenytumanov , sometimes length is different, but all the files are synchronized and t=0 is indeed always the same.

from stylegestures.

Hi @evgenytumanov,

@simonalexanderson knows a lot more about the specifics of the data than I do, but my understanding is that the synchronisation in the original Trinity Speech-Gesture Dataset occasionally has problems. Personally, I would recommend that you train your system on the data from the GENEA Challenge instead. That data is a processed version of the original Trinity material, and was recently made available in the same repository and is covered by the same license agreement. The challenge dataset has been preprocessed to fix synchronisation issues and similar, and to create a canonical train and test split for the data (you will have to create your own dev set from the training portion, though).

To read more about the GENEA Challenge and its data, please see our first paper made available on Zenodo, which we uploaded earlier today. Some colleagues and I are also arranging a workshop associated with the GENEA Challenge on Sunday (two days from now), and a major part of the workshop programme consists of presentations and discussions of different gesture-generation systems trained on the GENEA Challenge data. If you're interested, registration is open at no cost at the GENEA Workshop 2020 website.

from stylegestures.

Hi @SDNAFIO, and thank you for your interest in our work!

The gesture data is available at http://trinityspeechgesture.scss.tcd.ie/ . Trinity College Dublin require interested parties to sign a license agreement and receive approval before gaining access the material, so we cannot host it here.

The Edinburgh locomotion data and associated processing code can be obtained at https://bitbucket.org/jonathan-schwarz/edinburgh_locomotion_mocap_dataset . We are in contact with Edinburgh regarding sharing our small extension of that material (different frame rate and trajectory smoothing), but we do not yet know what the eventual decision will be.

Finally, the quadruped data is linked from GitHub at https://github.com/sebastianstarke/AI4Animation . A direct link to the data is http://www.starke-consult.de/UoE/GitHub/SIGGRAPH_2018/MotionCapture.zip , but please also see the associated content on GitHub for license information etc.

UPDATE: Our pre-processed version of the human locomotion data can be found here:
https://kth.box.com/s/quh3rwwl2hedwo32cdg1kq7pff04fjdf

from stylegestures.

Hi,
The last features in the Holden et al. code are related to foot contacts and not needed for MoGlow. You should only use the joint positions and the 3 root motion features. If I’m not mistaking (I don’t have the code in front of me right now) that should sum up to 66. Hope this answers your question!

it helps! thanks a lot.

from stylegestures.

For the record, the run-time issues that @wangzheliu posted about in this thread are not related to the training data and were solved in a separate issue, issue #5. If you are having similar issues, look there first.

from stylegestures.

Hi @saeed1262,

Is there a specific reason that you used joint positions instead of joint angles? I guess one reason could be the topology of joint angle space which might not be compatible with the current architecture.

Good question. From my personal perspective and my possibly fallible recollection, I would say that the reasons for using joint angles were 1) that's what our starting dataset, the Edinburgh Locomotion MOCAP Database from Habibie et al. (2017), and its associated processing code used, 2) the topological concerns you mention, and 3) inexperience on my part as to the importance of using joint angles in graphics applications, especially as we were targetting non-graphics conferences at first. However, it turns out that joint angles with an appropriate parameterisation work equally well, and that's what we used for the gesture-generation paper at EUROGRAPHICS and its corresponding code in this repo.

Within a month there will be a substantial update to the MoGlow paper on arXiv and its associated demonstration video on YouTube to present more experiments, including strong results on locomotion data parameterised using joint angles, without using any postprocessing of the raw output (in contrast to many existing motion-generation systems). The code will also be updated to match. Please e-mail us if you wish to read the revised paper before it becomes available on arXiv.

Also, is this possible to have preprocessed data represented by joint angles?

The GENEA Challenge data that @Svito-zar mentioned in a comment three days ago is a preprocessed dataset of gesture motion represented by joint angles. It can be downloaded from a subfolder of the Trinity Speech-Gesture Dataset server after signing their license agreement and receiving approval.

from stylegestures.

I can add that another reason for usíng joint positions was to assess model capabilities without the extra foot-sliding you get from angle errors propagating the hierarchy (small errors at the upper leg can cause the foot to move substantially). But as Gustav mentioned above, our later experiments using joint angles did not show such problems.

from stylegestures.

Unfortunately, I do not know the specifics of how the quadruped data was preprocessed. I think @simonalexanderson is the only person who can answer that.

from stylegestures.

Thanks for the quick response!

I'm particularly interested in training with the locomotion dataset.
So to my understanding, the Edinburgh locomotion data, as it is available for download, is recorded with 60Hz, while you used a variation with 20Hz (according to "framerate": 20 in locomotion.json).
Is it sufficient to adapt the framerate setting to 60Hz to obtain feasible results with the available dataset, or are some further changes needed?

from stylegestures.

Hello again,

The framerate setting only points to which dataset file to use. These files should be called "all_locomotion_train_XXfps.npz" and "all_locomotion_test_XXfps.npz" and reside in the data directory specified in the locomotion.json file (see ./motion/datasets/locomotion.py). To get going, you can download the 60 fps Edinburgh datasets, rename them to all_locomotion_txxxx_60fps.npz and set the json framerate to 60 fps (or better up – resample the data to 20 fps and rename the files accordingly).

Aside from the difference in frame rate, we applied more smoothing the motion of the root node (on the floor underneath the character), to remove footfall cues such as periodicity in the root-node motion. This is optional, but we believe it makes MoGlow models generalise better. We also augmented the data with the locomotion trials from the CMU and HDM05 databases. To reproduce our processing, you need to download the data at http://theorangeduck.com/media/uploads/other_stuff/motionsynth_data.zip and modify the processing script called export.py. We 1) pooled data, 2) downsampled to 20 fps, and 3) applied smoothing with 5 frames on the trajectory position and 10 frames for the trajectory direction. See the MoGlow manuscript on arXiv at https://arxiv.org/abs/1905.06598 for more details.

Hope this helps!

from stylegestures.

Here is a link to our pre-processed version of the locomotion data. Please see the included README for details.

https://kth.box.com/s/quh3rwwl2hedwo32cdg1kq7pff04fjdf

from stylegestures.

@SDNAFIO , has your question been answered?

from stylegestures.

Hello again,

The framerate setting only points to which dataset file to use. These files should be called "all_locomotion_train_XXfps.npz" and "all_locomotion_test_XXfps.npz" and reside in the data directory specified in the locomotion.json file (see ./motion/datasets/locomotion.py). To get going, you can download the 60 fps Edinburgh datasets, rename them to all_locomotion_txxxx_60fps.npz and set the json framerate to 60 fps (or better up – resample the data to 20 fps and rename the files accordingly).

Aside from the difference in frame rate, we applied more smoothing the motion of the root node (on the floor underneath the character), to remove footfall cues such as periodicity in the root-node motion. This is optional, but we believe it makes MoGlow models generalise better. We also augmented the data with the locomotion trials from the CMU and HDM05 databases. To reproduce our processing, you need to download the data at http://theorangeduck.com/media/uploads/other_stuff/motionsynth_data.zip and modify the processing script called export.py. We 1) pooled data, 2) downsampled to 20 fps, and 3) applied smoothing with 5 frames on the trajectory position and 10 frames for the trajectory direction. See the MoGlow manuscript on arXiv at https://arxiv.org/abs/1905.06598 for more details.

Hope this helps!

hi @ghenter, thanks for your detailed description here. I'm little confused about the export.py's results. It's have 73 dims feature per frame, but 66 dims you offered in https://kth.box.com/s/quh3rwwl2hedwo32cdg1kq7pff04fjdf . Could you tell me what am i missing or offer an preprocess script for the raw data? Thanks!

from stylegestures.

Hi,
The last features in the Holden et al. code are related to foot contacts and not needed for MoGlow. You should only use the joint positions and the 3 root motion features. If I’m not mistaking (I don’t have the code in front of me right now) that should sum up to 66. Hope this answers your question!

from stylegestures.

Hi,
The last features in the Holden et al. code are related to foot contacts and not needed for MoGlow. You should only use the joint positions and the 3 root motion features. If I’m not mistaking (I don’t have the code in front of me right now) that should sum up to 66. Hope this answers your question!

it helps! thanks a lot.

hi,Did you run this program successfully,I run the program,but show some wrong,

(moglow) [liu@no3@node02 StyleGestures]$ python train_moglow.py '/hparams/locomotion.json' locomotion
File "train_moglow.py", line 49
trainer = Trainer(**built, data=data, log_dir=log_dir, hparams=hparams)
^
SyntaxError: invalid syntax
Do you meet same wrong?I use pytorch1.5,python3.7.thank you very much.

from stylegestures.

Hi,
Did anyone successfully get access to gesture data ?
I submitted my request but they don't reply for a week...

from stylegestures.

Hi,
Did anyone successfully get access to gesture data ?
I submitted my request but they don't reply for a week...

me too

from stylegestures.

Hi,
The last features in the Holden et al. code are related to foot contacts and not needed for MoGlow. You should only use the joint positions and the 3 root motion features. If I’m not mistaking (I don’t have the code in front of me right now) that should sum up to 66. Hope this answers your question!

it helps! thanks a lot.

hi,Did you run this program successfully,I run the program,but show some wrong,

(moglow) [liu@no3@node02 StyleGestures]$ python train_moglow.py '/hparams/locomotion.json' locomotion
File "train_moglow.py", line 49
trainer = Trainer(**built, data=data, log_dir=log_dir, hparams=hparams)
^
SyntaxError: invalid syntax
Do you meet same wrong?I use pytorch1.5,python3.7.thank you very much.

Your path to the json file is incorrect, try ’hparms/locomotion.json’ (without leading ‘/’). Python 3.7 and pytorch 1.5 should be fine. Hope this helps.

from stylegestures.

Hi,
The last features in the Holden et al. code are related to foot contacts and not needed for MoGlow. You should only use the joint positions and the 3 root motion features. If I’m not mistaking (I don’t have the code in front of me right now) that should sum up to 66. Hope this answers your question!

it helps! thanks a lot.

hi,Did you run this program successfully,I run the program,but show some wrong,
(moglow) [liu@no3@node02 StyleGestures]$ python train_moglow.py '/hparams/locomotion.json' locomotion
File "train_moglow.py", line 49
trainer = Trainer(**built, data=data, log_dir=log_dir, hparams=hparams)
^
SyntaxError: invalid syntax
Do you meet same wrong?I use pytorch1.5,python3.7.thank you very much.

Your path to the json file is incorrect, try ’hparms/locomotion.json’ (without leading ‘/’). Python 3.7 and pytorch 1.5 should be fine. Hope this helps.

(base) [liu@no3@node05 StyleGestures]$ python train_moglow.py 'hparams/locomotion.json' locomotion
File "train_moglow.py", line 49
trainer = Trainer(**built, data=data, log_dir=log_dir, hparams=hparams)
^
SyntaxError: invalid syntax
I run the program On your advice，but show same wrong,Do you have the same problem

from stylegestures.

Hi,
Did anyone successfully get access to gesture data ?
I submitted my request but they don't reply for a week...

Hi,
Yes I did. I filled the request form and they provided a username and password. However, the dataset has missing part. Also, the moglow doesn't have pre-processing for the gesture dataset.
Is anyone able to provide some hints for preprocessing trinity?
Thanks!

from stylegestures.

the dataset has missing part

@pjyazdian, out of interest, what specific parts/aspects are you missing in the data? If you believe data was incorrectly omitted you may contact the dataset provider about that.

moglow doesn't have pre-processing for the gesture dataset. Is anyone able to provide some hints for preprocessing trinity?

Regarding processing the Trinity gesture data, we are looking to provide preprocessing guidelines and code in the near future, but that will be some time after our team members are back from their summer holidays. Until then, you will find a description of what we view as the most important preprocessing steps in this comment on issue #2.

from stylegestures.

@pjyazdian Now the data processing and guidelines are available, please see the updated README. Let me know if something is unclear or you run into problems processing the data.

from stylegestures.

hi guys ,who have datasets The gesture data is available at http://trinityspeechgesture.scss.tcd.ie/.please give me one thanks!!!
[email protected]

from stylegestures.

Dear @yutao007,

The gesture dataset at the URL you provided is protected by a license agreement that restricts distribution. You may only obtain it by agreeing to that license agreement and downloading it from the Trinity Speech-Gesture Dataset site, and not via e-mail from third parties.

from stylegestures.

thanks,i have register ,but still not received

from stylegestures.

I see. My understanding is that registrations are processed manually at Trinity College Dublin, so I cannot tell when you will be given access, but I hope it will be soon. Unfortunately, we cannot send the material to you since we don't have proof of you signing the agreement, nor do we have authorisation from the dataset creators to share the data with others.

from stylegestures.

you are right .can you send model to me , i just play demo like youtobe

from stylegestures.

sorry ,i just say when you can open litt;e model you have trained . wo can play demo .thanks again .

from stylegestures.

Providing a pre-trained model is issue #4 in this repository. We are planning to package and upload such a model here in the near future, but it is not feasible to do that right now because of some imminent deadlines on our side.

from stylegestures.

thanks again .hope it quickly.

from stylegestures.

Trinity Speech-Gesture Dataset contains now also the GENEA Challenge data release, which has motions in BVH format.

I think this issue can be closed now.

from stylegestures.

Yes, although we might want to update the readme slightly to reflect this (e.g., that the processed data is available in a specific subfolder on the TCD server) and also check that the code in the repo actually is compatible with the data as released (e.g., does not assume that the data is unprocessed).

from stylegestures.

Hi,

Is there a specific reason that you used joint positions instead of joint angles? I guess one reason could be the topology of joint angle space which might not be compatible with the current architecture.
Also, is this possible to have preprocessed data represented by joint angles?
Thanks

from stylegestures.

Thanks for the detailed response, Gustav and Simon.

from stylegestures.

Hi. Can you please explain how you preprocessed the quadruped data?

from stylegestures.

Hi.
@simonalexanderson, I'm trying to replicate your results on Trinity data. I'm quite confused with audio-animation synchronization. Current data on https://trinityspeechgesture.scss.tcd.ie/ is located in four folders
BVH_from_clap/ , FBXs/ , FBXs_from_clap/ and Audio

I guess, that the right way to synchronize things is
Convert FBXs/ to BVHs/ somehow, and then use synchronize_trinity.py to synchronize the data. It was your pipeline I guess. Is it true?

Another way I guess to get pretty much the same is
Use straightaway BVH_from_clap/ and Audio/ . They look synchronized. But I need to cut some scenes according to information in offsets.csv. Is this statement true?

from stylegestures.

@ghenter I'm looking at the GENEA data and cannot figure out why audio and corresponding motion file are not of the same length. Training data, Recording_001.wav is 12:02 long but the Recording_001.bvh is 11:58 long. Why?

I guess the t=0 is the same moment for these files, is it right?

from stylegestures.

Ok, thank you, guys!

from stylegestures.

Hi. Can you please explain how you preprocessed the quadruped data?

Hi. Any clue regarding this question?

from stylegestures.

Hi @ghenter and @simonalexanderson ,
Do you have the list of samples extracted from CMU and HDM05 for locomotion data?

Thanks

from stylegestures.

Hi @saeed1262,
Here are the lists of the files we used for the evaluations, inculding the samples we used from the Edinburgh Locomotion Dataset. Note that the final MoGlow paper also has visualizations with skinned characters. For those we used the LAFAN1 and Kinematica datasets.

train_files.txt
test_files.txt

from stylegestures.

Thanks, @simonalexanderson!
That helps a lot.
Can I know why for the joint angles you used LAFAN1 instead of what you used for joint positions?

from stylegestures.

This was due to the higher data quality of LAFAN1 and Kinematica. Joint angle representations cause more footsliding than joint positions, as output errors sum up through the skeleton hierarchy. We found that the pooled LAFAN1 and Kinematica data caused less footsliding than the pooled CMU, HDM05 and Edinburgh data. Note that results in our videos were obtained without any post-processing steps s.a. foot stabilisation, which is quite common in other studies.

from stylegestures.

That makes sense. Thanks, @simonalexanderson!

from stylegestures.

Can this issue be closed now?

from stylegestures.

Hi @ghenter and @simonalexanderson,

I am trying to reproduce the MoGLOW results using the joint angle representation. In the repo you mention that for this application you use data from the LAFAN1 and Kinematica datasets. Do you train the model using these datasets only? If so, can you provide details for the train/test splits?

Thanks a lot!

from stylegestures.

Hi, This link doesn't seem to be working:
https://kth.box.com/s/quh3rwwl2hedwo32cdg1kq7pff04fjdf
It showed that:
This shared file or folder link has been removed or is unavailable to you.

from stylegestures.

Hi @zf223669,
Our university recently transitioned from using Box to Dropbox OneDrive, invalidating all our data-sharing links.
Since I don't have a copy of the data myself, @simonalexanderson will have to create an updated link to the data.

Edit 2022-05-02: The university has migrated to OneDrive, not Dropbox as previously claimed.

from stylegestures.

Hi @zf223669, Our university recently transitioned from using Box to Dropbox, invalidating all our data-sharing links. Since I don't have a copy of the data myself, @simonalexanderson will have to create an updated link to the data.

Thank you!

from stylegestures.

@zf223669 Here's the updated link to the dataset, shared via OneDrive:
https://kth-my.sharepoint.com/:f:/g/personal/simonal_ug_kth_se/EuzCqSiScf5EvwWgZJ8EZ3wB24oPsKca9klzGzxzV-4N6g?e=BG1ELf
Simon will also update the readme shortly.

from stylegestures.

Hellow, the newest dataset link does not seem to work.
It showed that:
Länken har upphört att gälla.

from stylegestures.

@NoGameNoLive Unfortunately (and annoyingly), university policy requires all links from the new data sharing (OneDrive) to expire after at most 180 days. That is what has now happened here.

@simonalexanderson will have to create a new link and add it to the readme. Regrettably, that is unlikely to happen until next week due to how busy his schedule is at the moment. In the long term we will have to look for another solution, such as Zenodo.

from stylegestures.

@NoGameNoLiveUnfortunately (and annoyingly), university policy requires all links from the new data sharing (OneDrive) to expire after at most 180 days. That is what has now happened here.

@simonalexandersonwill have to create a new link and add it to the readme. Regrettably, that is unlikely to happen until next week due to how busy his schedule is at the moment. In the long term we will have to look for another solution, such as Zenodo.

Thank you!

from stylegestures.