Is there a function similar to relative or absolute from First Order Model? about motion-cosegmentation HOT 9 CLOSED

Thanks for the quick response.

Okay, I understand where you're coming from, so let me try to word it a different way. Is it possible to run this like running First Order Model in reverse? For example, driving the source image with a video, but utilizing all of the cosegmentation tech to choose which parts are driven.

Sorry if it isn't clear, but that's the best way I can think of wording it at the moment.

from motion-cosegmentation.

It is still not clear what you want. Can you draw some picture?

from motion-cosegmentation.

I don't see how ffmpeg can help you here, sorry. I guess internally imageio uses ffmpeg.

from motion-cosegmentation.

Not sure how it should work. If you did not warp every feature it will be inconsistent, for example if you warp only the eye, and leave other parts of face intact, eyes will overlap with the nose or mouse.

from motion-cosegmentation.

Hello again. Just to check, is there any update on something like this being possible?

from motion-cosegmentation.

Sorry for not being clear. I think a better way to say it is if we can use this like First Order Model with segmentation. Here are two scenarios.

First:

• Step 1: Find the driving video.

• Step 2: Place the eyes, mouth, and nose from the driving video and onto the source image.

• Step 3: Transfer the motion from the driving video to the source image with the segmented features of the former going onto the latter.

Thinking about it this way, it's like running this script in reverse. Currently, you can transfer the features of the source image to the target video whether it's unsupervised or not. I would like to do the opposite, and transfer it the other way around.

Second:

Another scenario would be if I want to keep the source images face shape, and place it onto the target video without morphing the face to the target.

An example would be this. The target video has a face that is round. The source image has a face that is triangular. I would like to transfer the source, triangular face to the target round face while keeping the face shape.

Concluding

The reason for wanting to do this is that I feel that this project is a bit more powerful than the First Order Model Repo (even though they're relatively similar), and I would like to fully switch over for my use case. Hope that's more clear.

from motion-cosegmentation.

OK. I see. There is no relative option here, because in fomm you only need source image and driving keypoints for generation. Here you need driving segments, obtaining "relative segments", rather than relative keypoint is not trivial.

For the first scenario you first animate your image with fomm, and then transfer segments that you need with this model.

For the second you just want to preserve background? If so you can copy background segment and blend it with relatively animated image in feature space.

from motion-cosegmentation.

OK. I see. There is no relative option here, because in fomm you only need source image and driving keypoints for generation. Here you need driving segments, obtaining "relative segments", rather than relative keypoint is not trivial.

Alright, that one makes a ton of sense and clears up a lot of confusion.

For the first scenario you first animate your image with fomm, and then transfer segments that you need with this model.

I just tried your suggestion and it works well!

For the second you just want to preserve background? If so you can copy background segment and blend it with relatively animated image in feature space.

I'm not sure I understand this one, but the suggestion for the first scenario makes this a non issue, thanks!

I just have one more question before I close this. I see that you aren't using FFMPEG to render the result videos. Is there any way to get better quality on the results? Thanks again.

from motion-cosegmentation.

Okay I see. Thanks for the help!

from motion-cosegmentation.