Git Product home page Git Product logo

Comments (7)

yosefl20 avatar yosefl20 commented on June 22, 2024 1

I did everything as I can to port the code to mps. Sadly, mps does not implement ten::_fft_r2c at PyTorch 2.0.1. And I got this from console:

NotImplementedError: The operator 'aten::_fft_r2c' is not currently implemented for the MPS device. If you want this op to be added in priority during the prototype phase of this feature, please comment on https://github.com/pytorch/pytorch/issues/77764. As a temporary fix, you can set the environment variable 'PYTORCH_ENABLE_MPS_FALLBACK=1' to use the CPU as a fallback for this op. WARNING: this will be slower than running natively on MPS.

from video-retalking.

mertuner avatar mertuner commented on June 22, 2024 1

I was able to run this on mps. Still slow but significantly better than CPU. Here is the 5 second video result.

MacBook Air 2020 M1/16GB RAM/8 Core GPU

MPS ===> 29min 36sec
CPU ===> 50min 4sec

Let me know how long yours take. And if you want to try yourself on mps:

1-) face_parsing.py ---> line 44 to --->

im = cv2.resize(im.astype((np.float32)), (self.size, self.size))

2-) inference.py --->

def main():    
    # adding gpu support on Apple M1/M2/M3 Chip Devices
    if not torch.backends.mps.is_available():
        if not torch.backends.mps.is_built():
            print("MPS not available because the current PyTorch install was not "
              "built with MPS enabled.")
        else:
            print("MPS not available because the current MacOS version is not 12.3+ "
              "and/or you do not have an MPS-enabled device on this machine.")
    elif torch.cuda.is_available():
        device = 'cuda'
    elif torch.backends.mps.is_available():
        device = torch.device("mps")
    else:
        device = 'cpu'

...rest

3-)

conda env config vars set PYTORCH_ENABLE_MPS_FALLBACK=1
conda activate <video_retalking>

We do step 3 because there is this one operation is yet to be implemented using mps.

It should work.

Cheers.

from video-retalking.

LittleTerry avatar LittleTerry commented on June 22, 2024

same question here

from video-retalking.

foundVanting avatar foundVanting commented on June 22, 2024

+1

from video-retalking.

jdbbjd avatar jdbbjd commented on June 22, 2024

Have you solved the problem

from video-retalking.

HotWordland avatar HotWordland commented on June 22, 2024

+1

from video-retalking.

aviadavi avatar aviadavi commented on June 22, 2024

@yosefl20 thanks for you insights. i'm unable to get it run on my M1 CPU. I set this env variable to 1 ..but still gets the error

(video_retalking) a@Mac-a video-retalking % python3 inference.py \              
  --face examples/face.mp4 \
  --audio examples/audio.m4a \
  --outfile results/1_1.mp4
/opt/homebrew/Caskroom/miniconda/base/envs/video_retalking/lib/python3.8/site-packages/torchvision/transforms/functional_tensor.py:5: UserWarning: The torchvision.transforms.functional_tensor module is deprecated in 0.15 and will be **removed in 0.17**. Please don't rely on it. You probably just need to use APIs in torchvision.transforms.functional or in torchvision.transforms.v2.functional.
  warnings.warn(
[Info] Using cpu for inference.
[Step 0] Number of frames available for inference: 377
[Step 1] Landmarks Extraction in Video.
Traceback (most recent call last):
  File "inference.py", line 345, in <module>
    main()
  File "inference.py", line 81, in main
    kp_extractor = KeypointExtractor()
  File "/Users/a/Git/video-retalking/third_part/face3d/extract_kp_videos.py", line 16, in __init__
    self.detector = face_alignment.FaceAlignment(face_alignment.LandmarksType._2D)   
  File "/opt/homebrew/Caskroom/miniconda/base/envs/video_retalking/lib/python3.8/site-packages/face_alignment/api.py", line 77, in __init__
    self.face_detector = face_detector_module.FaceDetector(device=device, verbose=verbose, **face_detector_kwargs)
  File "/opt/homebrew/Caskroom/miniconda/base/envs/video_retalking/lib/python3.8/site-packages/face_alignment/detection/sfd/sfd_detector.py", line 31, in __init__
    self.face_detector.to(device)
  File "/opt/homebrew/Caskroom/miniconda/base/envs/video_retalking/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1145, in to
    return self._apply(convert)
  File "/opt/homebrew/Caskroom/miniconda/base/envs/video_retalking/lib/python3.8/site-packages/torch/nn/modules/module.py", line 797, in _apply
    module._apply(fn)
  File "/opt/homebrew/Caskroom/miniconda/base/envs/video_retalking/lib/python3.8/site-packages/torch/nn/modules/module.py", line 820, in _apply
    param_applied = fn(param)
  File "/opt/homebrew/Caskroom/miniconda/base/envs/video_retalking/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1143, in convert
    return t.to(device, dtype if t.is_floating_point() or t.is_complex() else None, non_blocking)
  File "/opt/homebrew/Caskroom/miniconda/base/envs/video_retalking/lib/python3.8/site-packages/torch/cuda/__init__.py", line 239, in _lazy_init
    raise AssertionError("Torch not compiled with CUDA enabled")

any idea how to workaround this?

from video-retalking.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.