Comments (7)
I did everything as I can to port the code to mps. Sadly, mps does not implement ten::_fft_r2c at PyTorch 2.0.1. And I got this from console:
NotImplementedError: The operator 'aten::_fft_r2c' is not currently implemented for the MPS device. If you want this op to be added in priority during the prototype phase of this feature, please comment on https://github.com/pytorch/pytorch/issues/77764. As a temporary fix, you can set the environment variable 'PYTORCH_ENABLE_MPS_FALLBACK=1' to use the CPU as a fallback for this op. WARNING: this will be slower than running natively on MPS.
from video-retalking.
I was able to run this on mps. Still slow but significantly better than CPU. Here is the 5 second video result.
MacBook Air 2020 M1/16GB RAM/8 Core GPU
MPS ===> 29min 36sec
CPU ===> 50min 4sec
Let me know how long yours take. And if you want to try yourself on mps:
1-) face_parsing.py ---> line 44 to --->
im = cv2.resize(im.astype((np.float32)), (self.size, self.size))
2-) inference.py --->
def main():
# adding gpu support on Apple M1/M2/M3 Chip Devices
if not torch.backends.mps.is_available():
if not torch.backends.mps.is_built():
print("MPS not available because the current PyTorch install was not "
"built with MPS enabled.")
else:
print("MPS not available because the current MacOS version is not 12.3+ "
"and/or you do not have an MPS-enabled device on this machine.")
elif torch.cuda.is_available():
device = 'cuda'
elif torch.backends.mps.is_available():
device = torch.device("mps")
else:
device = 'cpu'
...rest
3-)
conda env config vars set PYTORCH_ENABLE_MPS_FALLBACK=1
conda activate <video_retalking>
We do step 3 because there is this one operation is yet to be implemented using mps.
It should work.
Cheers.
from video-retalking.
same question here
from video-retalking.
+1
from video-retalking.
Have you solved the problem
from video-retalking.
+1
from video-retalking.
@yosefl20 thanks for you insights. i'm unable to get it run on my M1 CPU. I set this env variable to 1 ..but still gets the error
(video_retalking) a@Mac-a video-retalking % python3 inference.py \
--face examples/face.mp4 \
--audio examples/audio.m4a \
--outfile results/1_1.mp4
/opt/homebrew/Caskroom/miniconda/base/envs/video_retalking/lib/python3.8/site-packages/torchvision/transforms/functional_tensor.py:5: UserWarning: The torchvision.transforms.functional_tensor module is deprecated in 0.15 and will be **removed in 0.17**. Please don't rely on it. You probably just need to use APIs in torchvision.transforms.functional or in torchvision.transforms.v2.functional.
warnings.warn(
[Info] Using cpu for inference.
[Step 0] Number of frames available for inference: 377
[Step 1] Landmarks Extraction in Video.
Traceback (most recent call last):
File "inference.py", line 345, in <module>
main()
File "inference.py", line 81, in main
kp_extractor = KeypointExtractor()
File "/Users/a/Git/video-retalking/third_part/face3d/extract_kp_videos.py", line 16, in __init__
self.detector = face_alignment.FaceAlignment(face_alignment.LandmarksType._2D)
File "/opt/homebrew/Caskroom/miniconda/base/envs/video_retalking/lib/python3.8/site-packages/face_alignment/api.py", line 77, in __init__
self.face_detector = face_detector_module.FaceDetector(device=device, verbose=verbose, **face_detector_kwargs)
File "/opt/homebrew/Caskroom/miniconda/base/envs/video_retalking/lib/python3.8/site-packages/face_alignment/detection/sfd/sfd_detector.py", line 31, in __init__
self.face_detector.to(device)
File "/opt/homebrew/Caskroom/miniconda/base/envs/video_retalking/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1145, in to
return self._apply(convert)
File "/opt/homebrew/Caskroom/miniconda/base/envs/video_retalking/lib/python3.8/site-packages/torch/nn/modules/module.py", line 797, in _apply
module._apply(fn)
File "/opt/homebrew/Caskroom/miniconda/base/envs/video_retalking/lib/python3.8/site-packages/torch/nn/modules/module.py", line 820, in _apply
param_applied = fn(param)
File "/opt/homebrew/Caskroom/miniconda/base/envs/video_retalking/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1143, in convert
return t.to(device, dtype if t.is_floating_point() or t.is_complex() else None, non_blocking)
File "/opt/homebrew/Caskroom/miniconda/base/envs/video_retalking/lib/python3.8/site-packages/torch/cuda/__init__.py", line 239, in _lazy_init
raise AssertionError("Torch not compiled with CUDA enabled")
any idea how to workaround this?
from video-retalking.
Related Issues (20)
- 训练复现的下半张脸,颜色总是不一致,为什么? HOT 1
- reshape error HOT 4
- python 3.8 not match basicsr==1.4.2 which is in requirements.txt??? HOT 3
- Linux 环境下,按照python3.8版本安装,dlib会报错
- Is there ANYWAY, to make this FAST, FASTER? And to use GPU, Pytorch, Batchj size etc? HOT 6
- When will the training code for the model be made public? HOT 2
- UnboundLocalError: local variable 'mask_sharp' referenced before assignment HOT 3
- Accessing elements in a 2D array
- ModuleNotFoundError: No module named 'face_alignment'
- nvrtc compilation failed + 'keypoints' errors HOT 1
- Problem producing high resolution output HOT 9
- 您好,请问一下可以重新进行训练模型吗?我觉得对于中文语音来说,7秒之内唇形很准确,但是7秒之后就不准了,有些停顿的地方都不停顿。
- SSL Issue with usage HOT 1
- TensorBody.h:338:30: note: declared here 338 | DeprecatedTypeProperties & type() const
- sieve improvements HOT 4
- 提升生成速度
- model running speed
- [Step 5] FaceDet:: 0%|
- 请问传入什么样的mp4文件会获得较好的效果
- 求助,折腾48小时了,还没好,报错解决不掉 HOT 4
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from video-retalking.