torinmb / mediapipe-touchdesigner Goto Github PK
View Code? Open in Web Editor NEWGPU Accelerated MediaPipe Plugin for TouchDesigner
License: MIT License
GPU Accelerated MediaPipe Plugin for TouchDesigner
License: MIT License
hey dom and torin,
i see an issue where on some machines the webpage would run and show the camera but the prediction-loop is not initialzed. i tracked this down to what i believe is a wrong assumption in your setup() routine. i am not really familiar with js so maybe i'm reading things wrong, but here goes:
i wonder if instead, at the end of setup() we don't start the camera but always start the prediction-loop. in the loop we check if the camera is running. so whenever the camera is then started or changed, prediction would start?!
Hi Torin, Hi Dom, such a great work!!! I have a problem: SpoutCam is not showing in the webcam menu. I have downloaded the newest versions etc. I am on PC windows 10. What can I have done wrong? thank you soooo much!!!!!
It only happens to me that every time I close a project then when I reopen it MediaPipe.tox is offline with a red cross and also the related tool (facetracking for example)?. Is there a way, maybe saving everything in the same folder, MediaPipe.tox etc. included (?) but I have already tried and every time I restart I have to delete the old nodes and replace them .
Thank u in advantage <3
Love u guys
Andrea
I noticed while using the object detection portion of the TOX that the bounding boxes that Touchdesginer is drawing do not align with bounding boxes that Mediapipe is drawing. They are offset and don't accurately map back onto the video.
Thanks for putting this resource together, it is a really great tool and addition to touchdesigner!
I tried all the troubleshooting methods I could find, but none of them seemed to help. My TD version is 2022.33910. I registered & unregistered on Spoutcam as directed in the tut as well. I also tried changing the port to 3002, but no good. My webcam and internet connection are fine. I tried checking Spoutcam on the Mediapipe website and it worked fine. My starting res is 1280 x 720 as mentioned.
I hope the screenshots help you find out the issue. Really excited to use the plugin.
thanks!
I wanna try to use your facial detection on a video not the webcam so I wanna know if there is a way to do this ?
Have a nice day
Hi Torin and Dom, Thanks for all amazing tool. Making it easy for beginners to get started.
I have some problems while using webcam with SpoutCam. I selected 'webcam' in videofilein TOP, and selected 'SpoutCam' in the Mediapipe tox. Both of the resolutions are set to 1280 x 720. However, the Mediapipe window shows RGB noise.
When I use laptop cam and obs in Mediapipe tox, it is successful. Using the webcam, Mediapipe window have image but not working.
I also tried changing the port to 3002, but not working.
I found an interesting part: when I select 'SpoutCam' in Mediapipe tox, window shows RGB noise. However, when I open videofilein TOP and also select 'SpoutCam', videofilein TOP shows the image.
These are the images that I tried loading http://localhost:9222/.
Select 'webcam' in Mediapipe tox
Select 'SpoutCam' in Mediapipe tox
TD Version:2022.33910, 2022.35280
MediaPipe TD Release:v0.2.2
System:Windows 11
If there's anything I haven't noticed or done wrong, please let me know! Thank you for your guidance!
Hi, thank you for your hard work on this one. I have one issue to report, I do not seem able to find the spoutcam on the drop-down menu of the Mediapipe container (on windows 11). I have registered the Spoutcam as detailed on the instructions page.
Also when I load the spoutcam on the VideoDeviceIn TOP it shows up as just black.
Any help would be appreciated!
EDIT: I figured the error I am getting on the Mediapipe container might be enlightening:
Similar to other issues, I'm getting the endless loading screen of media pipe. I believe I have set everything up correctly, but I am not getting a "SpoutCam" Option in the Media Pipe Webcam dropdown. Some screenshots:
Specs:
TD Version: 2022.33910
MediaPipe TD Release: v0.2.1
OS: Windows 10
Please let me know if I'm leaving any information out. Really excited to get into this, everything looks great!
Hi there! Love your content, brightly put and well made!
I am running windows 10, x64 bit system, with the latest version of TD (2022.33910)
and connecting a proper USB Logitech webcam - downloaded the latest version of the plugin Release v0.2.1.
Once launching the MediaPipre TouchDesigner.toe file, all the components shows as expected, but the loading container of MediaPipe stays loading endlessly.
I can turn on and off the parameters, but nothing affects the input of the MediaPipe container, keeping it in loading state.
My computer has no problem recognizing the camera as a Video device Input, and it even shows in the MediaPipe Parameters as a Webcam option (so no webcam detection issues),
and tried it on the MediaPipre website to make sure it worked (even tho i would rather use it through your cool Nodes! haha).
Also, absolutely no red cross appears anywhere, so all good but still loading. Only suspicious thing i found was about the webrowser module in the MediaPipe Container - it appears pitch black. I reckon might be because the toe file has problems connecting the web page that has the tools? Camera privacy issues? Still searching haha
Thanks for your awesome contribution! Looking forward to try it!
Best of luck and your guidance is much appreciated!
Noticed that older versions of mediapipe still work but after each save and file rename TD will fail to load all the media pipe assets and whole networks are gone inside the mideapipe window , including operator IN and OUT nodes.
could this be due to the new release update ?
it was saving a and reloading fine 2 weeks ago last release update was 7 days ago?
images below of the standard network vs the missing one ,
im teaching a class MAY 11TH using touchdesinger and mediapipe was a big part of the workshop.
please get back to me
I´ve testing mediapipe directly from the MediaPipe TouchDesigner.toe file, and it work just nice.
but when I open a new TD project and try to import any of the .tox files from mediapipe it just doesnt import, I´ve tried to import and drag n´drop them, but they doesnt show at my workspace, it doesnt show a popup or warning neither.
Maybe someone have an alternative way of import them?
Hey. I've set up Media Pipe on 2 PC's now. I am trying to set it up on one more, but the MediaPipe tox webcams are updating. I have Spoutcam setup and everything, but the webcams scroll down doesn't change. I've tried a bunch of fixes, but none have worked.
Hi guys! First of all, thank you for your work on TD: I have followed you for 2 years on YT.
I have an issue with this plugin. I'm using Windows 10 with the new release built of TD: 2023.10130.
In the dropdown menu for the webcam, it isn't visible the HD Camera of the PC (MSI GP75 Leopard) and also the MediaPipe component presents a red cross:
Inside the component the issue is this:
How can I fix this?
Thanks for your time ;)
Disregard, as it was a configuration issue.
Hi,
in you're example i'm trying to match the white faceMask to the canonical_face_model connected to the transformmatrix you put on your example.
rotation, ... seems to be ok. But the Z (zoom in / out with head distance from camera. When i'm closed to the camera the canonical_face_model is smaller and bigger when i'm far for the camera.
Any idea ?
My idea is to make a "kind of snapchat filter" with a 3D head oculter that follow the head size.
Thanks
First of all, thanks for all the wonderful open source work!
Hand tracking works through spoutCam however the performance is quite poor. Perhaps five to seven frames per second. I'm running a high end machine. New threadrippers, 256 gigs of ram, A6000 graphics card. Windows 10 pro Version 10.0.19045 Build 19045, TD 2022.33910.
One thing I am confused about is whether there's a way to track directly from a video device or webcam. I can't get that method to work at all. Only SpoutCam input seems to work.
I've followed correctly step by step the guide from the video but the virtual camera in TD doesn't show up, here some screenshoots. I've also tried to reset the camera multiple times or change the Websocket Port or the Mediapipe Port by 1 (ex. 3001 to 3002) but still not working. What I can do to fix this problem?
My device: MacBook Pro M1 2020, TD 2022.35320
Hello, Im trying your project for the first time (laptop windows 11 rtx 2070 with td version 2022.33910) and I cant get it to work. Followed all the steps (spout,...) and still it seems that the mediapipe tox is empty and dont know why... tried re-init, reimporting the tox, among other things without luck. Here some screenshots.
Thanks in advance,
Joan
Hey there!
I was testing on a few different computers and noticed that on some PCs Mediapipe wouldn't load.
From taking a look at the webpage I saw that this error was occurring,
Failed to load module script: Expected a JavaScript module script but the server responded with a MIME type of "text/plain". Strict MIME type checking is enforced for module scripts per HTML spec.
I went into the Mediapipe tox and added these lines of code in the mp_webserver_callbacks dat.
if fileName.endswith('.js'): mimeType = ['application/javascript'] else: mimeType = mimetypes.guess_type(fileName, strict=False)
This solved the issue for me and allowed the TOX to function correctly.
Could it support other TOP input, not just limited to using a webcam?
Thanks
The tool is incredible.
With my team we are implementing it right now and we are trying to take it to a festival in France for an installation!
///
I'm having problems, despite having followed the whole walkthrough and tutorial, to use facetracking_ to track the full movement of the head. It seems to only take rotation and not translation and position. Attached is a short video of the error I got stuck in.
Thank you in advance for your support!!!
Andrea
Hi, I just found this library and first of all, it looks awesome in terms of ease of use!
But since I have been dabbling in MediaPipe a bit and I'm currently trying to decide on the best approach for my project, I'd just be interested in why you chose to go with the JS npm version of MediaPipe for this project. Wouldn't the Python version of mediapipe be somewhat more intuitive or the more obvious option, given TouchDesigner's Python integration? Curious about the thoughts behind this decision and potential downsides you are already aware of. Thank you!
I'm sure that this isn't a real issue but I can't figure out how multiple poses are meant to work. I have dialed up the pose count from 1 and while media pipe seems to track multiple poses just fine, the output only gives me coordinates for 1 pose at a time, usually flipping back and forth between my pose and the person's next to me.
How is this function meant to work?
Also love these toxs, thank you!
Hi,
I have this strange issue. Everything is working fine except the face_tracking blend_shape info. It can detect my face, but it doesn't register the blend_shape information, it's not showing the data at least. However, it does show the info from all the other tracking options.
I have attached some screenshots. I'm curious to hear if there is something I'm missing, or something I might not be aware of.
Im using a MacBook Pro M1, 16 GB memory with macOS Sonoma Version 14.0
Touchdesigner 099 Non-Commercial 64-bit build 2023.11170
Mediapipe release v0.3.2
Thanks :)))
Hi, it is an amazing plung-in for TD, thank for sharing this!
I am a beginner of TD. I created a point following the movement of one finger point,and I wanted to use an over TOP to add the circle TOP above the video of realtime me. I connected the mediapipe directly to the over TOP,but I don't want the grean line and red dot still appearing there. Is there any method to do that excepting add one more camera but to use the realtime video in the mediapipe?
Hello,
thank you for this fantastic project first of all.
I've had the MediaPipe TouchDesigner.toe working for a short moment, but then changed the number of faces to detect, as suggested in the tutorial video. Ever since it does not work anymore. It shows the raw webcam picture in the MediaPipe component but it does not seem to be processed further. Tried restarting, loading again, etc.
Windows 11, TouchDesigner 2023.11290, Release 0.3.3, Logitech BRIO webcam
Thanks for any hints!
Hi, I can't wait to use this, except my Mac camera feed isn't showing on the media pipe when I select FaceTime HD camera. I've tried the pulse button and made sure that my Mac allows Touch Designer in the security settings. Please advise as to what I should do next. I have a feeling TD is trying to choose the webcam OBS virtual and T's phone. Neither of which Im familiar with.
Thanks
Hi there. Again, so in love with this implementation and how smoothly you can install.
No worries about running servers ourselves or anything. Really beautiful.
One Issue I keep having for a couple of days now is that sometimes the ML won't run.
Video-feed (through ndi in my case) runs fine.
And I am checking the port in localhost in the browser. And indeed I see the models start running in the console outputs.
Not that I have a lot of slack in my wits, So I am at their end.
hehe.
I have also checked the resolution. it is 1280x720.
So I don't think this is the problem too?
Thank you very much!
Hi,
First, thank you for your great works! It helps me a lot.
I have an issue about the face tracking sample.
It can detect my face, the face part and the texture part work well. However, the halo part does not work. It doesn’t move with my face and show nothing. I'm curious to hear if there is something I'm missing.
(Also, the same problem with others, it does show the info from all the other tracking options.)
(By the way, I want to know how to change the resolution of the camera? I’m using the FaceTime HD camera of Mac, it shows error message but seems working to show my face. Do I need to change?)
I have attached the screenshot.
——————————————
MacBook Pro (Intel, 2020) with macOS BigSur 11.7.10
Touchdesigner 099 build 2022.35320
Mediapipe release v0.3.2
——————————————
Thank u sooo much.
Hello there,
I am new to TouchDesigner. I'm trying to setup this great mediapipe TD plugin with an NDI camera as my source.
As suggested, I have replaced the VideoDeviceIn TOP with the NDI in TOP and I have followed the instructions in the youtube video for setting up SpoutCam in my windows environment.
My issue is that SpoutCam is not showing as a webcam source in the mediapipe container.
Any ideas?
Thanks in advance!
(I have the latest releases of the plugin and SpoutCam)
Very nice work. Thanks a lot. It brings me a lot forward in my project in which I need the object detection. One Question: Is it possible to connect a Realsense Camera to it and get the distances of the obejcts?
Hi, I'm trying to detect more than 2 hands, but it's always detecting maximum 2 hands. If I set the hand count to 1 it still detects 2, tried setting it to detect 4 or 6 but still detecting 2.
Using Windows 11 Enterprise version 22H2 OS build 22621.2428.
Any suggestions?
Thanks
Hey there! I just wanted to express my gratitude for the amazing work you all have done. I really appreciate the fact that you share it with others. However, I encountered an error while trying to open the project file. Although I accepted the prompt to open the project, the MediaPipe seems to be stuck in the loading state and is not working properly.
HI Guys,
I would really love to create an interactive app, but I have only one camera as an input device. Therefore I would like to make it split screen, by compositing two mediapipe instances, one for the left half and one for the right half of the screen. It is easy to do this with the output, but I would love to do this with the input too. This way the left half of the screen would detect gestures for only the mentioned half of the input. I can transform webrender's output, but it won't change field of view for gesture detection - which I'd love to crop/mask out. Do you have any idea how could i do this?
thanks a lot,
R
Hello Torin,
Awesome work on this repo. I'm trying to figure out how it would be possible to modify the code to enable RTSP feed input instead of a webcam.
The reason, I'm trying to figure this out is because, I realize that when needing multiple types of data, faces, landmarks, and so on, we get better results by running multiple MediaPipe bases for marginal performance costs. I'm using NDI webcams instead of Spout right now, because it allows me to share the same feed over 4 MediaPipe bases on the same machine, but I'm limited by the 4 NDI Cam sources.
I believe that with a multicast RTSP it would be possible to use the same source across multiple instances of MediaPipe bases.
When I save the file, in the same folder of release with toxes, it does not save anything I've worked or I've added when I open the file. If I open the file in other folders(files from the backup folder), it still contains all the work I've done but it's having a connecting issue loading the necessary files (I'm guessing toxes files need to be in the same folder?) I tried moving the same file(from one of the backups) which had all the data in it to the file where the toxes folder is(to the folder in the screenshot) and when I open the file, all the work disappears... How can I fix this?
Can the number of detected Poses be modified?
Thanks
Encountering error "Webcam failed to start. Is it being used somewhere else? Input must also be capable of 720p, 8 bit." with SpoutCam The footage I am using is 720p, and I cant see any other app using SpoutCam.
Windows 11
TouchDesigner 202311340
Release v0.4.2
This looks awesome! I'm excited to try your project. I've been experimenting with mediapipe js, but have never used TouchDesigner.
I want to recommend the Chrome Desktop Capture as another option for bringing video into the mediapipe-touchdesigner.
In this repo I used Chrome's Desktop Capture as the video input source for landmark detection.
OBS-face-tracking-with-P5js
In this setup a webcam source is created in OBS, then a projector window is opened on that source. Chrome is used to capture the OBS projector window. Landmark data is sent from Chrome to OBS via websockets.
Using NDI, Spout/Syphon or OBS virtual camera accomplish the same purpose, but this options may be useful too.
Thanks for sharing this project!
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.