Use Chrome Desktop Capture as video source? about mediapipe-touchdesigner HOT 9 OPEN

Hey @UUooc, thanks so much for this suggestion! I've been trying to figure out an elegant way to get video from TouchDesigner to Chrome that doesn't require any external plugins and this might be that solution!

I'll have a tinker and see if I can get a similar chain running. In theory it should be possible with a Window COMP in TouchDesigner (make a new window with something in it), and then use that as the capture source... I'll see how the theory stacks up with reality

from mediapipe-touchdesigner.

I've started a MediaPipe for OBS project. Electron js is used to run the MediaPipe models locally and send the landmarks to OBS. The landmarks are stored in a Text Source as JSON. The Advanced Scene Switcher plug-in reads the landmarks JSON to control OBS.

Thanks for the inspiration!

from mediapipe-touchdesigner.

Oh wow super cool! Being able to trigger scene changes or effects in OBS with poses or hand gestures will be super fun - great work!

from mediapipe-touchdesigner.

Just thought I'd drop an update in here after spending the day trying to get this to work with the embedded Chromium that TouchDesigner uses in case you've seen it work otherwise.

It seems that the command line options to allow auto-accept of window or screen sharing don't work in CEF, and as there's no mechanism to show menus with CEF, I can't get this working right now.

First up, it seems that chrome.desktopCapture only works if you're running a chrome extension (which I guess may be possible, but something to investigate later)

The more standard way of requesting a media source is navigator.mediaDevices.getDisplayMedia which works great when running the page in Chrome - you get a pop-up asking what you want to share, select the window you want, it all works.

Permission denied

When running in the CEF instance inside TouchDesigner, I get a permission denied error when trying to ask for a capture device. So far the options I've tried are:

• No options specified
• Enable Media Stream toggled on and off
• --auto-select-window-capture-source-by-title="MP_capture"
• --auto-select-desktop-capture-source="MP_capture"

(where MP_capture is the name of the window with some test content)

Limited success

If I add --use-fake-ui-for-media-stream I do get a video stream (yay!) but it's a whole screen of a random monitor, not the window name I've specified, and no apparent way to change it.

Other things to try

Some reports said adding the option --enable-chrome-runtime would run a full version of Chrome with all the navigation UI and that may allow me to try other things, but that doesn't seem to work in the TouchDesigner embedded version.

@UUoocl if you know of any other option flags I could try, or other ways to make this work that would be great. It sounds like someone else found the issue and submitted a PR that got rolled into the Chromium code last month, so maybe it'll get fixed in CEF:
https://www.magpcss.org/ceforum/viewtopic.php?f=6&t=19676

from mediapipe-touchdesigner.

Looping back to this - I just tried the same things in TouchDesigner 2023 which now has CEF 115 (vs CEF 100 in TouchDesigner 2022) and it does slightly different things with different toggles, but the end result is the same - I can only get complete, all-screens sharing working, I don't seem to have any ability to force only a specific window to be captured.

from mediapipe-touchdesigner.

I went down a rabbit hole chasing this one to see if it's anything in CEF we can manipulate. The short version seems to be that

--auto-select-window-capture-source-by-title="MP_capture"
--auto-select-desktop-capture-source="MP_capture"

Don't do anything in CEF - they only affect full Chrome browser.

I found this little ditty in the CEF source code that says if no ID is asked for, the default is to share Screen -1 (entire screen)

        media_id =
            content::DesktopMediaID(content::DesktopMediaID::TYPE_SCREEN,
                                    -1 /* webrtc::kFullDesktopScreenId */);
      }
      video_devices.emplace_back(request_.video_type, media_id.ToString(),
                                 "Screen");
    }

As a (probably sensible) security feature, JS code running in a browser is not allowed to ask for the sharing of any specific Window or Screen... so for now I'm not sure if there's a way to make this work.

One workaround I considered is do as above and crop the entire screen to only the window we want, but for that to work the window we want to capture from would need to be fully visible on the screen at all times, which is less than ideal. I am hoping there's another way that would work with the window hanging off the edge of a screen so it's only 1 or 2 pixels visible, but the window is drawn by the GPU and thus could be captured by Chrome (this works in full Chrome browser, so that part is possible).

I've raised this ticket with the CEF team to see what they say... chromiumembedded/cef#3667

from mediapipe-touchdesigner.

from mediapipe-touchdesigner.

In Electron the browser can not access the Get User Media directly either. The work around is to use node to query the available devices then pass the results to the browser. https://www.electronjs.org/docs/latest/api/desktop-capturer
The window "ID" has been more reliable then using the window "name" when setting the video stream.

Another option to bring video in could be webRTC. webRTC can run on the localhost and only needs a websocket server to connect 2 browsers.

There is an example of connecting a browser to the OBS CEF browser MediaPipe for OBS project with webRTC. https://github.com/UUoocl/MediaPipe_for_OBS

from mediapipe-touchdesigner.

Ohhh thank you for those links @UUoocl !

It looks like the secret sauce that Electron is using is a couple of Chrome-specific options you can pass to getUserMedia called chromeMediaSource and chromeMediaSourceId

When I get some time to tinker next, I can try them out and see if I can pull the list of Window IDs, then push the relevant ID to getUserMedia with the chromeMediaSourceId option... that might just do it.

from mediapipe-touchdesigner.