wicg / document-picture-in-picture Goto Github PK

It should be throw a {{RangeError}}, not throw a "{{RangeError}}" {{DOMException}}.

Setting the aspect ratio is important for content that has a natural aspect ratio, like a video.

The value of initialAspectRatio (or width and height) currently sizes the complete outer window, including its title bar, resulting in a window too wide for the content. The title bar height could vary by OS, font settings, locale, and eventually browser.

Users using picture-in-picture generally want to move the video out of the way of something else to multitask. Having an inappropriately sized window displaying black padding around the video will be an annoyance.

The window should be sized so that the window content is the requested aspect ratio; the browser can determine on the overall size of the window accommodating title bar within whatever constraints or preferences apply.

Hi! This proposal ended up in the Chromium security&privacy triage queue, and I had a look at it.

I'm a bit concerned about the potential of this proposal to bypass security directives. This might have to replicate all the inheritance mechanisms used by popups initial empty documents. Examples include sandboxing, Permissions Policy, COOP, etc.

Third party iframes opening popups have also been a security nightmare in the past, and the feature should probably be completely off for them.

Thanks,
Arthur

Migrated from steimelchrome/steimelchrome.github.io#8

From domenic@:

"
Your spec includes two normative monkeypatches (see W3C TAG design principles and Anne's blog for more on why this is bad):

https://steimelchrome.github.io/draft_spec.html#one-pip-window monkeypatches other parts of the spec, with a normative "must" requirement. Instead, that requirement needs to be incorporated (and made precise---what are "appropriate events"??) in the place where picture-in-picture statuses change.

https://steimelchrome.github.io/draft_spec.html#relative-urls monkeypatches the entire web platform's URL resolution. This is quite bad! And probably not what you intended. I suspect instead you want to create the new browsing context / Window / Document with an overridden base URL of some sort.
"

Update the spec for video element picture-in-picture in document pip window.

• videoElement.requestPictureInPicture() in a document pip window will reject with NotSupportedError.
• document.PictureInPictureEnabled will return false in a pip window. (Not sure if this is needed in spec)

I only found this by accident by reading the non-normative introduction:

The PiP window cannot be navigated (any window.history or window.location calls that change to a new document will close the PiP window).

Is this still intended to be implemented? If so, we've got some fun ahead of us, figuring out how to spec it. Some questions to answer:

• Are same-document navigations, e.g. history.pushState() or location.hash = "foo", OK?
• What about navigations that target a Content-Disposition: attachment or status-code 204/205 destination? Those never actually move to a new document. (But, we only find out after doing a network round trip.)

It should be given [=this=]'s [=relevant global object=].

Doing so will create the auxiliary browsing context, but not the traversable it lives in. Use https://html.spec.whatwg.org/#creating-a-new-top-level-traversable with a non-null opener instead. (targetName can be the empty string.)

It would be good go to have index.bs file processed and available as an HTML page in https://wicg.github.io/document-picture-in-picture the way all other WICG repo work.

@steimelchrome Is that planned?

For now, it seems like we only have https://steimelchrome.github.io/draft_spec.html

With lockAspectRatio gone, I think we should remove initialAspectRatio from the spec and tell web developers to only use width and height to set the desired PiP window size. We could make sure that the aspect ratio is respected as well when specifying those implementation wise.

const pipWindow = await documentPictureInPicture.requestWindow({
  initialAspectRatio: player.clientWidth / player.clientHeight,
});

can be replaced with

const pipWindow = await documentPictureInPicture.requestWindow({
  width: player.clientWidth,
  height: player.clientHeight,
});

@steimelchrome
What do you think?

Return the last Window opened by this if it exists and is still open. Otherwise, return null.

This Window should be stored on the DocumentPictureInPicture object, so you can do something like "Return this's last-opened window."

Then, that "last-opened window" field can be updated at clear points in time, probably right before the last step of requestWindow()? Or maybe as part of the queued task in step 15? (The difference is observable, by synchronously checking documentPictureInPicture.window after calling requestWindow().)

Migrated from steimelchrome/steimelchrome.github.io#9

From domenic@:

"
You currently say

Let target browsing context be a new browsing context navigated to the about:blank URL.

This is not really enough. For example, you arguably haven't created a Window or a Document. And we don't generally navigate to about:blank; it's the initial location on which a browsing context starts. And this gives no info on the browsing context group or agent cluster of the created browsing context.

Instead, use https://html.spec.whatwg.org/#creating-a-new-auxiliary-browsing-context . You can get a sense from reading that and the algorithms it calls, all the details you're missing. Thankfully you should just be able to call that algorithm and have it taken care of for you :).
"

I'm working on a webextension and tried documentPictureInPicture.requestWindow() from within a sidePanel. Unfortunately it doesn't work from there (onenter and window attributes are both null).

Hello,

For some use cases related to Video conferencing (one of the cited use cases for this new feature) it would be great if the PictureInPicture could be triggered programmatically.
eg. when receiving an incoming video call in a web-based videoconference solution.
or showing call controls for an inbound call that is automatically accepted (call center agent)

Popups created programmatically with windows.open are blocked by default but a notification is shown to let the user allow popups if needed.

Could PiP behave in the same way? this would enable use cases that are not currently possible on web apps but are used in desktop applications.

Right now they are hidden in a security considerations section. Instead, follow the guidelines at w3ctag/design-principles#441 .

Hey guys, I need to create a specific custom Layout UI for the PiP window with React custom components, along with their business logic from hooks, state control and their styles. But I can't seem to find a proper way to do it. Is this even supported by now?

When I tried to move a button element wrapper from parent document to children window, that had event listeners attached to it such as onClick, the event got lost in the process. All the logic behind it as well for like switching visibility or controlled rendering based on some component state. It even replaced it for "null" for a different state rendering in the button. (Simple "Play" | "Pause" button element switching)

Any help related to this matter would be greatly appreciated. Keep up the good work!

Migrated from steimelchrome/steimelchrome.github.io#7

From domenic@:

"
Check out https://html.spec.whatwg.org/#event-loop-for-spec-authors .

You cannot throw from in parallel. You cannot look at a document's style sheets in parallel. You cannot manipulate promises in parallel. You cannot create new DocumentPictureInPictureSession objects, or even Window objects, in parallel.

Basically, you need to clearly separate the stuff that's on the main thread, and happens before you go in parallel, from the stuff that's in the "browser process", in parallel. And carefully post tasks when you need to sync back to the main thread.

This also applies when navigating to existing concepts. E.g. you use "associated Document" a lot---but whose associated Document? Associated Document is, by definition, the property of a Window object. So presumably, you need to navigate from the one object you have a handle to---the this value inside your method steps---pass that along to the "open DocumentPictureInPictureSession algorithm", then look at documentPictureInPicture's relevant global object's associated Document.
"

I think this would be a monkeypatch to https://html.spec.whatwg.org/#the-rules-for-choosing-a-navigable step 8.

I noticed that every call to the window.documentPictureInPicture.requestWindow creates a new pip window but destroys the previous one. Is that intentional? Is there a way to create multiple windows?

I think you need to change this line:

Since DocumentPictureInPicture.requestWindow() returns a Window which exposes moveTo(), I expected it to work, but in practice it doesn't. My use case is that I have a PiP dashboard that I like to move to a particular area of the screen, which of course is different from the default position in the lower right. Should this work? I was unsure whether to file a Chromium bug or a spec Issue, similar to #33. Sorry if it's more of an implementation problem.

Migrated from steimelchrome/steimelchrome.github.io#10

From domenic@:

"
The style sheet copying text currently says:

the CSS style sheets applied the current associated Document should be copied and applied to the target browsing context’s associated Document.

First, I suspect "should" is not intended here (i.e., the behavior is not optional).

Second, there often is more than one CSS style sheet applied to a document. There can be many, in fact: some from <style>, some from , some from adoptedStyleSheets... Which ones do you want to copy? As per steimelchrome/steimelchrome.github.io#6, you need to navigate to each of these collections in a precise manner, and gather up the appropriate objects to copy over.

Third, what exactly are the appropriate objects to copy over, and what does copying mean? Are you going to insert a <style> for each source <style>, a for each source , etc.? What if some of them failed to load? Are you going to copy over the text exactly, including e.g. syntax errors and exact whitespace, or will you instead copy over some serialization and then deserialize it? All of these things are observable.
"

The Document PiP could become a "wet dream" for all kinds of aggressive advertisers, which could serve always-on-top intrusive ad popups disturbing users while browsing web pages. How will you prevent such an unpleasant behavior?

Is it really necessary to have a title bar in PiP window?
Also it would be nice to be able to further reduce the size of the window.
At the moment I can't open a PiP window with size 200px x 200px

fire an event named enter using DocumentPictureInPictureEvent on this with its bubbles attribute initialized to true

The this object doesn't have any parent, so having it bubble is pointless.

I suspect it's either https://html.spec.whatwg.org/#close-a-top-level-traversable or maybe just the last 2 steps, if you want to skip beforeunload.

Hey, let's say I got many buttons, components and its functionality attached to them and this PiP window needs to inherit.

How can I properly pass it? Because I tried adding a console.log to the play button in a handler function but is not being triggered within the PiP window. It seems I can only share some of the CSS styles and inner DOM elements, but not actual Javascript and all the styling I added to custom buttons.

Here's the test code:

const VideoPlayer = (): JSX.Element => {
  // Handle to the picture-in-picture window.
  const pipWindow = useRef(null);
  const videoContainerRef = useRef(null);
  const playerRef = useRef(null);
  // const [isPip, setIsPip] = useState(false);

  const handlePlay = () => {
    console.log("playyyyy");
  };

  const enterPiP = async () => {
    const pipOptions = {
      width: playerRef.current.clientWidth,
      height: playerRef.current.clientHeight,
      copyStyleSheets: true,
    };

    // @ts-ignore
    pipWindow.current = await documentPictureInPicture.requestWindow(
      pipOptions
    );

    // Style remaining container to imply the player is in PiP.
    if (videoContainerRef.current) {
      // setIsPip(true);
      playerRef.current.classList.add("pip-mode");

      // Add player to the PiP window.
      pipWindow?.current?.document.body.append(playerRef.current);

      // Listen for the PiP closing event to put the video back.
      pipWindow?.current?.addEventListener(
        "pagehide",
        onLeavePiP.bind(pipWindow.current),
        {
          once: true,
        }
      );
    }
  };

  // Called when the PiP window has closed.
  const onLeavePiP = () => {
    // Remove PiP styling from the container.
    if (videoContainerRef.current) {
      // setIsPip(false);
      playerRef.current.classList.remove("pip-mode");

      // Add the player back to the main window.
      videoContainerRef.current.append(playerRef.current);

      pipWindow.current = null;
    }
  };

  return (
    <div ref={videoContainerRef} className="videoContainer">
      <div ref={playerRef} className="videoPlayer">
        <video
          id="video"
          autoPlay
          muted
          playsInline
          loop
          src="https://storage.googleapis.com/media-session/sintel/trailer.mp4"
        ></video>
        <div className="videoControls">
          <button className="videoControlButton" onClick={handlePlay}>
            Play
          </button>
          <button className="videoControlButton">CC</button>
          <button className="videoControlButton" onClick={enterPiP}>
            Open PiP
          </button>
        </div>
      </div>
    </div>
  );
};

export default VideoPlayer;

Currently history.back/forward/go(delta) doesn't work inside the iframe of the DPIP window

In the current implementation, after i change the url inside the iframe and go back using history.back nothing happens

Please, add support for history API for iframes inside Document Picture in Picture

Repro: https://pages.p.chashkin.site/labs/dpip-history-repro/

It defaults to false.

Would it be possible to customize the color of the tab in the pip window, by default its black

Chrome: v114
OS: MacOS Ventura 13.4.1
Video Background processor: https://www.npmjs.com/package/@twilio/video-processors

This video background processor is based on Google implementation.

These processors run TensorFlow Lite using MediaPipe Selfie Segmentation Landscape Model and requires WebAssembly SIMD support in order to achieve the best performance.

STR:
1. Start a video and apply the video background from tab1
2. Activate Picture and Picture functionality
3. Click on tab2 and check the video inside PiP

Actual result: video is extremely lagging
Expected result: video is playing

Note: When I deactivate the video background feature, the video inside PiP works fine again

Do PiP windows have the ability to insert JS into them? For example, if I want to display a test alert:
pipWindow.innerHTML += "<script>alert("hello world")</script>"

I tried this on Chrome 114 (Mac OS) with the experimental flags on, but it did not seem to recognize the JS.

Thanks!

If the relevant global object of this is not a top-level browsing context

First, the global object will be a Window. You cannot use the predicate "top-level browsing context" on a Window object; they're different things.

Second, in whatwg/html#6315 we generally moved away from browsing contexts.

So, the right way to do this is:

If [=this=]'s [=relevant global object=]'s [=Window/navigable=] is not a [=top-level traversable=], then...

It would be nice to list partners who are interested in the API and document their specific use cases (if any) in the https://github.com/WICG/document-picture-in-picture/blob/main/README.md.

Steps 14 and 15 treat target browsing context like a Window object. It is not. You need to go from a browsing context (or traversable navigable, per recent updates) to the Window object.

You probably want either https://html.spec.whatwg.org/#create-a-fresh-top-level-traversable or https://html.spec.whatwg.org/#creating-a-new-top-level-traversable , depending on whether these things have openers.

In #10 we discussed relative URL resolution. You said

I was able to just remove the relative URLs section since that is actually taken care of for free by the "creating a new auxiliary browsing context and document" algorithm.

I checked and... I think you're right? But, the HTML spec's behavior doesn't seem to match any browsers?? I filed whatwg/html#9310 to track the potential HTML spec bug.

So, either HTML has a bug, and fixing it will break this spec. (I.e., it will mean that document PiP windows get an about:blank base URL instead of your desired inherited base URL.) Or, there's something subtle about HTML that we both missed, and this spec is currently broken.

I suspect what we'll have to do here is monkeypatch https://html.spec.whatwg.org/#fallback-base-url to have a case for checking if document's navigable is a document-picture. (See w3ctag/design-principles#441 for some in-progress guidance on how to do that well, from this spec.) But probably first we need to resolve my confusion about what HTML says...

Hi, since the spec doesn't have a limitation on the minimum size, would it be possible to allow creating a window smaller than 300x300?

WebKit folks raised questions around window.open that I know I had reading this for the first time. It might be worthwhile to add a section to the explainer outlining why this is not the approach picked, and its downsides.

I believe spec should say that documentPictureInPicture.requestWindow() MUST reject with "NotAllowedError" when invoked from a PiP window as introduction says the PiP window cannot open more windows.

Thanks for bringing such great features to the web!
I'm a web developer and I would love to use the document pip feature, but there's one major disadvantage compared to video-pip windows. As a mac user I'm used to work with virtual desktops and the video pip-window is always shown on the current virtual desktop. Unfortunately, the document-pip window does not follow along, it just stays at the virtual desktop where it was opened. That makes no sense to me. The main purpose of such a window is to be always present to the user. With the current behaviour, it is just an advanced popup window.
I would love to see a requirement in the spec saying that it must be always visible to the user and therefor be present on the current active desktop.
Thanks in advance!

Can we have a option to customize the titlebar (like WindowControlOverlay) if it's in a PWA?

For example, can we customize the yellow area like WindowControlOverlay (with keeping blue area).

A special case of #8.

See https://html.spec.whatwg.org/#pdf-viewer-supported for precedent on how to declare this.

close a top-level traversable invokes unload a document, which invokes unloading document cleanup steps, which invokes the exit picture-in-picture algorithm.

The call to "unload a document" is recursive, as well.

So, I think steps 2-3 of "close any existing picture-in-picture windows" are not needed.

We've implemented Document PiP in Whereby, and were wondering if it could be an idea to consider Picture-in-Picture as a type of "display mode". For reference, display modes are used with e.g. PWAs, which can have display modes of e.g. standalone or fullscreen, reflecting their window state. In a similar way, Picture-in-Picture can be considered a (more temporary) type of display mode.

https://www.w3.org/TR/mediaqueries-5/#display-mode defines "display mode" as follows:

A display mode represents how the web application is being presented within the context of an OS (e.g., in fullscreen, etc.). Display modes correspond to user interface (UI) metaphors and functionality in use on a given platform.

This seems to match with how PiP windows behave and what they do. A display-mode media feature for picture-in-picture would then allow us to write specific CSS rules that are only applied when (part of the) the application is shown in picture-in-picture mode. For example:

@media all and (display-mode: picture-in-picture) {
  body {
    margin: 0;
  }
  h1 {
    font-size: 0.8em;
  }
}

In this example, we're adding a modification that removes margins on the body element, and reduces the font-size of titles when in PiP mode — this could be done in an attempt to better fit the content in question inside the PiP window, while the rest of the CSS is just applied as normal.

NB: there is already a :picture-in-picture pseudo class, but in my opinion, it's more for one-off adjustments; if a number of rules need to be modified, a special display mode seems to be a better fit.

Migrated issue from steimelchrome/steimelchrome.github.io#6

From domenic@:

"
It seems like maybe browsing contexts own DocumentPictureInPictureSessions? If so that should be declared somewhere. But it's a surprising choice; do you really want a PiP session to last if the user navigates from https://example.com/ to https://other.example/? (A browsing context is like a browser tab.)

In general, you want to declare up-front what types of objects own what other types of objects, and add definitions for that ownership. Then, when you access an object, you need to traverse the ownership graph. An example from https://wicg.github.io/navigation-api/ is how it declares each Window owns a navigation API (an instance of Navigation), and each Navigation owns an entry list.
"

I found this by chance reading the non-normative introduction.

What exactly is meant by "opening window"? Is it the exact Window object, so e.g. if I navigate the opener from one page to a different one, the PiP window should close? Or is it the navigable (~ browser tab), so I could go across multiple pages and keep the PiP window open? Or something in between?

It sounds like the model here is that every Window has a boolean, "is document picture-in-picture", which is either true or false, false by default. Or maybe it should belong to a navigable instead? Or a top-level traversable?

Then you should check that boolean.

I loved the Pomodoro timer example as a non-video use case.
But I think that any kind of timer would be a valid use case. e.g. for a time tracking app, it'd be great to have an always-on-top tiny window that tells me it's running, and enables me to change the thing I'm currently doing. I'm sure there are other forms of timers that can make use of this.

So, please consider expanding the use case :)