Support for mpris interface about qtwebflix HOT 69 CLOSED

@petrmanek @omni6
Thanks guys!

from qtwebflix.

By quick examination of mainwindow.cpp and some fiddling around in my browser's dev console, I have made the following observations:

• document.querySelector('video') can still be used access the HTML's <video> object.
• The property paused can be used to check if the video is playing or not. If everything else fails, checking it once per second using e.g. QTimer can work nicely to determine the player's playback state.
• If used properly, a JavaScript listener could be added to the <video> to notify the Qt program that player's playback state has changed. The only thing left to figure out is how to asynchronously signal a QWebView from JavaScript.
• To play/pause the video, the functions .play() and .pause() can be used without problems.
• To adjust volume, the property volume can be used with a float from 0 to 1. Netflix's UI seems to adjust automatically even when the volume is set from the console.
• I don't know how to get the current playing video's name. This picture shows the DOM hierarchy, from which it may be extracted: https://gyazo.com/2f11b18ce79c05ef153d83073518c762

from qtwebflix.

It will become a split package... we would need qtmpris and qtdbusextended. So far no problem but atm a bit out of time... Will do this evening.

from qtwebflix.

https://aur.archlinux.org/packages/qt-dbus-extended-git/
https://aur.archlinux.org/packages/qt-mpris-git/

from qtwebflix.

Check out this commit. It can toggle play/pause and full screen and report back player state playing/paused/stopped. I have tested it and it seems to work nicely with playerctl.

from qtwebflix.

@petrmanek I was able to get the text of the video title like this,

elem = document.querySelector('.PlayerControls--control-element.video-title .ellipsize-text ');
elem.innerHTML
elem.textContent

Hope that helps!

from qtwebflix.

@gort818 Glad you like it! 🎉

mpris itself is built on D-Bus, so most of the modern SystemD-powered distros are supported out of the box. In addition, popular WMs like KDE, Gnome, xfce or mate have mpris-capable widgets or sidebars, making it a really convenient feature.

Still, qtmpris and qtdbusextended can now be considered to be dependencies (or we can put some macro magic around them to make them optional).

For this reason, they either:

1. need to be either included as a part of QtWebFlix's package, or
2. need to be packaged on their own (like @omni6 did for arch's AUR) and be referenced, so that package managers will install them prior to QtWebFlix.

With regards to the code, the current state is rather experimental and temporary. It's definitely a good idea to move the mpris-specific additions out into a separate class / file. It'll however be necessary to expose mainwindow's web view, since the new class will want to run JavaScript commands on it.

Thanks for the suggestion concerning the title. I have already put it to use (see the latest commit). It seems to work nicely for movies. For TV shows it, however, concatenates the show's title with the episode name, like so:

MythBustersS4:E15Steam Cannon

from qtwebflix.

@gort818 Nice, seems to work.

from qtwebflix.

@gort818 Check it out! I've devised a pure Qt way of doing the same thing without having to launch another process and rely on curl and grep being installed.

The metadata now looks like this:

$ playerctl metadata
{
  'mpris:artUrl': <'https://occ-0-3141-2773.1.nflxso.net/art/4c566/9f9ff80dbad539bfced0e018edffe601a594c566.jpg'>, 
  'mpris:length': <int64 3016192000>, 
  'mpris:trackid': <objectpath '/com/netflix/title/70141922'>, 
  'xesam:title': <'MythBusters S4:E28 Anti-Gravity Device'>
}

from qtwebflix.

Victory! It took me a while (I'm not your regular QMake user) but I figured out how to do it.

So the current state of the fork is as follows:

1. Repositories qtmpris and qtdbusextended are now referenced as submodules. For this reason, all package build files (bump @omni6) have to use --recursive when cloning QtWebFlix, otherwise the lib/ directory will be left empty.
2. I had to create patched .pro files for both libraries. Unfortunately, this was a necessary step in order to bypass pkg-config and build both libraries as static. Both configuration files can be found in the lib/ directory along with their README.md -- they will have to be kept up-to-date with their counterparts in their respective repos.
3. qtwebflix.pro was moved to src/src.pro and configured to link with the built libraries. The new qtwebflix.pro contains dependency information to ensure that the libraries get built ahead of the executable.
4. The executable binary is now located in src/qtwebflix within the build directory and has grown to ~360 kB. It no longer requires qtmpris and qtdbusextended as runtime dependencies, since they are both linked statically.

Let me know if it works for you.

from qtwebflix.

Would this give the ability to control with KDE-connect?

from qtwebflix.

@omni6 I think yes. As far as I found, KDE supports mpris, and KDE-connect uses it to control multimedia playback.

from qtwebflix.

@petrmanek I would love to, but I have no idea how to do it at this moment. Any help would be appreciated.

from qtwebflix.

@gort818 I'd be happy to give you a hand. I just need to get a bit more familiar with the code base.

from qtwebflix.

I found this adaptor for bridging mpris and Qt. Seems like the toughest part is going to be figuring out whether playback is going on and controlling it via JavaScript.

from qtwebflix.

So, to showcase the referenced features I have created a simple fork of the repo. It can issue basic commands to the video player but still has no ties to mpris.

At the moment, I'm a bit hesitant about adding dependency on the D-Bus adapter I found, as it doesn't seem to be present as a package in any of the major Linux distributions. Does anyone have a better alternative?

from qtwebflix.

I can package it for arch, looks quite easy...

from qtwebflix.

Ok, both packages build with no problems. Can immediately upload... you need it for development? Otherwise i would wait as long we need it as dependency for qtwebflix...

from qtwebflix.

@omni6 Cool, feel free to put them up to AUR.

from qtwebflix.

Check out this commit. It can toggle play/pause and full screen and report back player state playing/paused/stopped. I have tested it and it seems to work nicely with playerctl.

Basic control from kdeconnect works, i can play/pause/resume. But it would be great if you can get the dead buttons to work. Don't know if that is possibe but instead of skip we could use that button for fast forward/backward. Picture of now playing movie would also be nice.

from qtwebflix.

@omni6 Check it out, I've put up another iteration. This one should start animating the time bar in such a way that it's possible to see video duration and the current time position within the video.

Unfortunately, I've encountered difficulties when trying to set the time position within the video to absolute or offset values. Seems like Netflix has some sort of protective scripts which determine whether the position jumps suspiciously large intervals.

from qtwebflix.

@petrmanek This is awesome! Unfortunately I do not know much about mpris, how would we support users of other distros? Could we make mpris a separate class, move it out from mainwindow? I think I know how to get the name of the video playing in netflix.

from qtwebflix.

Nothing a lil regex magic cannot fix, give me a minute

from qtwebflix.

elem.innerHTML.replace(/(<([^>]+)>)/g, " ").replace(/ +(?= )/g,'').replace(/ +(?= )/g,'').trim();

from qtwebflix.

@omni6

Picture of now playing movie would also be nice.

I will try and see if I can get the thumbnail, no guarantees but that would be awesome.

from qtwebflix.

@gort818 If you can get me a public URL of the thumbnail, I already have a mpris metadata key for it. ;)

from qtwebflix.

@petrmanek I cannot seem to get it, it seems the box art images are from a http request.. I do not know how to grab the response data.

here is the header

Host: occ-0-33-37.1.nflxso.net
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:64.0) Gecko/20100101 Firefox/64.0
Accept: */*
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate, br
Referer: https://www.netflix.com/watch/80179786?trackId=14170035&tctx=1%2C1%2C6128ba7e-b24d-43df-adf5-50dee97b9ee3-2897852%2C1ef4d6c9-aa5a-46fb-9df5-e72802422ec4_51598400X19XX1546203694257%2C1ef4d6c9-aa5a-46fb-9df5-e72802422ec4_ROOT
Connection: keep-alive

and here is the GET request

https://occ-0-33-37.1.nflxso.net/art/8a79c/98ef726068a6f753c6ae8c4f2507717af028a79c.jpg

from qtwebflix.

@petrmanek I have noticed many other qt programs that use mpris without extra dependencies.

Here how it was implemented in babe https://github.com/KDE/babe/blob/master/src/kde/mpris2.h

from qtwebflix.

I created a new branch with qtmpris and qtdbusextended incuded

https://github.com/gort818/qtwebflix/tree/mpris

it will build the libs that way we dont have to worry about packaging it

Edit. NVM just going to make it more complicated

from qtwebflix.

@gort818
Ad thumbnail: We don't need the actual data from the request. Just the URL is fine for mpris (it has a dedicated ArtURL metadata key). Can you retrieve that somehow? Also, I've noticed that the <video> object has a poster property which is supposed to contain a similar URL, perhaps that might help.

Ad KDE/babe: That would be my preference as well. Unfortunately, the implementation seems rather minimalistic. I've also found a fork of this repo, which attempted to support mpris. However, according to issues and the commit log, it seems like they didn't get too far implementation-wise. If you have any other suggestions, feel free to reference them -- I haven't found too many thus far.

Ad branch: Kudos to you if you actually got the project to build. I've attempted the same thing but rather unsuccessfully. Since they're both on GitHub, I'd definitely suggest including both libs as submodules instead of copy-pasting them into the repo. This way, we can keep up with any modifications in the future. I'd also prefer linking both libs statically to avoid any clashes in /usr/lib in distros (like arch), where you can install them independentely of QtWebFlix.

At the moment, I'm a bit at loss about where to continue the development of this feature. Should I give you permission to push to my fork? Would you rather prefer a pull request to the mpris branch? Or should I disregard the new branch for the moment and continue working on the fork, which we'll merge at a later time?

from qtwebflix.

thumbnail is a pain...

item= document.querySelector('video').offsetParent.id // gets us the title id
curl -i -H "Accept: application/json"  "https://www.netflix.com/title/80131551"  // get us json info
grep -Po 'image":.*?[^\\]",' text.txt   // gets us the image 

Hope that helps! now just to add it all to qt :)

I would say continue working on the fork, you can see use the mpris fork for reference to see how I built the libs and the executable if we go that route.

Thanks so much for your efforts I really like what you have done it is awesome!

from qtwebflix.

Clementime player uses mpris you can check that one out.

also I see ubuntu has a package https://launchpad.net/ubuntu/+source/qtmpris/0.1.0-2

from qtwebflix.

here is an idea to implement it...

 // get netflix title
   view->page()->runJavaScript(
        "document.querySelector('video').offsetParent.id",
        [](const QVariant &result){
            qDebug() << "Value is: " << result.toString() << endl;
        }
    );
 
   // using the title add it to 'https://www.netflix.com/title
    QProcess* proc = new QProcess();
    QString cmd( "/bin/sh" );
    QStringList args;
    args << "-c" <<"curl -s -i -H 'Accept: application/json'  'https://www.netflix.com/title/80131551' | grep -Po '\"image\": *\\K\"[^\"]*\"' ";
    proc->start(cmd, args);
    proc->QProcess::waitForFinished(-1);
    QString output(proc->readAll());
    output.remove(QRegExp("[\\n\\t\\r]"));
    output.remove(QRegExp("[^a-zA-Z:/.-\\d\\s]"));
    qInfo()<< output;

from qtwebflix.

@petrmanek That is beautiful! Nicely done! Thanks so much! We should have a check if url starts with netflix then run this since it is specific to netflix.

@omni6 check it out!

from qtwebflix.

Awesome looking good!

from qtwebflix.

Sweet!

from qtwebflix.

Here is how to get amazon prime video title

document.querySelector('div.webPlayer').innerText.split('X')[0].trim()

from qtwebflix.

Can arrow keys, enter and back get implemented?

from qtwebflix.

@omni6 It all comes down to bypassing Netflix's protections. If you try to modify <video>'s position directly, it just redirects you to that error page.

Perhaps if instead we tried to simulate a click on the position bar, we could achieve the result without raising suspicions.

@gort818 Do you think we could find the position bar in the DOM tree?

from qtwebflix.

I don't mean to modify the movie Position. This is meant to navigate on the streaming webpage.

from qtwebflix.

I guess we could make prev/next navigate TV show episodes. However, I'm not sure how to get to their IDs (or URLs). Any suggestions?

from qtwebflix.

Oh, i was sure about that it is possible to navigate with keyboard on the webpage. You know, with the arrow keys, enter and maybe escape/backspace for back and send this commands over mpris... but navigate with keyboard doesn't work.

from qtwebflix.

Here is how to get box art for amazon prime

document.querySelector('div.av-fallback-packshot').childNodes[0].getAttribute('src')

document.querySelector('div.av-bgimg__div').getAttribute('style')

It is one or the other.

I will try to find position bar for netflix

from qtwebflix.

@petrmanek
try this

document.querySelector('div.PlayerControlsNeo__progress-control-row.PlayerControlsNeo__progress-control-row--row-standard')

No idea how to interact though

from qtwebflix.

@petrmanek I got fast forward and backword... no seek yet :(

// fast forward
document.querySelector('button.touchable.PlayerControls--control-element.nfp-button-control.default-control-button.button-nfplayerFastForward').click()

//rewind
document.querySelector('button.touchable.PlayerControls--control-element.nfp-button-control.default-control-button.button-nfplayerBackTen').click()

from qtwebflix.

@petrmanek and here is next epsidoe

document.querySelector('button.touchable.PlayerControls--control-element.nfp-button-control.default-control-button.button-nfplayerNextEpisode').click()

from qtwebflix.

@petrmanek and here is the seek bar

document.querySelector('div.scrubber-bar')

Cannot find a way to intereact

from qtwebflix.

@petrmanek and here is the back button maybe use it as stop ?

document.querySelector('button.touchable.PlayerControls--control-element.nfp-button-control.default-control-button.button-nfplayerBack.tooltip-button.tooltip-button-pos-center.tooltip-button-align-right').click()

from qtwebflix.

For specific netflix stuff we can do something like this.

QString MprisInterface::getSite() {
    QString site = webview->url().toString();
    if ( site.contains("netflix")) {
       QString = "javascript code"
        return code
}

and then call it

void MprisInterface::getMetadata(std::function<void(qlonglong, const QString&, const QString&)> callback) {



    QString code = getSite();
              webview->page()->runJavaScript(code, [callback](const QVariant& result)

from qtwebflix.

@gort818 Wow, you've been busy! I'm going to put the code to a good use.

I was thinking about supporting multiple different services other than Netflix (like Amazon Prime, etc.), and I think there's a better solution than to basically have series of if's or switch's in every function.

Instead, I would propose to solve this by inheritance. Turn MprisInterface into an abstract interface, and then have the actual implementation present in NetflixMprisInterface and PrimeMprisInterface where it's safe to assume we're dealing with the single respective streaming service. This way we may also avoid branching the mpris logic the same way in every single function body.

The job of MainWindow would then be to communicate with MprisInterface and occasionally exchange it for other implementation when user switches services. What do you think about that?

P.S. I don't have Prime subscription, so you'll have to help me test those features.

from qtwebflix.

@petrmanek Yeah that sounds good if we are going to support other services, I only have a netflix and prime account right now, so I guess when can start with those.

Also I thought about what you said earlier I will add the qt-mpris libs as submodules and compile them as static libraries I think that is the easiest solution, I will let you know when I have pushed to the mpris branch.

Again thanks for all your hard work I am loving it !

from qtwebflix.

@petrmanek Ok got everything building with static libs https://github.com/gort818/qtwebflix/tree/mpris

I believe you just have to include the header files and you should be good to go!

from qtwebflix.

@petrmanek nvm not working..

from qtwebflix.

@petrmanek Ok I think it is good to go

from qtwebflix.

@petrmanek Ok did a nice test was able to use mpris with no issues, the executable has doubled in size from 127K to 332K :) let me know if you have any issues

from qtwebflix.

@gort818 trying now... will let you know how it went

from qtwebflix.

@gort818 Are you sure it builds from the sources?

When I tried the same thing, it worked nicely until I uninstalled qtmpris and qtdbusextended packages in /usr/lib/. From that point on, it started complaining that

Project ERROR: dbusextended-qt5 development package not found

The build directory currently looks like this:

build
├── lib
│   ├── qtdbusextended
│   │   ├── Makefile.dbusextended-qt
│   │   └── src
│   │       ├── dbusextendedabstractinterface.o
│   │       ├── dbusextendedpendingcallwatcher.o
│   │       ├── dbusextended-qt5.prf
│   │       ├── libdbusextended-qt5.prl
│   │       ├── libdbusextended-qt5.so -> libdbusextended-qt5.so.1.0.0
│   │       ├── libdbusextended-qt5.so.1 -> libdbusextended-qt5.so.1.0.0
│   │       ├── libdbusextended-qt5.so.1.0 -> libdbusextended-qt5.so.1.0.0
│   │       ├── libdbusextended-qt5.so.1.0.0
│   │       ├── Makefile
│   │       ├── moc_dbusextendedabstractinterface.cpp
│   │       ├── moc_dbusextendedabstractinterface.o
│   │       ├── moc_dbusextendedpendingcallwatcher_p.cpp
│   │       ├── moc_dbusextendedpendingcallwatcher_p.o
│   │       ├── moc_predefs.h
│   │       └── pkgconfig
│   │           └── dbusextended-qt5.pc
│   └── qtmpris
│       ├── Makefile.mpris-qt
│       └── src
└── Makefile

To me, this suggests that pkg-config does not see the dbusextended-qt5.pc file, and that's why the qtmpris target fails.

from qtwebflix.

the whole stuff is too fragmented... can we do a merge please?

For pkgbuild i think i found a solution to init the submodules:

source=("git://somewhere.org/something/something.git"
        "git://somewhere.org/mysubmodule/mysubmodule.git")

prepare() {
  cd something
  git submodule init
  git config submodule.mysubmodule.url $srcdir/mysubmodule
  git submodule update
}

from qtwebflix.

@omni6 Nice work!

In the meantime, I've refactored the code. We currently have MprisInterface base class which does the generic stuff and NetflixMprisInterface which contains JavaScripts and timers specific to Netflix.

This way, we can later add implementations for Amazon Prime, HBO Go, etc. and still have everything nicely organized.

from qtwebflix.

No wait with honour... i haven't used your patched *.pro files. When i do i can't get a proper make install to work. Yes sure because of no pkgconfig... can we get a complete install rule when doing
make -C ${srcdir}/src/src.pro INSTALL_ROOT="${pkgdir}" install

including:

• binary
• libs
• .desktop
• qtwebflix.svg

?

from qtwebflix.

@omni6 The libs are not required at runtime -- only at build time. So, you don't need to include them in the package.

You should be able to get everything just by executing the following commands in the project root:

mkdir build
cd build
qmake -c release ..
make
INSTALL_ROOT="${pkgdir}" make install

Disclaimer: I'm not sure how will handle the .desktop and .svg file.

from qtwebflix.

@petrmanek
but when the libs are not required, i can use the old way and do it manually...
i will test...

from qtwebflix.

here we go:
PKGBUILD

from qtwebflix.

@omni6 Seems nice. But we need to merge it to the base repo before going ahead with the release.

from qtwebflix.

@petrmanek sure, this is for testing. Only thing to change is url

Will wait for your go @gort818 !

from qtwebflix.

@petrmanek

Disclaimer: I'm not sure how will handle the .desktop and .svg file.

no worries, not a problem.

from qtwebflix.

@petrmanek

2. I had to create patched .pro files for both libraries. Unfortunately, this was a necessary step in order to bypass pkg-config and build both libraries as static. Both configuration files can be found in the lib/ directory along with their README.md -- they will have to be kept up-to-date with their counterparts in their respective repos.

Oops should have let you know that sorry!
Looking good, whenever you are ready to send a pull request send it on over.

A question though, how are you going to check which mpris interface to use?

@omni6 package looks good I will let you know when we are good to go.

from qtwebflix.

@gort818 Pull request underway.

A question though, how are you going to check which mpris interface to use?

MainWindow can switch interfaces based on the current URL's hostname in the QWebEngineView.

from qtwebflix.

i have to push, aur package broken...

from qtwebflix.