dotnettools / sharpgrabber Goto Github PK

Hi , Javid
First of all thank you for making this lib <3

Second , I have this exception All time ! does YouTube ban IP or something else ?

I try it on my Germany Server but Exception happened again !

GrabException: Failed to get metadata information from https://youtube.com/get_video_info?video_id=nErrHNfZEAk&eurl=https%3A%2F%2Fyoutube.googleapis.com%2Fv%2FnErrHNfZEAk&html5=1&c=TVHTML5&cver=6.20180913.

can you please update the desktop version of the app? the link seems dead.

Hey, thanks for creating SharpGrabber. I'm using it in one of my apps and it is very useful for downloading audio tracks from YouTube!

Today, I encountered the following exception when grabbing a YouTube video:
Failed to locate source code of container object for helper functions.

It seems you already fixed this issue in the following commit: 869ba54
But the fix is not included in the latest release of SharpGrabber.YouTube on nuget. Could you create a new release with the fix included?

Lastest version of SharpGrabber.Desktop(v3.2)

Error:
  An assembly specified in the application dependencies manifest (SharpGrabber.Desktop.deps.json) was not found:
    package: 'Microsoft.Win32.SystemEvents', version: '4.5.0'
    path: 'runtimes/win/lib/netcoreapp2.0/Microsoft.Win32.SystemEvents.dll'

Also, maybe best way for GUI app is that have self-contained version. It's can have large size, but works out of box and do not requires installed runtime.

Subj. Sorry my english.

Is there a simple example of how to download and save a video to a file using the link ?

Hello:
I want to use your repo to download some live streaming data with u3m8 URL.
But I want to do some testing to see if I can use your repo in C# console project or not.
I have done this:
I downloaded your repo, and change target platform to x64, and rebuilt solution with 4 projects, and it works.
Then I created one C# console project targetting .NET 5.0, and add project reference for 3 DLLs, which were built by repo target x64 platform.
The 3 DLLs are:
SharpGrabber.Converter.dll
SharpGrabber.Desktop.dll
SharpGrabber.dll
I also install xunit nuget package:
PM> Install-Package xunit -Version 2.4.1
The following is my code:
using DotNetTools.SharpGrabber.Internal.Grabbers;
using System;
using System.Diagnostics;
using Xunit;

namespace SharpGrabberDownload1
{
class Program
{
public static async void Test_YouTube_Normal()
{
try
{
var grabber = new YouTubeGrabber();
var result = await grabber.GrabAsync(new Uri("https://www.youtube.com/watch?v=ThUS9fj-mCA"));
Assert.False(result.IsSecure);
Assert.Equal(23, result.Resources.Count);
}
catch (Exception ex)
{
Debug.Print(ex.Message);
}
}

    static void Main()
    {
        Test_YouTube_Normal();
    }
}

}
It is simply one of your file (only one test):
public class GrabTests
in DotNetTools.SharpGrabber.Tests NameSpace.
I can get my program compiled, but when I run it or debug it, nothing happens. I can’t stop at var result = await grabber.GrabAsync(new Uri("https://www.youtube.com/watch?v=ThUS9fj-mCA"));
The program simply ends without any error message.
Please let me know if it is a correct way to use your repo in a C# console project or in a C# WinForm project.
If not, then which way is correct?
I want to use your repo in either C# console project or WinForm project, not others.
I am using Windows 10 Version 21H1, and Visual Studio 2019 Version 16.10.4
Thanks,

This may be of help... ??

// Found here: https://bytes.com/topic/c-sharp/answers/870187-login-youtube-using-c

private void login() {

      CookieContainer cookieContainer = new CookieContainer();
      HttpWebRequest PreRequest = (HttpWebRequest)WebRequest.Create("https://www.google.com/accounts/ServiceLogin?utm_source=/places&utm_medium=van&utm_campaign=en&continue=http://www.google.com/local/add/businessCenter%3Fservice%3Dlbc%26gl%3DUS%26hl%3Den-US&service=lbc&hl=en-US&gl=US");
      PreRequest.CookieContainer = cookieContainer;
      PreRequest.AllowAutoRedirect = true;
      HttpWebResponse PreResponse = (HttpWebResponse)PreRequest.GetResponse();
      Stream PreResponseStream = PreResponse.GetResponseStream();
      PreResponse.Cookies = PreRequest.CookieContainer.GetCookies(PreRequest.RequestUri);
      Encoding PreEnc = System.Text.Encoding.UTF8;
      StreamReader PreResponseStreamReader = new StreamReader(PreResponse.GetResponseStream(), PreEnc, true);
      String PreMyHTML = PreResponseStreamReader.ReadToEnd();
 
      String Pattern = "name=\"GALX\"[\\s]*value=\"([a-zA-Z0-9-_\\.]*)\"";
      Match MyMatch = Regex.Match(PreMyHTML, Pattern);
      String GALX = MyMatch.Groups[1].ToString();
 
      HttpWebRequest Request = (HttpWebRequest)WebRequest.Create("https://www.google.com/accounts/ServiceLoginAuth?service=lbc&ltmpl=sso&continue=http%3A%2F%2Fwww.google.com%2Flocal%2Fadd%2FbusinessCenter%3Fservice%3Dlbc%26gl%3DUS%26hl%3Den-US&service=lbc&uilel=3&ltmpl=sso&hl=en_US&ltmpl=sso&GALX=" + GALX + "&Email=USERNAME%40gmail.com&Passwd=PASSWORD&rmShown=1&signIn=Sign+in&asts=");
      Request.CookieContainer = cookieContainer;
      Request.Method = "POST";
      Request.Referer = "https://www.google.com/accounts/ServiceLogin?utm_source=/places&utm_medium=van&utm_campaign=en&continue=http://www.google.com/local/add/businessCenter%3Fservice%3Dlbc%26gl%3DUS%26hl%3Den-US&service=lbc&hl=en-US&gl=US";
      Request.MaximumAutomaticRedirections = 50;
      Request.AllowAutoRedirect = true;
      Request.KeepAlive = true;
      Request.ContentLength = 0;
 
      // Get the response
      HttpWebResponse Response = (HttpWebResponse)Request.GetResponse();
 
      Stream ResponseStream = Response.GetResponseStream();
      Response.Cookies = Request.CookieContainer.GetCookies(Request.RequestUri);
 
      // Read the response from the stream
      Encoding Enc = System.Text.Encoding.UTF8;
      StreamReader ResponseStreamReader = new StreamReader(Response.GetResponseStream(), Enc, true);
 
      String MyHTML = ResponseStreamReader.ReadToEnd();
      this.Cookies = cookieContainer;
      this.Response.Write(MyHTML);
 
      String Pattern2 = "location\\.replace\\(\"(http:\\/\\/www\\.google\\.com\\/local\\/add\\/businessCenter[a-zA-Z0-9\\\\\\?\\%_-]*)\"\\)";
      Match MyMatch2 = Regex.Match(MyHTML, Pattern2);
      String YTRedirect = MyMatch2.Groups[1].ToString();
      String Temp = YTRedirect.Replace("\\x3d", "=");
      String Temp2 = Temp.Replace("\\x26", "&");
      YTRedirect = Temp2;
 
      HttpWebRequest PosterRequest = (HttpWebRequest)WebRequest.Create(YTRedirect);
      PosterRequest.CookieContainer = cookieContainer;
      PosterRequest.Method = "GET";
      HttpWebResponse PosterResponse = (HttpWebResponse)PosterRequest.GetResponse();
      Stream PosterResponseStream = PosterResponse.GetResponseStream();
      PosterResponse.Cookies = PosterRequest.CookieContainer.GetCookies(PosterRequest.RequestUri);
      Encoding PosterEnc = System.Text.Encoding.UTF8;
      StreamReader PosterResponseStreamReader = new StreamReader(PosterResponse.GetResponseStream(), PosterEnc, true);
      String PosterMyHTML = PosterResponseStreamReader.ReadToEnd();
 
    }

But I don't know how or where to best integrate this and thought I'd ask before digging into the code.

This might or might not be a bug, I'm not too familiar with the HLS spec. I'm trying to grab the following root m3u8:

#EXTM3U
#EXT-X-VERSION:3
#EXT-X-STREAM-INF:BANDWIDTH=97341,CODECS="mp4a.40.5"
https://aradiostation.com/hls-radio/gold/chunklist.m3u8

In the Playlist document the code always expects the RESOLUTION uri token to be present (Link to check). In this case the m3u8 does not have it. Is this radio station using a non compliant format or does the HLS grabber always assume that it tries to grab video instead of audio only.

Or am I just doing something wrong :')

Stack trace:

   at System.Text.RegularExpressions.ThrowHelper.ThrowArgumentNullException(ExceptionArgument arg)
   at System.Text.RegularExpressions.Regex.Match(String input)
   at DotNetTools.SharpGrabber.Hls.PlaylistDocument.ParseUriToken(LoadContext context)
   at DotNetTools.SharpGrabber.Hls.PlaylistDocument.LoadAsync(PlaylistTokenizer tokenizer)
   at DotNetTools.SharpGrabber.Hls.HlsGrabber.InternalGrabAsync(Uri uri, CancellationToken cancellationToken, GrabOptions options, IProgress`1 progress)
   at DotNetTools.SharpGrabber.GrabberBase.GrabAsync(Uri uri, CancellationToken cancellationToken, GrabOptions options, IProgress`1 progress)
   at DotNetTools.SharpGrabber.MultiGrabber.InternalGrabAsync(IEnumerable`1 candidateGrabbers, Uri uri, CancellationToken cancellationToken, GrabOptions options, IProgress`1 progress)

For some YouTube videos the grabbed results contain links for more than one .mp4 files with same resolution, which when muxed with the .m4a audio file using MediaMuxer on android, one of them fails. It only happens if files with invalid/unsupported format are passed to the MediaMuxer...tell me if I am missing on knowing something about file formats, else please look into this...

Curl error 3: malformed

Pretty much says it all, I'm basically using your example and this is a bit of a mystery bug in my project.

var grabber = new YouTubeGrabber();
var result = await grabber.GrabAsync(new Uri("<URL of a YouTube video>"));
IList<IGrabbed> grabbedResources = result.Resources;

As far as I can tell the fix requires updating curl.
Is this an old example of usage? I noticed the multigrabber doesn't work, am I missing something?

Hello:
I found the nuget version of SharpGrabber.HLS. I want to see if I can use this new nuget package to download some M3U8 HLS and save to my local hard disk.
I created one C# console project, target .NET 5.0.
And add the following NUGET packages:
PM> Install-Package SharpGrabber.Hls -Version 1.0.0
PM> Install-Package SharpGrabber -Version 2.0.0

The following is my C# code:

using DotNetTools.SharpGrabber;
using DotNetTools.SharpGrabber.Exceptions;
using DotNetTools.SharpGrabber.Grabbed;
using DotNetTools.SharpGrabber.Hls;
using System;
using System.IO;
using System.Net.Http;
using System.Threading.Tasks;

namespace SharpGrabberDownloader
{
	class Program
	{
		public const string M3U8_Link1 = 
			@"http://playertest.longtailvideo.com/adaptive/oceans_aes/oceans_aes.m3u8";
		public const string Local_MP4_File = 
			@"C:\Videos\oceans_aes.mp4";

		private static readonly HttpClient _http = new HttpClient();
		private static async Task Download(Uri uri)
		{
			var downloader = new MediaDownloader(grabbed, path, CurrentGrab);
			await downloader.DownloadAsync();
		}

		static void Main(string[] args)
		{
			var uri = new Uri(M3U8_Link1);
			await Download(uri);
			Console.WriteLine("Done!");
		}
	}
}

But I have some issues:

1. From solution explorer, I can see the yellow warning mark besides the SharpGrabber.Hls nuget package, it could mean that SharpGrabber.Hls is not compatible with .NET 5.0, right?
2. I got compiler error:
   The type or namespace name 'MediaDownloader' could not be found (are you missing a using directive or an assembly reference?)
   It seems the MediaDownloader is NOT a public accessible type.
   How I can solve this?
   Finally, what I want to do is just download the M3U8_Link1, and save as a MP4 file, saved at
   Local_MP4_File, but I need download the HLS asynchronously, as I have many other similar HLS to download, some of them are in real-time, as some sports games videos on going.
   But I also need the downloader can stop and combine all downloaded parts to make one MP4 file after the download finishes or the HLS stops sending more data.
   Please advise,
   Thanks,

Discussed in #81

after two or three request , i get this !

...

Hello, vulgar indecent pornographic sites are contrary to human dignity and pono actresses have spoken of rape and sexual assault suffered, these sites destroy society and the family so it will be good to remove the support for these disgusting sites, thank you !

Discussed in #82

Hi. Sometimes the videos have not image or 5 images, sometimes its just 3 or 2 but GrabbedImage Class show us 5 and with out any images.

forexample this link
https://www.youtube.com/watch?v=2g8JeSq4Ofc

its just have 4 image . but the GrabbedImage is 5 . and the last one is this
https://img.youtube.com/vi/2g8JeSq4Ofc/maxresdefault.jpg

what should we do for this problem?

Looked at your wiki and readme. Code examples of how to use grabber to do something as easy as downloading a youtube video + youtube audio and muxing them into a single file is , i believe, one way you could dramatically improve adoption. Just a suggestion to add a few examples that others can emulate. I did look through your tests, I also did not see the above example.

Thanks and Best wishes,

Hello, is there an example to send a login and receive cookies before grabbing from a site (eg. Instagram, fb etc.)?
BTW, perfect job!!

Some videos on YouTube such as https://www.youtube.com/watch?v=Fpgd3ac3_nM are missing adaptive_fmts and url_encoded_fmt_stream_map. In such cases, video info seems to exist in player_response. This issue is about adding support for such videos.

Actually no I lied, last we spoke you asked me if there were problems with sharp grabber. And I said no because I couldn't think of it at the time. But there are 3 issues...

1. The multi grabber example is broken
2. The xxx grabbers are obsolete because those sites charge for videos now and locked up shop, of course this was the second or first thing I tried
3. If you're using unity at least, and you compile your own source there are two pesky errors from ffmpeg..

NoiseLib::GatherNoiseTypes: Failed to get types from assembly: FFmpeg.AutoGen, Version=4.3.1.0, Culture=neutral, PublicKeyToken=9b7632533a381715 UnityEditor.EditorAssemblies:ProcessInitializeOnLoadAttributes (System.Type[])

NoiseLib::GatherFractalTypes: Failed to get types from assembly: FFmpeg.AutoGen, Version=4.3.1.0, Culture=neutral, PublicKeyToken=9b7632533a381715
UnityEditor.EditorAssemblies:ProcessInitializeOnLoadAttributes (System.Type[])

I don't know if you will fix these things, or someone else will, but do understand I likely need to build in ffmpeg into my project but meh.. needs a lite version without anything but youtube and vimeo and muxed streams only to dump ffmpeg. I hate useless baggage.

Hi, I'm trying to download a video from youtube. This video cannot be found or downloaded by SharpGrabber, there are some URLs, not all. Does anyone know why this happens? Is there something different in the video posted? If anyone can help with any tips or solutions. Thank you

Hello:
I can use VLC Media Player to play the video with the following M3U8 link:
https://bitdash-a.akamaihd.net/content/MI201109210084_1/m3u8s/f08e80da-bf1d-4e3d-8899-f0f6155f6efa.m3u8
And I can save the stream as a video in my local C: drive.
I want to know if I can do this with SharpGrabber.
I didn’t see any example on how to save live stream into a video file in local hard drive.
Please give me an example on how to do this, you can use the above M3U8 link.
By the way, I am using Windows 10 (Version 21H1), and my IDE is Visual Studio 2019 (Version 16.10.4)
Thanks,

The converted formats are played flawlessly on PC and Android according to recent tests by myself.
However, I've tested some of these videos on PS4 and its media player is glitching after sometime for each video. This must be diagnosed and fixed.

Install-Package : A parameter cannot be found that matches parameter name 'Version'.
At line:1 char:30

• Install-Package SharpGrabber -Version 2.0.2

•                          ~~~~~~~~
  

  • CategoryInfo : InvalidArgument: (:) [Install-Package], ParameterBindingException
  • FullyQualifiedErrorId : NamedParameterNotFound,Microsoft.PowerShell.PackageManagement.Cmdlets.InstallPackage

Add a few extension methods to conveniently get highest quality.

Nullreference Exception is thrown by YouTube grabber since yesterday. I looked at your code and checked some urls... https://www.youtube.com/get_video_info return 410 GONE.. maybe this could be the problem.

We've already defined image size for grabbed images. An estimated image quality is also necessary to let library users choose the image with most or least quality or resolution. ImageQuality shall be defined as double to let grabbers proportionally define the quality or resolution of the image. For example, YouTube provides images with qualities hq, lq etc. in which case we don't know the size of the images and YouTubeGrabber does not provide any information about the difference of GrabbedImages. However thanks to this new property defined, the metadata of those images are now different.

hi sir. how should we use Instagram Grabber ? i use this code

var grabber = MultiGrabber.CreateDefault();
var result = await grabber.Grab(new Uri("https://www.instagram.com/p/B7JJacRFh37/?igshid=16wmbgm2oa5m9"));
List grabbedResources = result.Resources;

but i get this error = >
DotNetTools.SharpGrabber.Exceptions.GrabParseException: 'Failed to obtain metadata from the Instagram page.'

OR

var grabber = new InstagramGrabber();
var result = await grabber.Grab(new Uri("https://www.youtube.com/watch?v=b3ChL9mqPkM"));
IList grabbedResources1 = result.Resources;

i get the same error

Using SharpGrabber Desktop, I tried grabbing multiple youtube vids and they all download super slow. It took almost 10 min. to download an 11MB audio file. Vimeo files also don't work - it gives me a "Grab Error - the URL is not supported" message. I tried https://vimeo.com/692425535.

Am I supposed to configure something first?

I have m3u8 playlist and when I choose desired video quality (eg. full HD) how to extract ts files so I can make playable file?

When you minimize the main form if a video is in the process of downloading and then unminimize it after some time (at least 5 minutes), often the restored window is completely black. If you resize the window a little, it triggers the form to be redrawn.

I think it would be cleaner to have a config file that uses this threshold, it would be more readable too.
By the way is Math.Abs necessary?

Downloading a 1MB video takes ca. 2 minutes..

var grabber = GrabberBuilder.New()
  .UseDefaultServices()
  .AddYouTube()
  .AddVimeo()
  .Build();

var grabResult = await grabber.GrabAsync(new Uri("someYtVideoUrl"));
var mediaFiles = grabResult.Resources<GrabbedMedia>().ToArray();

using (var client = new HttpClient())
using (var response = await client.GetAsync(mediaFiles[5].ResourceUri))
{
	response.EnsureSuccessStatusCode();
	using (var downloadStream = await response.Content.ReadAsStreamAsync())
	{
		// ...
	}
}

Maybe this is a HttpClient issue..?

Using this release I am not able to get pornhub video urls. The quality json is empty.

I was test this url "https://www.instagram.com/p/CZRJeuUNXHV"

but not working and show this message "An item with the same key has already been added. Key: instapp:hashtags"

If I searched YouTube videos, I could only find mp4 and webm. The audio format does not support mp3, so how can I get it without conversion?

Other websites provide mp3, so it seems to be something that needs to be implemented on the code.

I want to make this into xamarin. so I just want to provide download feature

It says I need .NET Core 3.1. I searched for it and got this https://dotnet.microsoft.com/en-us/download/dotnet/3.1
Which do I download? The SDK 3.1.417 or the ASP.NET Core Runtime 3.1.23?

Then I just need the SharpGrabber Desktop and it'll work?

Also the screenshot example you have of it grabbing the youtube video. It says Download and Convert. Is it re-encoding the file or directly downloading it? I don't want a conversion of the file I want a direct download. Is that what this program does?

Anyone willing to create the code because I have no idea on how to create it, due to me not having any coding experience with dotnet. Also I would like to do this w/o having to make it using tokens if possible as they are annoying to create

https://www.youtube.com/watch?v=4mECX60FVJQ
GrabbedMedia.ResourceUri=
https://r1---sn-h51xq5-q5jl.googlevideo.com/videoplayback?expire=1631161132&ei=zDY5Yej6Iear3LUPs9Ka0Ao&ip=203.95.221.49&id=o-ABO__WkAtQ021lkpUUBN4Q-l21x9nJPZRqZ6fWVyYWFt&itag=137&aitags=133%2C134%2C135%2C136%2C137%2C160%2C242%2C243%2C244%2C247%2C248%2C278&source=youtube&requiressl=yes&mh=B9&mm=31%2C29&mn=sn-h51xq5-q5jl%2Csn-h5576n7k&ms=au%2Crdu&mv=m&mvi=1&pl=25&gcr=bd&initcwndbps=1133750&vprv=1&mime=video%2Fmp4&ns=SCA4q_Twbe1wjU3a5jP9yzgG&gir=yes&clen=114691908&dur=344.969&lmt=1631121474701980&mt=1631139166&fvip=3&keepalive=yes&fexp=24001373%2C24007246&beids=23886212&c=WEB&txp=5311224&n=XHDKcdJmGD-lvWA&sparams=expire%2Cei%2Cip%2Cid%2Caitags%2Csource%2Crequiressl%2Cgcr%2Cvprv%2Cmime%2Cns%2Cgir%2Cclen%2Cdur%2Clmt&lsparams=mh%2Cmm%2Cmn%2Cms%2Cmv%2Cmvi%2Cpl%2Cinitcwndbps&lsig=AG3C_xAwRgIhAL1ME-AV9Y5eKslMSJKV-yeqqDLQiJqFJ5FE-pnO_2TFAiEA3jMi_dEADFqivW_MV--LadrdIljcjFTQrkHnZ31CxIQ%3D&sig=0QJ8wRAIgUDEKqmHmySZJBwn3Dix4f9BYyLUAsYKtwAVM5Lb6Ld8CIHPzLYegkAbIbPkV9DJyR0mj1GLfhu3B9qtMT7e5_D6o

Most videos in this link are not working https://www.youtube.com/hashtag/vevo

Hello!!!
I download slow from YouTube.
See yt-dlp/yt-dlp#6369.

Hello.

I am using this code

var grabber = GrabberBuilder.New()
                .UseDefaultServices()
                .AddHls()
                .Build();

 var grabResult = await grabber.GrabAsync(new Uri("https://cs4-9v4.vkuseraudio.net/s/v1/ac/pYZ1qXNzF1JtetJ5187d-Ur6TUGJsFrBugQJeHTMCCL9Lx8LSZUi0DlPe_4JmRDD50P6fOWvDlmld_XCxXxAwyZ5SaU2B1NN9NSyY3UIsagcJdLTVrT7qhcHxg_mS6rrfbrQAQRMn0ZHXEMaJgN3NI4VC3GY_VUIsil_1G7w0DOA5oo/index.m3u8"));

and it returns some information, but I do not understand how to download all parts of that m3u8 file to one ts file and then convert it to mp3. Do we have some code example of how to do it?

Duke

I'm trying to make it compile in my machine, I have .Net Core 3.1 installed, and I'm receiving the following errors... I tried some workaround but without success, do you know the cause? Thanks for your work.

Microsoft.Common.CurrentVersion.targets(3222, 5): PFX signing not supported on .NET Core
NuGet.Build.Tasks.Pack.targets(198, 5): [NU5026] The file 'C:\projects\Grabber\src\SharpGrabber\bin\Debug\netstandard2.0\SharpGrabber.dll' to be packed was not found on disk.