I will set up the AppVeyor build system to get automatic notification of failing / succeeding builds.

For the version v1 of this project (or partially when AngleSharp.Core reaches v1) some documentation is required.

The outline for the IO project is (roughly):

• Introduction
• New requesters
• IO pipeline(s) and data caches
• (Sample) use-cases w. (simple) examples
• Core interfaces
• Services and extensibility

When using HttpClient, we can disable automatic redirects by using this:

var handler = new HttpClientHandler { AllowAutoRedirect = false };
var client = new HttpClient(handler);

However, when using HttpClientRequester, the request will still be redirected.

var requester = new HttpClientRequester(client);
var configuration = Configuration.Default.WithDefaultLoader(requesters: new[] { requester });
var context = BrowsingContext.New(configuration);
var document = await context.OpenAsync("url that will be redirected");
Console.WriteLine(document.Url); // this is a different url which proves that a redirect happened

Bug Report

Prerequisites

• Can you reproduce the problem in a MWE?
• Are you running the latest version of AngleSharp?
• Did you check the FAQs to see if that helps you?
  I haven't found the FAQ
• Are you reporting to the correct repository? (there are multiple AngleSharp libraries, e.g., AngleSharp.Css for CSS support)
• Did you perform a search in the issues?

For more information, see the CONTRIBUTING guide.

Description

That's super strange bug I found. The code

var requester = new DefaultHttpRequester("Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:99.0) Gecko/20100101 Firefox/99.0");
var config = new Configuration().WithDefaultLoader().With(requester);
var document = await BrowsingContext.New(config).OpenAsync(urlAddress);

is not working if the Form was created and urlAddress is https

Steps to Reproduce

1. Download a project from here or:

• Create .NET 6 Winforms project
• Get AngleSharp from nuGet
• Paste this code

2. Launch it. Don't see "all ok" message
3. Comment var form = new Form1(); line and launch again
4. See "all ok" message
5. Uncomment the line, remove "https"//" from link
6. See "all ok" message

Expected behavior: [getting either document or exception]

Actual behavior: [infinite loop]

Environment details: [Windows 10, .NET 6]

Legacy pages may still have links to resources available on FTP servers. Therefore, the AngleSharp.Io library should provide a requester for this protocol.

1. I created new VS project
2. I installed AngleSharp 0.9.4 package, and AngleSharp.IO 0.1 package.
3. I tried to run this code:

            var requesters = new IRequester[] { };
            var service = new LoaderService(requesters);

TypeLoadException thrown.

As mentioned in AngleSharp issue 169. I'm going to copy the code and tests over from my own AngleSharp.NetHttp project.

Right now, it looks like the AngleSharp.Io package targets net45. Are there any plans to make this package (or a subset of functionality available on other TFMs?

In particular, I was wondering if the new .NET Platform standard will be supported. The piece that I worked on before (an adapter for HttpClient) should work on netstandard1.1, since this is the earliest version that supports HttpClient.

AngleSharp.Io should also come with a special scheme (by default called about) that can be extended in an IDictionary<string, Func<IRequest, Task<IResponse>>> like manner.

Essentially, this offers the possibility to integrate any kind of about-like pages in whatever scenario.

For more on the about scheme see Wikipedia.

We should slowly start to align all (official) AngleSharp extensions. .NET core should be used as the platform of choice, however, with different targets. Most likely, we add netstandard1.0 (or later) with potential external dependencies (e.g., x-plat filesystem). Important is that all features, e.g., an HTTP client as currently proposed, are available. In the end this will determine the starting target.

Hi Florian,

I scarp url using AngleSharp.Io i do this:
var handler = new HttpClientHandler { Proxy = new WebProxy($"{proxy?.Address}:{proxy?.Port}",false), PreAuthenticate = true, UseDefaultCredentials = false, };
var config = Configuration.Default.WithRequesters(handler).WithTemporaryCookies() .WithDefaultLoader(new LoaderOptions { IsResourceLoadingEnabled = true });

but i get following error :"A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond."
i do this:
var client = new HttpClient(handler: handler, disposeHandler: true) { Timeout = TimeSpan.FromMinutes(2) };

but still get the error

Is there any code samples of using custom IRequester?

I am trying to create multithreaded process where each thread will use a different proxy. I saw another issue that said this can be done with IRequester but I am not exactly sure how to implement it.

I can see it is a parameter of:

Configuration.Default.WithRequesters();

But I cannot see how I am supposed to initialize it with proxy information.

Thank you.

I am trying to get AngleSharp to use both a proxy and set header properties like this:

        var handler = new HttpClientHandler
        {
            Proxy = new WebProxy(ProxyMan.GetProxy()),
            UseProxy = true,
            PreAuthenticate = true,
            UseDefaultCredentials = false
        };
        var requester = new DefaultHttpRequester();
        requester.Headers["User-Agent"] = Tools.GetAgentString();
        requester.Headers["Accept-Language"] = "en-US";
        requester.Headers["Accept-Charset"] = "ISO-8859-1";
        requester.Headers["Content-Type"] = "text/html; charset=UTF-8";
        var config = Configuration.Default
            .WithRequesters(handler)
            .With(requester)
            .WithTemporaryCookies()
            .WithDefaultLoader();
        var context = BrowsingContext.New(config);
        var doc = await context.OpenAsync(Url);

When I added the Header requester, it stopped the proxy handler from working. I know there is some conflict between the .WithRequesters() and the .With() (or WithRequester() ) because they are separately part of AngleSharp and AngleSharp.Io but I cannot locate the proper syntax for doing both in the same request.

Apologies for the 'bug' label. Should be a help request but I can't figure out how to change that now.

Thanks.

Over the years this project had great contributors and sponsors. Moving forward the last year has shown that dedicated support (e.g., as provided by AWS and JetBrains) is crucial to allocate time for maintenance and move it forward. I'd like to continue in this mode; not only sometimes cutting out some of my spare time, but actually being able to have dedicated time slots.

So with this sticky note I call to support this project. It would be really wonderful and there are still some plans for a potential v2 that would benefit a lot from additional support (as well as its ecosystem, esp. CSS / JS / ...).

Background

In the past we already had some great sponsors who brought this project forward.

By far the largest contribution came from AWS:

Closely behind (and still active) is the sponsorship from JetBrains:

And other companies:

| | |

We also had individuals (much appreciated!) that have been very gracious:

• daveaglick
• vmeganathan81
• jfoshee
• Mrxx99
• iamcarbon
• DorYos
• LukeTOBrien
• zunjae
• Alellujah
• Gavin-Williams

🙏 thanks to everyone!

Also to be clear; AngleSharp and all associated libraries will always be free and MIT licensed.

There is no consequence from no sponsors coming in - except that my time available for the project will definitely be less as compared to having some sponsorship.

One final remark: While GitHub sponsorships are potentially the best way of supporting the project there are also other ways, e.g., getting in touch regarding consulting on AngleSharp best practices if you use it or directly getting in touch regarding development of specific features.

(Original post / issue available at AngleSharp/AngleSharp#1163)

It's available in 0.15.0, but it seems to be missing in 0.16.0

The memory cookie provider from the core has way too many problems to be the ultimate solution. Therefore, AngleSharp.Io should start introducing a better version - handling all sorts of cookies and introducing (optional) save-to-disk capabilities.

For the latter the persistent vs. session cookie question is important. To be flexible AngleSharp.Io should work against a file system abstraction that comes with a default implementation. For the moment, this should be as lightweight as possible.

Writing the cookie handling from scratch may be tedious, but should be ultimately rewarding.

Bug Report

An "unhandled exception" cause the full application crash. The configuration include a requester and also allow javascript execution.

var requester = new HttpClientRequester(httpClient);
var config = Configuration.Default.WithRequester(requester).WithDefaultLoader(new LoaderOptions { IsResourceLoadingEnabled = true }).WithJs();

But a network issues cause the app crashes. Looks like the request occurs in an isolated thread and the program can't catch the exception.

Stack Trace:

Unhandled exception. System.Net.Http.HttpRequestException: Error while copying content to a stream.
 ---> System.IO.IOException:  Received an unexpected EOF or 0 bytes from the transport stream.
   at System.Net.Security.SslStream.<FillBufferAsync>g__InternalFillBufferAsync|215_0[TReadAdapter](TReadAdapter adap, ValueTask`1 task, Int32 min, Int32 initial)
   at System.Net.Security.SslStream.ReadAsyncInternal[TReadAdapter](TReadAdapter adapter, Memory`1 buffer)
   at System.Net.Http.HttpConnection.FillAsync()
   at System.Net.Http.HttpConnection.CopyToContentLengthAsync(Stream destination, UInt64 length, Int32 bufferSize, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnection.ContentLengthReadStream.CompleteCopyToAsync(Task copyTask, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionResponseContent.SerializeToStreamAsync(Stream stream, TransportContext context, CancellationToken cancellationToken)
   at System.Net.Http.HttpContent.LoadIntoBufferAsyncCore(Task serializeToStreamTask, MemoryStream tempBuffer)
   --- End of inner exception stack trace ---
   at System.Net.Http.HttpContent.LoadIntoBufferAsyncCore(Task serializeToStreamTask, MemoryStream tempBuffer)
   at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
   at AngleSharp.Io.Network.HttpClientRequester.PerformRequestAsync(Request request, CancellationToken cancel)
   at AngleSharp.Io.BaseRequester.RequestAsync(Request request, CancellationToken cancel)
   at AngleSharp.Io.BaseLoader.LoadAsync(Request request, CancellationToken cancel)
   at AngleSharp.Browser.DefaultNavigationHandler.NavigateAsync(DocumentRequest request, CancellationToken cancel)
   at AngleSharp.Dom.Document.LocationChanged(Object sender, ChangedEventArgs e)
   at System.Threading.Tasks.Task.<>c.<ThrowAsync>b__139_1(Object state)
   at System.Threading.QueueUserWorkItemCallback.<>c.<.cctor>b__6_0(QueueUserWorkItemCallback quwi)
   at System.Threading.ExecutionContext.RunForThreadPoolUnsafe[TState](ExecutionContext executionContext, Action`1 callback, TState& state)
   at System.Threading.QueueUserWorkItemCallback.Execute()
   at System.Threading.ThreadPoolWorkQueue.Dispatch()

Provide a nuspec file and a PowerShell script to automatically deploy the most recent version.

The AngleSharp.Io library should provide the WebSocket interface and a proper implementation of it. It official specification can be found at http://www.w3.org/TR/2011/WD-websockets-20110419/#the-websocket-interface.

Right now the interface is directly implemented in a class. This approach seems to be fine. Some method implementations are missing though.

The AdvancedCookieProvider does not cater for cookies lacking the "www" on the front.

The website I am working with has cookies with the "www.websitename.com" domain and cookies with the ".websitename.com" domain.

On adding the cookies the code currently strips off the "." and when selecting cookies to send on the next request, ignores all these cookies because they don't match the host name.

Should the code be altering the Domain name of cookies??

I think it needs to leave the Domain alone when set, and do a comparison on the basis of the host and the cookie domain excluding a leading "www." or "."

New Feature Proposal

Currently this doesn't support the same .NET standard versions as the normal AngleSharp package.

Description

It would be nice if the io package supported the same versions for .net framework and .net standard as the normal AngleSharp. The issue seems to have been discussed in #9 but from looking at the csproj it currently only support Windows NT and .net standard 2.0

Background

Easier framework alignment with AngleSharp

Local webpages are opened using the file:// protocol. The relative links will also follow this scheme requiring a requester to understand the protocol and fetch local resources. Therefore, the AngleSharp.Io library should provide a requester for this protocol.

Sample code:

var config = Configuration.Default.WithDefaultLoader().WithCookies();
var context = BrowsingContext.New(config);
await context.OpenAsync("http://www.xxx.com/");
var doc = await context.OpenAsync("http://www.xxx.com/abc.jpg");          
var data = doc.Body.TextContent;

When debug into the code, I can see the binary data of the image has been converted to UTF-8 string in doc.Boday.TextContent. That means the image has been downloaded. However I can't restore the image binary data with the UTF-8 string as some data has lost during the conversion.

Is there a way that I can get the original binary data of the image? For some reason, I can't download the image using regular way.