evdb / http-async Goto Github PK

Started writing this as a comment on one of the open PRs, but decided it should be its own issue.

We've got 3 open issues and 2 open PRs. I don't have a lot of time to look at them, especially since none of my current projects are in Perl 😢

@evdb are you interested in maintaining this at all?

If not I'd be happy to do a bit of work to try to find a new project maintainer. I could write a blogs.perl.org post, post on London.pm, and/or give comaint on CPAN to ADOPTME.

@evdb / @kaoru - I would love to help make a Cache (HTTP::Async::Cached) sub module that will cache responses for the HTTP clients to reduce the throttling this module can do.

Any ideas on how to do that? I am reading the code base and wanted to get your feedback on the best way to do it?

Love this module btw, use it often for a lot of my web tools. It is amazing fast and versatile. Kudos.

Is there any way we can speed up the test suite?

When I need to install my app along with all dependencies, this module is esp. slow (2.75 minutes to run tests).

It's mainly caused by the t/bad-hosts.t test file. If I set the timeout() to 5 seconds there, the overall test runtime drops to ~ 40 seconds. Dunno if having a shorter timeout there is acceptable or not.

I'm on Linux Mint 17 with Perl 5.18.1

This module does not seem to work for https urls. Am I doing something wrong, or does this module not support https?

Hi,

Three HTTP::Async test files establish a listening HTTP service using IO::Socket::INET, attempt to test that service using "http://localhost:$port" url_roots, and then run into the problem on hosts with IPv6 that localhost resolves to ::1 rather than 127.0.0.1 and so connections to the test service will fail.

1. t/dead-connection.t - could be fixed by either using IO::Socket::IP rather than IO::Socket::INET, or by explicitly creating the request using 127.0.0.1:

my $req = HTTP::Request->new( GET => "http://127.0.0.1:$port/" );

I don't know which is the more portable solution.

2. In both t/bad-connections.t and
3. t/peer-addr.t

TestServer->new() results in an IPv4 socket being created.
$s->started_ok() returns "http://localhost:$port"

I've submitted a bug for Test::HTTP::Server::Simple (https://rt.cpan.org/Public/Bug/Display.html?id=124363) with a suggestion that would address these two cases, as well as possibly any other modules that follow the Test::HTTP::Server::Simple synopsis.

Otherwise, these two cases could be addressed by the test files using "http://127.0.0.1:$port"rather than the string returned by started_ok().

With both suggestions, 127.0.0.1 might not be a fully portable assumption, but the number of people that might catch out is probably a lot smaller than the number of people having IPv6 capable hosts.

Many Thanks,
Rich

Hi,

I'm having problems with this wonderful module when setting a special Accept header and getting a 303 redirect from the server.

I cannot get the following request with HTTP::Async:

curl --head -H 'Accept: application/x-research-info-systems' http://dx.doi.org/10.1126/science.169.3946.635                                         

HTTP/1.1 303 See Other
Server: Apache-Coyote/1.1
Vary: Accept
Location: http://data.crossref.org/10.1126%2Fscience.169.3946.635
Expires: Fri, 21 Mar 2014 10:24:17 GMT
Content-Type: text/html;charset=utf-8
Content-Length: 196
Date: Thu, 20 Mar 2014 13:50:37 GMT

Testing it in Perl with HTTP::Async 0.24:

use strict;
use warnings;
use v5.10;

use HTTP::Async;
use HTTP::Request;

my $async = HTTP::Async->new();
my $headers = HTTP::Headers->new( Accept => 'application/x-research-info-systems' );

my $error = $async->add(HTTP::Request->new( GET => 'http://dx.doi.org/10.1126/science.169.3946.635', $headers));
my $ok = $async->add(HTTP::Request->new( GET => 'http://data.crossref.org/10.1126%2Fscience.169.3946.635', $headers));

while ( my ($response, $req_id) = $async->wait_for_next_response ) {
        say "$req_id) " . $response->code . ": " . $response->message . " from: " . $response->base;
        say "Content-Type is: ". $response->header('Content-Type');
}

I would have hoped that both requests would be fine but it is not the case:

2) 200: OK from: http://data.crossref.org/10.1126%2Fscience.169.3946.635
Content-Type is: application/x-research-info-systems
1) 500: Internal Server Error from: http://data.crossref.org/10.1126%2Fscience.169.3946.635
Content-Type is: text/html

It would look as if HTTP::Async does not retain the original header.

If you let curl follow the redirect it works as I would expect:

curl -LH 'Accept: application/x-research-info-systems'  http://dx.doi.org/10.1126/science.169.3946.635                                           

TY  - JOUR
DO  - 10.1126/science.169.3946.635
UR  - http://dx.doi.org/10.1126/science.169.3946.635
TI  - The Structure of Ordinary Water: New data and interpretations are yielding new insights into this fascinating substance
T2  - Science
AU  - Frank, H. S.
PY  - 1970
DA  - 1970/08/14
PB  - American Association for the Advancement of Science (AAAS)
SP  - 635-641
IS  - 3946
VL  - 169
SN  - 0036-8075
SN  - 1095-9203
ER  - 

Thanks,
Kasper

Not sure if this is by design or a bug, but when I'm adding HTTP::Requests to https sites it seems there is a small blocking delay after adding each request. With 30 requests, it takes ~ 4 seconds to add them to the HTTP::Async object before I can begin polling for responses.

I initially thought maybe a blocking DNS lookup was to blame but the hostnames are all in the localhost DNS cache so it should be very fast. Any ideas?

I noticed this when my Jenkins instance (running on port 8080) tried to install HTTP::Async, which is a dependency of the application I'm trying to test, but it wouldn't surprise me if I've sent CPAN-testers failure reports for it too because of the same thing, when I try to test it in multiple perl builds at the same time.

A quick n dirty fix would be to try to bind the HTTP::Server::Simple used for testing to 8080 but if that fails try 8081, then 8082, etc

I think it should be as simple as adding a new() method to TestServer.pm that is just a wrapper around SUPER->new(), but which after calling the parent constructor sets $self->port(...), tries to start, if not increments and repeat ad infinitum. you may want to then stop the server (leaving it configured with a known-good port) so that start_ok works - but maybe not, given that you're not actually testing the server but testing an async client.

LWP as LWP::Auth::Negotiate to achieve GSSAPI connection. Was wondering if you do point me to how I can estabilsh secure kerberos connection.