jayjamesjay / http_req Goto Github PK

I have no clue how to do a get request with headers, I have tried the request builder which after calling send just hangs nothing happens.

Hi,
I want to get my dependencies up to date but I'm blocked with rustls as you use version 0.17.

Is it possible to update rustls to 0.21.7 ?
Thanks !

http_req currently does not allow specifying a timeout for the entire request (i.e. until the request body is finished transferring), which means an established connection will keep going indefinitely if the remote host keeps replying.

This is fine for the simple use cases, like a user downloading something interactively, but enables denial-of-service attacks in automated scenarios: if the remote host keeps transferring data at a really low speed, e.g. several bytes per second, the connection will be open indefinitely. This makes it easy for an attacker who can submit URLs to the server to cause denial of service through some kind of resource exhaustion - running out of RAM, networking ports, etc.

It seems the library directly manages TCP connections, thus is unaware to proxies.

It would be nice to support making connections through proxy, ideally respecting environment variables such as https_proxy or socks_proxy like cURL does. This would make http_req usable in many corporate networks.

This is the error I see when upgrading from 0.7 -> 0.8:

   Compiling webpki v0.22.0
   Compiling webpki-roots v0.22.0
   Compiling http_req v0.8.0
error[E0308]: mismatched types
  --> /home/wez/.cargo/registry/src/github.com-1ecc6299db9ec823/http_req-0.8.0/src/tls.rs:82:39
   |
82 |             .add_server_trust_anchors(&webpki_roots::TLS_SERVER_ROOTS);
   |                                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expected struct `webpki::TLSServerTrustAnchors`, found struct `TlsServerTrustAnchors`
   |
   = note: expected reference `&webpki::TLSServerTrustAnchors<'_>`
              found reference `&TlsServerTrustAnchors<'static>`

error[E0308]: try expression alternatives have incompatible types
   --> /home/wez/.cargo/registry/src/github.com-1ecc6299db9ec823/http_req-0.8.0/src/tls.rs:148:13
    |
148 | /             webpki::DNSNameRef::try_from_ascii_str(hostname.as_ref())
149 | |                 .map_err(|_| HttpError::Tls)?,
    | |_____________________________________________^ expected struct `webpki::name::DNSNameRef`, found struct `DnsNameRef`

Hi, I am working with this repo to send a post request to my server. But I always get a error, which is "status":400,"error":"Bad Request","message":"Required request body is missing:...". It seams that the server cannot get the body. Please help me out.

Strive to make more use of const fn, which has been recentyl improved and gained new features. This will reduce the run time cost (in specific scenarios) and enable more use cases for current functions.

it will block in get/post when the response is 204

Are there any plans on adding async support?
Let me know if a pull request in that direction would be welcome.

Hello developers,

In version 0.4.4, this code fails:

use std::io;
use http_req::request::Request;
fn main() {
    let uri = &"https://bing.com".parse().unwrap();
    if let Err(err) = Request::new(&uri).send(&mut io::sink()) {
        eprintln!("{:?}", err);
    }
}

The message printed is: Parse(Invalid).

If the url is changed to "https://www.bing.com", it works fine.

May I ask if you could fix?

Thank you,

Edited

"https://bing.com" returns 301, while "https://www.bing.com" returns 200. Maybe it was the cause?

Hello,

i try to use this Library for an simple cookie-request with an json-response and an file-download with an cookie-request.

1. Send Header with cookies
2. receive as Response an json (chunked?)
3. request file for download with cookies

thx

Hello,
i have a little problem, for that code:

let steamUrl: Uri = "https://store.steampowered.com/api/appdetails/?appids=794270&filters=basic&l=de".parse().unwrap();
let mut writer = Vec::new();
let res = request::get(steamUrl.to_string(), &mut writer).unwrap();

He don't respect the &l=de parameter and send all response content in english. The filters parameter he accept fine. I think it is the Request-Header that i must tweak. I think i must send Accept-Language like Firefox. How can i made this?

thx

Hello,

In version 0.4.4 this call fails: Uri::from_str("https://bing.com").

May I ask if you could fix?

Thank you,

Hi,

Currently there is no option for read timeout on internal TcpStream instances. May I ask if you could add new option(s) for that?

I would like to code and send you a pull request for your review. If you agree, please let me know so I can start. Please give me some directions, such as:

• Name of new function(s).
• Place(s) where to put them.
• Should those functions take a Duration, or Option<Duration>? If it's an Option, which default value should be?
• ...

Thank you,

I'm tryinng to implement a HTTPS2 client, but it failed with status 400.

The repreduction project goes as https://github.com/sammyne/http-playground/tree/master/v2.

Really appreciate if someone can help me out~

Hello!

First of all, thanks for the lightweight library.

I've noticed that raw IPv6 addresses in HTTP URLs cannot be parsed as an internal Uri, and thus cannot be connected to.

Here's an example of a valid URL w/ an IPv6 address:

https://[2a03:b0c0:3:d0::ba8:8001]:443/

And the error that's generated when attempting to .parse() the URL string:

Err(Parse(UriErr))

Hi,

I am not sure what is happening:

Cargo.toml:

http_req = {version="0.4.9", default-features = false, features = ["rust-tls"]}

fn fetch_adlist() {
    let adlist_url = "https://gist.githubusercontent.com/l1x/51e7ec94526dc7d0aba77b31378df48b/raw/a4515ac76d41a29df971e4ceab9e9916b71eea1b/tmux.conf";
    let mut writer = Vec::new();
    let res = request::get(adlist_url, &mut writer);
    match res {
        Ok(res) => info!("Status: {} {}", res.status_code(), res.reason()),
        Err(err) => debug!("{:?}", err),
    }
}

thread 'main' panicked at 'called `Option::unwrap()` on a `None` value', src/libcore/option.rs:345:21

Documentation for connect_timeout() states:

If None is provided, TcpStream::connect will be used, and it will not time out.

This is not actually true: TCP connection attempts will time out, and this is enforced by the operating system.

Here's how it works on Linux and Mac OS: http://willbryant.net/overriding_the_default_linux_kernel_20_second_tcp_socket_connect_timeout

Windows also enforces a timeout: https://serverfault.com/questions/193160/which-is-the-default-tcp-connect-timeout-in-windows

For reference, all other Rust HTTP client libraries do not set a default connection timeout, while libcurl sets it to 300 seconds by default.

Relative URis such as /path aren't handled correctly. Rather than .path() and .resource() returning /path, they simply return None.

use http_req::uri::Uri;
fn main() {
	// I am expecting `Some("/path")` but receive `None`
	println!("{:?}", "/path".parse::<Uri>().path());
}

Hello, would it make sense to be able to set the RequestBuilder's body via a Request?
Currently, I only see the ability to send a body when using the RequestBuilder.

Perhaps even the ability to implement From<RequestBuilder> for Request would be nice.

Cheers,

Cole

copy_with_timeout() as implemented in v0.7.2 may exceed its deadline:

In here the timeout is only checked after the read operation is issued. However, a call to read() may block according to the documentation. If used on a reader that reads from a network socket, IIRC it will block waiting for data from the remote server until the underlying TCP timeout expires (5 minutes by default).

I believe this is a fundamental limitation of the approach; I am not aware of any ways to fix it without spawning an auxiliary thread or using an async runtime.

If a read timeout is desired, it should be configured on the socket explicitly instead.

If a timeout for the entire request is desired, it can be implemented either by spawning an auxiliary thread that interrupts the main thread (as done in attohttpc), or by setting the read timeout to the minimum of read timeout and the remaining time until the deadline for the entire request (as done in minreq).

Hello @jayjamesjay,
Is it possible for you to make stream public in Conn structure to be able to create a Conn ?

I need to be able to do things like

...
let stream = StreamOwned::new(session, stream);
Ok(Conn { stream })

I make a PR if it's ok for you #62 😃

http_req panics on fetching either of these websites: hit.de, uhu.es

Backtrace:

thread 'main' panicked at 'slice index starts at 18446744073709551614 but ends at 2', src/libcore/slice/mod.rs:2680:5
stack backtrace:
   0: backtrace::backtrace::libunwind::trace
             at /cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.40/src/backtrace/libunwind.rs:88
   1: backtrace::backtrace::trace_unsynchronized
             at /cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.40/src/backtrace/mod.rs:66
   2: std::sys_common::backtrace::_print_fmt
             at src/libstd/sys_common/backtrace.rs:77
   3: <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt
             at src/libstd/sys_common/backtrace.rs:59
   4: core::fmt::write
             at src/libcore/fmt/mod.rs:1057
   5: std::io::Write::write_fmt
             at src/libstd/io/mod.rs:1426
   6: std::sys_common::backtrace::_print
             at src/libstd/sys_common/backtrace.rs:62
   7: std::sys_common::backtrace::print
             at src/libstd/sys_common/backtrace.rs:49
   8: std::panicking::default_hook::{{closure}}
             at src/libstd/panicking.rs:195
   9: std::panicking::default_hook
             at src/libstd/panicking.rs:215
  10: std::panicking::rust_panic_with_hook
             at src/libstd/panicking.rs:472
  11: rust_begin_unwind
             at src/libstd/panicking.rs:376
  12: core::panicking::panic_fmt
             at src/libcore/panicking.rs:84
  13: core::slice::slice_index_order_fail
             at src/libcore/slice/mod.rs:2680
  14: http_req::request::Request::send
  15: http_req_test::main
  16: std::rt::lang_start::{{closure}}
  17: std::rt::lang_start_internal::{{closure}}
             at src/libstd/rt.rs:52
  18: std::panicking::try::do_call
             at src/libstd/panicking.rs:296
  19: __rust_maybe_catch_panic
             at src/libpanic_unwind/lib.rs:79
  20: std::panicking::try
             at src/libstd/panicking.rs:272
  21: std::panic::catch_unwind
             at src/libstd/panic.rs:394
  22: std::rt::lang_start_internal
             at src/libstd/rt.rs:51
  23: std::rt::lang_start
  24: __libc_start_main
  25: _start
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

Tested on http_req version 0.5.4

Environment

• http_req-v0.6.0

Sample Code

use http_req::{request::RequestBuilder, tls, uri::Uri};
use std::net::TcpStream;

fn main() {
    let addr: Uri = "https://www.alphavantage.co/query?function=CURRENCY_EXCHANGE_RATE&from_currency=USD&to_currency=CNY&apikey=QLKN0O595A7Y5344".parse().unwrap();
    let mut writer = Vec::new();

    let stream = TcpStream::connect((addr.host().unwrap(), addr.corr_port())).unwrap();
    let mut stream = tls::Config::default()
        .connect(addr.host().unwrap_or(""), stream)
        .unwrap();

    let response = RequestBuilder::new(&addr)
        .header("Connection", "Close")
        .send(&mut stream, &mut writer)
        .unwrap();

    if !response.status_code().is_success() {
        panic!("invalid status: {:?}", response.status_code());
    }

    //let headers = response.headers();
    //println!("headers: {:?}", headers);

    //println!("writer: {:?}", &writer);
    let response_body = unsafe { String::from_utf8_unchecked(writer) };
    println!("{}", response_body);
}

Expectation

Output should go as

{
    "Realtime Currency Exchange Rate": {
        "1. From_Currency Code": "USD",
        "2. From_Currency Name": "United States Dollar",
        "3. To_Currency Code": "CNY",
        "4. To_Currency Name": "Chinese Yuan",
        "5. Exchange Rate": "7.03200000",
        "6. Last Refreshed": "2020-02-21 11:22:02",
        "7. Time Zone": "UTC",
        "8. Bid Price": "7.03100000",
        "9. Ask Price": "7.03300000"
    }
}

Bug got

1B0
{
    "Realtime Currency Exchange Rate": {
        "1. From_Currency Code": "USD",
        "2. From_Currency Name": "United States Dollar",
        "3. To_Currency Code": "CNY",
        "4. To_Currency Name": "Chinese Yuan",
        "5. Exchange Rate": "7.03200000",
        "6. Last Refreshed": "2020-02-21 11:22:02",
        "7. Time Zone": "UTC",
        "8. Bid Price": "7.03100000",
        "9. Ask Price": "7.03300000"
    }
}
0

Where do the 1B0 prefix and 0 suffix come from?

Really appreciate if someone can help~

Using TryFrom instead of FromStr makes it possible to reduce memory allocation by replacing Strings inside structs with &str.

In some website, it could make stange response.
For example, one of the largest search engines Baidu:

let mut body = Vec::new();
http_req::request::Request::new(&"https://www.baidu.com/".try_into().unwrap())
    // .header("Referer", "") // It will be directed to http://www.baidu.com
    .send(&mut buf)
    .unwrap();
println!("{}", String::from_utf8_lossy(&body));