algesten / hreq Goto Github PK

On some websites, e.g. http://netclusive.de, hreq panics with the following message:

thread 'main' panicked at 'Incoming body is not valid utf-8: FromUtf8Error { bytes: [ /* byte dump omitted */ ], error: Utf8Error { valid_up_to: 9701, error_len: Some(1) } }', /home/shnatsel/.cargo/registry/src/github.com-1ecc6299db9ec823/hreq-0.7.1/src/body.rs:676:35

25718 websites out of the top million from Feb 3 Tranco list are affected.

Tested using this code. Test tool output for all occurrences is too large to post here, but here's a list of domains where it happened: hreq-utf8-panic-domain-list.txt.gz

Perhaps more in line with the idea of having a small async HTTP client. Doesn't depend on mio, implementation is significantly simpler than tokio and async-std, and only contains safe Rust.

• https://github.com/stjepang/smol
• Example HTTP+TLS client

When downloading http://auctionzip.com, hreq panics with the following message:

thread 'main' panicked at 'dangling store key for stream_id=StreamId(1)', /home/shnatsel/.cargo/registry/src/github.com-1ecc6299db9ec823/hreq-h2-0.2.7/src/proto/streams/store.rs:179:17

Only one websites out of the top million according to Feb 3 Tranco list is affected.

Tested using this code. Test tool output for the affected website: hreq-dangling-store-key.tar.gz

error: reached the type-length limit while instantiating `<std::boxed::Box<std::future::fr..., ()}]>>, ()}]>, ()}]>>>>>::into`
   --> /Users/martin/.rustup/toolchains/stable-x86_64-apple-darwin/lib/rustlib/src/rust/src/libcore/convert/mod.rs:559:5
    |
559 | /     fn into(self) -> U {
560 | |         U::from(self)
561 | |     }
    | |_____^
    |
    = note: consider adding a `#![type_length_limit="1527629"]` attribute to your crate

The problem seems to be due to using tracing::instrument of async blocks.

Related:

rust-lang/rust#75992
rust-lang/rust#54540 (comment)
tokio-rs/tracing#616

I've made a version of h2 that is "tokio free". Then I proceeded to analyze my dependencies in depth. I specifically am hunting dependencies that are pulled in by one single crate and nothing else. Some crates like h2 and async-trait, I'm accepting and not listing.

async-comp              adler32 v1.0.4
async-comp              crc32fast v1.2.0
async-comp, cookie      libc v0.2.70
async-comp, fut-util    memchr v2.3.3
cookie                  proc-macro-hack v0.5.15
cookie                  rustversion v1.0.2
cookie                  standback v0.2.8
cookie                  time v0.2.9
cookie                  time-macros v0.1.0
cookie                  time-macros-impl v0.1.1
fut-util                pin-project v0.4.17
fut-util                pin-project-internal v0.4.17
fut-util                pin-utils v0.1.0
fut-util                proc-macro-nested v0.1.4
futures-util            futures-macro v0.3.5
futures-util            futures-task v0.3.5
public-suff             error-chain v0.12.2
public-suff             idna v0.2.0
public-suff             lazy_static v1.4.0
public-suff             matches v0.1.8
public-suff             regex v1.3.7
public-suff             regex-syntax v0.6.17
public-suff             smallvec v1.4.0
public-suff             unicode-bidi v0.3.4
public-suff             unicode-normalization v0.1.12
public-suff             version_check v0.9.1
serde_json              ryu v1.0.4

In terms of figuring out alternatives the contenders are:

1. public-suffix
2. async-compression
3. cookie
4. futures_util

On some websites, e.g. http://fivestarflags.com, hreq fails with the following error:

Unexpected char in chunk size: ' '

Firefox and curl work fine.

143 websites out of the top million from Feb 3 Tranco list are affected.

Tested using this code and this Cargo.lock. Test tool output for all occurrences: hreq-unexpected-char-in-chunk-size.tar.gz

On some websites, e.g. http://venturesgo.com, hreq fails with the following error:

Other h2 error (poll_data): protocol error: flow-control protocol violated

Firefox, curl and reqwest (with rustls) work fine.

1458 websites out of the top million from Feb 3 Tranco list are affected.

Tested using this code and this Cargo.lock. Test tool output for all occurrences: hreq-control-flow-violated.tar.gz

When downloading http://guaranteedrate.com, hreq panics with the following message:

guaranteedrate.com:thread 'main' panicked at 'year must be in the range -100000..=100000', /home/shnatsel/.cargo/registry/src/github.com-1ecc6299db9ec823/time-0.2.25/src/date.rs:711:25

Three websites out of the top million according to Feb 3 Tranco list are affected.

Tested using this code. Test tool output for all affected websites: hreq-year-out-of-range.tar.gz

On some websites, e.g. http://canon.at, hreq fails with the following error:

CloseNotify alert received

Firefox, curl and reqwest (with rustls) work fine.

7480 websites out of the top million from Feb 3 Tranco list are affected.

Tested using this code and this Cargo.lock. Test tool output for all occurrences: hreq-closenotify-alert.tar.gz

On some websites, e.g. http://zzzebra.de, hreq panics with the following message:

'assertion failed: sz <= self.window_size', /home/shnatsel/.cargo/registry/src/github.com-1ecc6299db9ec823/hreq-h2-0.2.7/src/proto/streams/flow_control.rs:176:9

2130 websites out of the top million from Feb 3 Tranco list are affected.

Tested using this code. Test tool output for all occurrences: hreq-window-size-panic.tar.gz

Dependabot couldn't fetch one or more of your project's path-based Rust dependencies. The affected dependencies were ../hreq-h1/Cargo.toml.

To use path-based dependencies with Dependabot the paths must be relative and resolve to a directory in this project's source code.

View the update logs.

On some websites, e.g. http://economicinclusion.gov, hreq fails with the following error:

Too many chars in chunk size

Firefox and curl work fine.

39 websites out of the top million from Feb 3 Tranco list are affected.

Tested using this code and this Cargo.lock. Test tool output for all occurrences: hreq-too-many-chars-in-chunk-size.tar.gz

On some websites, e.g. http://athenahealth.com, hreq fails with the following error:

EOF before complete http11 header

Firefox and curl work fine. Reqwest also fails.

292 websites out of the top million from Feb 3 Tranco list are affected.

Tested using this code and this Cargo.lock. Test tool output for all occurrences: hreq-eof-before-header.tar.gz

On some websites, e.g. http://italotreno.it, hreq fails with the following error:

broken pipe

Firefox, curl and reqwest (with rustls) work fine.

3080 websites out of the top million from Feb 3 Tranco list are affected.

Tested using this code and this Cargo.lock. Test tool output for all occurrences (not all of them are reproducible now): hreq-broken-pipe.tar.gz

On some websites, e.g. http://emland.net, hreq fails with the following error:

proto: Failed to parse 'index.html' relative to: index.html

The exact path can be other than index.html, but the two paths seem to always be the same.

Firefox, curl and reqwest (with rustls) work fine.

1563 websites out of the top million from Feb 3 Tranco list are affected.

Tested using this code and this Cargo.lock. Test tool output for all occurrences: hreq-failed-to-parse-relative-to.tar.gz

Running strace -f cargo test tls_req_body100mb_with_size:

We find this:

[pid  4217] epoll_wait(3, [], 1024, 0)  = 0
[pid  4217] sendto(7, "\27\3\3@\21\233\276W7\230fU*t\203\345\340\265a\201\332\341\334|j\27I\273\277\2429\\"..., 16406, MSG_NOSIGNAL, NULL, 0) = 16406
[pid  4217] sendto(7, "\27\3\3\0\32\222\231\26\"\203\203\311z'\1j\v\202\376\35\377\3\6,\nER?\177\273\37", 31, MSG_NOSIGNAL, NULL, 0) = 31
[pid  4217] sendto(7, "\27\3\3@\21\3f\277\312\316\234\226\264\322\364\215\303]\215\22\2434pb\177\17W2W\216vN"..., 16406, MSG_NOSIGNAL, NULL, 0) = 16406
[pid  4217] sendto(7, "\27\3\3\0\32\245\"/\375#\300\265\312\364\352\26\201S\274\341NZ\371\364o\274\v!wx\252", 31, MSG_NOSIGNAL, NULL, 0) = 31
[pid  4217] sendto(7, "\27\3\3@\21\2374\n\205^Z\262\3224H\334\275L2?\372j\21\230\23\340c\300V\230.\206"..., 16406, MSG_NOSIGNAL, NULL, 0) = 16406
[pid  4217] sendto(7, "\27\3\3\0\32\342\16\346ve\252\16\251\"\212\325\357\200\264O\34\377\346\374F\370U\0250\365s", 31, MSG_NOSIGNAL, NULL, 0) = 31
[pid  4217] sendto(7, "\27\3\3@\21\256\310\355\345C4`\6L\363u4m\216\203\343kE\350\364U\325\5BM's"..., 16406, MSG_NOSIGNAL, NULL, 0) = 16406
[pid  4217] sendto(7, "\27\3\3\0\31\302|\362\330\216\23}\302g\342\261\204\270z\376Qx\7t\357\260q\231\0\"", 30, MSG_NOSIGNAL, NULL, 0) = 30
[pid  4217] epoll_wait(3, [{EPOLLIN|EPOLLOUT, {u32=2, u64=2}}], 1024, 0) = 1
[pid  4217] recvfrom(8, "\27\3\3@\21\233\276W7\230fU*t\203\345\340\265a\201\332\341\334|j\27I\273\277\2429\\"..., 8192, 0, NULL, NULL) = 8192
[pid  4217] recvfrom(8, "N\241\206\35\30?\360D5\365\336\3307qG\356\21\2752\1o\355\325_g\372\251\203\231\232\244\5"..., 8192, 0, NULL, NULL) = 8192
[pid  4217] recvfrom(8, "\212\235\247\2620V\26787\16\333c\241<[\25\250\0057b\334\252\27\3\3\0\32\222\231\26\"\203"..., 8192, 0, NULL, NULL) = 8192
[pid  4217] recvfrom(8, "\353\272b\325\371\226\251%\302\363w\31X\377V\247\342\24*\332\7>\234\25\362\305\357C\336MZL"..., 8192, 0, NULL, NULL) = 8192
[pid  4217] recvfrom(8, "\375\205C3U\315n\267\22\10\255\\\225\34\370W\321\374*\245v\371\340\211%)\23e\232\366j\374"..., 8192, 0, NULL, NULL) = 8192
[pid  4217] recvfrom(8, "\375\206Q\263\346h\304d\360\220)\221\272\351\202\360\371\359u\323\30\4@q\336\277\311\232\353\231\374"..., 8192, 0, NULL, NULL) = 8192
[pid  4217] recvfrom(8, "\271\35\363\272u\236j\250Y\357\177\215\326\307b\211\353&\347lXKZ\332\223\2Mg\311l\256\21"..., 8192, 0, NULL, NULL) = 8192
[pid  4217] recvfrom(8, "[E\356z^\263m1\24\322-Xq\221\231/a\234sa<s\344\377|x2\312\375\251\5\245"..., 8192, 0, NULL, NULL) = 8192
[pid  4217] recvfrom(8, "\357}\227\v\v&A\270\322\317\6\335\1\232\300Q\236\27]L2\203\257\r!\6\311\3470\315\224\214"..., 8192, 0, NULL, NULL) = 211
[pid  4217] recvfrom(8, 0x7ffd1d846798, 8192, 0, NULL, NULL) = -1 EAGAIN (Resource temporarily unavailable)
[pid  4217] sendto(8, "\27\3\3\0+5By\325!&\364q\230\24\4J\216\260\26w\250\17\305w\350m1\272\\~\v"..., 48, MSG_NOSIGNAL, NULL, 0) = 48

Putting println in src/tls.rs shows that every frame is followed by a flush, which means even if we buffered the write, we'd still hit the kernel on every 16kb frame.

It seems h2 does not automatically grow the window size. In hyper, there's an external BDP implementation that changes the window size on the connection.

https://github.com/hyperium/hyper/blob/master/src/proto/h2/ping.rs
https://github.com/hyperium/hyper/blob/master/src/proto/h2/client.rs#L117
https://github.com/hyperium/hyper/blob/master/src/proto/h2/server.rs#L300

On some websites, e.g. http://festival-piano.com, hreq fails with the following error:

http api: invalid format

Firefox and curl work fine.

1554 websites out of the top million from Feb 3 Tranco list are affected.

Tested using this code and this Cargo.lock. Test tool output for all occurrences: hreq-invalid-format.tar.gz

Since the release of tokio 1.0, I haven't been able to make an update to hreq-h2, which is a tokio free fork of h2. Given that I don't have time, I either have the choice of just abandoning this project, or do what I can to keep it going with minimal effort.

One big problem for hreq is being runtime agnostic. It takes considerable effort to stay on top of dependencies and maintaining a fork of h2.

Rust will eventually have some common way of making runtime agnostic code, but that time has not yet come. This project will be tokio only, until we have such support in rust.