dgrr / http2 Goto Github PK

I am not sure if the padding methods I am using are fast enough

Recent changes in fashttp broke the integration with http2

See: https://github.com/valyala/fasthttp/pull/1602/files

package main

import (
	"github.com/valyala/fasthttp"
	"github.com/dgrr/http2"
)

func main() {
    s := &fasthttp.Server{
        Handler: yourHandler,
        Name:    "HTTP2 test",
    }

    http2.ConfigureServer(s, http2.ServerConfig{})
    
    s.ListenAndServeTLS(...)
}

Results in:

/go/pkg/mod/github.com/dgrr/[email protected]/configure.go:70:16: cannot use cl.Do (value of type func(req *fasthttp.Request, res *fasthttp.Response) (err error)) as fasthttp.RoundTripper value in assignment: func(req *fasthttp.Request, res *fasthttp.Response) (err error) does not implement fasthttp.RoundTripper (missing method RoundTrip)

Streaming of data packets. Or support the fasthttp streaming feature.
Currently, if the packet is longer than 65535 the packets gets fragmented into multiple data packets without the EndFlag.

I think the mechanism should be similar for longer packets. Just look into it

I'd be nice if routers or framework implementations like gramework or fiber could choose to push some content before being asked to. Think about the possible implementation.

HTTP/2 protocol gives room for cancelling a request. That is because in section 5.1, when a stream is in half-closed state, the client can send a Reset frame to cancel the request and put the stream in the Closed state.

In general: Manage protocol errors/violations.

i tried the example code of client, but the resp.Header.Protocol() returns HTTP/1.1
could you help me use http2 client?

hc := &fasthttp.HostClient{
	Addr: "api.binance.com:443",
}

if err := fasthttp2.ConfigureClient(hc, fasthttp2.ClientOpts{}); err != nil {
	log.Printf("%s doesn't support http/2\n", hc.Addr)
}
req := fasthttp.AcquireRequest()
resp := fasthttp.AcquireResponse()
req.SetRequestURI("https://api.binance.com/api/v3/time")
req.Header.SetMethod("GET")
err := hc.Do(req, resp)
if err != nil {
	log.Fatalln(err)
}
fmt.Println(resp)
fmt.Printf("%d: %s, %s\n", resp.StatusCode(), resp.Body(), resp.Header.Protocol())

HPACK is not working correctly. The tests works, but whenever you try to make more than 1 request with the client to any server you get a violation of the HPACK format.

Why the tests works? Because it should do. The logic looks like correctly implemented. The tests are not biased because they are extracted from the RFC examples (here).

The problem is around here. If the indexing changes, it works sometimes. But the tests doesn't.

WTF

Keep-Alive connections get closed ReadTimeout seconds after connecting, even if they're actively being used to make HTTP requests:

I'm using a ReadTimeout of 5 seconds. As you can see, 5 and 10 seconds after the initial request, the browser has to connect again.

I believe this issue is basically the same as this one: golang/go#16450

Hey, thanks for your effort in making this. In order to help I'm trying to understand what's the state of this project in terms of compliance to the H2 spec. So I'm trying to run h2spec suite against simple server. Unfortunately it hangs during request initialization on this line for client and on this line for server. I might be wrong on this however it looks like that both server and client are waiting on TCP socket for data to come in.
While I was fiddling around I discovered that I can repeat this behaviour if I send a POST request with no data:

curl -v -k -XPOST "https://localhost:8443/"
*   Trying ::1...
* TCP_NODELAY set
* Connection failed
* connect to ::1 port 8443 failed: Connection refused
*   Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 8443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/cert.pem
  CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: O=fasthttp test
*  start date: Feb 17 21:19:20 2021 GMT
*  expire date: Feb 17 21:19:20 2022 GMT
*  issuer: O=fasthttp test
*  SSL certificate verify result: self signed certificate (18), continuing anyway.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x7fc47980ba00)
> POST / HTTP/2
> Host: localhost:8443
> User-Agent: curl/7.64.1
> Accept: */*
>
* Connection state changed (MAX_CONCURRENT_STREAMS == 100)!
^C <= ^^^ it hangs here so I had to kill the process with ctrl+c

For the record I also setup a simple HTTP/2 server from the stdlib for the reference and it mostly conform against h2spec (146 tests, 142 passed, 0 skipped, 4 failed)

Anyways I think h2spec could provide a good target and progress meter for this project. I'll try to figure out how to run it against this project, however if you have any clue where to look or how to fix that it would be great:)

A client can just ignore the GoAway and still sending frames afterwards.

Hope to have a working example using

    "golang.org/x/crypto/acme/autocert"

Thx

The main problem this lib has is that the dispatching of handlers is not performed in different coroutines, thus, the handleStreams coroutine blocks entirely until the handler finishes, which is not desirable.

panic: send on closed channel

goroutine 57799 [running]:
github.com/dgrr/http2.(*Ctx).resolve(0x40f3b6, {0xcc4820, 0xc00008a070})
/root/gopath/pkg/mod/github.com/dgrr/[email protected]/client.go:58 +0x36
github.com/dgrr/http2.(*Conn).writeLoop.func2.1({0x445d4f, 0xc000682f70}, {0xb1f200, 0xc000b03ce0})
/root/gopath/pkg/mod/github.com/dgrr/[email protected]/conn.go:414 +0x3d
sync.(*Map).Range(0xc000682f90, 0xc000682e78)
/root/go/src/sync/map.go:346 +0x2aa
github.com/dgrr/http2.(*Conn).writeLoop.func2()
/root/gopath/pkg/mod/github.com/dgrr/[email protected]/conn.go:412 +0x1bd
github.com/dgrr/http2.(*Conn).writeLoop(0xc001b98b40)
/root/gopath/pkg/mod/github.com/dgrr/[email protected]/conn.go:471 +0x407
created by github.com/dgrr/http2.(*Conn).Handshake
/root/gopath/pkg/mod/github.com/dgrr/[email protected]/conn.go:239 +0x7b

https://github.com/gofiber/fiber/runs/5364379599?check_suite_focus=true#step:4:1068

Do you support server push like http.Pusher? Thanks

Tests everywhere!

writeLoop: use of closed network connections when using firefox.

Section

Looks like there's something called Flow Control. Which, if I understood correctly, allows an HTTP/2 endpoint to allocate resources efficiently, thus giving some feedback of the state to the other endpoint. That means: If I am a proxy server in HTTP/2 and I have 1 client, my window (my read buffer or available memory buffer) might be 104800 bytes. But, while the server keeps running and more peers connects to the server I (the proxy) need to allocate better the resources. Thanks to the WINDOW_UPDATE we can tell and endpoint that our window is, instead, 65535 bytes. So, the endpoint should only be able to send use frames no longer than 65535 bytes.

Or at least that's what I understood...

So far this looks ok, but the problem introduced by the f**king RFC is that every frame can have it's own window and the connection too. I'd like to make a Flow control per server instance, so we can easily balance the window of multiple connections, and then every connCtx should have it's own window, derived from the server, and the same thing for Streams. The Streams should get the window from the connCtx

The client depends a lot on one single connection.

It seems as if using fasthttp2.ConfigureServer(server) does not work, even when the server is serving on a TLS listener. Am I doing something incorrect or is there no support for this?

	listener, err := net.Listen("tcp4", "0.0.0.0:443")
	if err != nil {
		panic(err)
	}

	tlsListener := tls.NewListener(listener, tlsConfig)

	server := &fasthttp.Server {
		Handler: ServeHTTP,
	}

	fasthttp2.ConfigureServer(server)

	go server.Serve(tlsListener)

I noticed that this project is not using go modules for dependency management. Any objections regarding it? If not I can create a PR to enable it

Here

Yes, that's quite a challenge. I don't want to use channels, but how do you keep being thread-safe then?
The main problem is that you can't serialize 2 requests at the same time, because one might modify the table and the other should apply those changes...

I'll need to investigate how nghttp2 does those kind of things.

Hi:
This http2 code not support proxy, right? fasthttp support proxy, but i test this code, it do not support it, yes?

When I try the http2 client example, I have trouble with the Transport method.
My code:

...

func main() {
	c := &fasthttp.HostClient{
		Addr:  "api.binance.com:443",
		IsTLS: true,
	}
	if err := http2.ConfigureClient(c, http2.OptionEnableCompression); err != nil {
		panic(err)
	}

	count := int32(0)
	var wg sync.WaitGroup
	for i := 0; i < 20; i++ {
		for atomic.LoadInt32(&count) >= 4 {
			time.Sleep(time.Millisecond * 100)
		}

		wg.Add(1)
		atomic.AddInt32(&count, 1)
		go func() {
			defer wg.Done()
			defer atomic.AddInt32(&count, -1)

			req := fasthttp.AcquireRequest()
			res := fasthttp.AcquireResponse()

			req.Header.SetMethod("GET")
			// TODO: Use SetRequestURI
			req.URI().Update("https://api.binance.com/api/v3/exchangeInfo")

			err := c.Do(req, res)
			if err != nil {
				log.Fatalln(err)
			}

			body := res.Body()

			fmt.Printf("%d: %d\n", res.Header.StatusCode(), len(body))
			res.Header.VisitAll(func(k, v []byte) {
				fmt.Printf("%s: %s\n", k, v)
			})
			fmt.Println("------------------------")
		}()
	}

	wg.Wait()
}

The result:

$ go run main.go 

# github.com/dgrr/http2
../../../.gvm/pkgsets/go1.16/global/pkg/mod/github.com/dgrr/[email protected]/client.go:201:3: c.Transport undefined (type *fasthttp.HostClient has no field or method Transport)

I see a GO_AWAY with CLOSED_STREAM error sent to client when WINDOS_UPDATE is received right after server sent END_STREAM and closed stream locally. This is seen when using fasthttp2 client which sends WINDOW_UPDATE despite it should already know that server closed stream locally - client has local half-closed stream and just received frame with END_STREAM flag.

RFC says:
Endpoints MUST ignore [WINDOW_UPDATE] or [RST_STREAM] frames received in this state, though endpoints MAY choose to treat frames that arrive a significant time after sending END_STREAM as a connection error ([Section 5.4.1] of type [PROTOCOL_ERROR]

Easy fix is to add FrameWindowUpdate type as allowed frames as in below patch. Ideally, timestamp of close event might be tracked to make decision based on age of closed stream.

diff --git a/serverConn.go b/serverConn.go
index cd0e2ea..3df2c5f 100644
--- a/serverConn.go
+++ b/serverConn.go
@@ -374,7 +374,7 @@ loop:
                                }
 
                                if _, ok := closedStrms[fr.Stream()]; ok {
-                                       if fr.Type() != FramePriority {
+                                       if fr.Type() != FramePriority && fr.Type() != FrameWindowUpdate {
                                                sc.writeGoAway(fr.Stream(), StreamClosedError, "frame on closed stream")
                                        }

Thx,

Ping valyala/fasthttp#972

Section

I am not sure about this one. Implementing this would imply that:

1. We have a coroutine per Stream (that would be ok, but we need to thing about overhead).
2. Only one coroutine reads from the connection and sends the frames read to the particular Stream (if any).
3. Communication using channels between Streams and main coroutine (the coroutine that reads from the conn).
4. Another coroutine to write to the conn. Here we can just have 2 channels, one for reading and another one for writing, but it's ok.

My concern about this is: The code will look so ugly and it will just create more edge cases, more goroutine overhead, etc...
Also, how does priorization and dependency of streams work? WTF, I don't want to have a graph of Streams.

If someone has any idea, you are welcome to comment down here.
The only point I am thinking about I'd be ok to implement is a coroutine per Stream. Streams doesn't live that long... But using a goroutine pool can do the work. Also, it would speed up quite a lot. It's already fast and is currently working synchronously, think async!

Might be better??? I don't know

I have no idea how the settings frame works. Do we need to sync our settings with the other endpoint or what?

I'm getting this panic pretty regularly (despite using recover middleware):

panic: send on closed channel
goroutine 4449 [running]:
github.com/dgrr/http2.(*serverConn).writePing(0xc000253340)
        github.com/dgrr/[email protected]/serverConn.go:148 +0xad
github.com/dgrr/http2.(*serverConn).sendPingAndSchedule(0xc000253340)
        github.com/dgrr/[email protected]/serverConn.go:940 +0x1e
created by time.goFunc
        time/sleep.go:180 +0x31

i would like it to be automatically detect incoming hostname so I dont have to WhiteList it.
any domains at all. how do i do that with this?

https://github.com/dgrr/http2/blob/master/examples/autocert/main.go

i know i can comment out the whitelist host but i dunno how to make it pre-detect/pre-specify the incoming ctx.Request.Host()