aiortc / aioquic Goto Github PK

View Code? Open in Web Editor NEW

1.6K 39.0 224.0 1.86 MB

QUIC and HTTP/3 implementation in Python

License: BSD 3-Clause "New" or "Revised" License

Python 97.12% C 2.88%

python quic asyncio network tls http3

aioquic's Introduction

aioquic

What is `aioquic`?

aioquic is a library for the QUIC network protocol in Python. It features a minimal TLS 1.3 implementation, a QUIC stack and an HTTP/3 stack.

QUIC was standardised in RFC 9000 and HTTP/3 in RFC 9114. aioquic is regularly tested for interoperability against other QUIC implementations.

To learn more about aioquic please read the documentation.

Why should I use `aioquic`?

aioquic has been designed to be embedded into Python client and server libraries wishing to support QUIC and / or HTTP/3. The goal is to provide a common codebase for Python libraries in the hope of avoiding duplicated effort.

Both the QUIC and the HTTP/3 APIs follow the "bring your own I/O" pattern, leaving actual I/O operations to the API user. This approach has a number of advantages including making the code testable and allowing integration with different concurrency models.

Features

minimal TLS 1.3 implementation conforming with RFC 8446
QUIC stack conforming with RFC 9000
- IPv4 and IPv6 support
- connection migration and NAT rebinding
- logging TLS traffic secrets
- logging QUIC events in QLOG format
HTTP/3 stack conforming with RFC 9114
- server push support
- WebSocket bootstrapping conforming with RFC 9220
- datagram support conforming with RFC 9297

Installing

The easiest way to install aioquic is to run:

pip install aioquic

Building from source

If there are no wheels for your system or if you wish to build aioquic from source you will need the OpenSSL development headers.

Linux

On Debian/Ubuntu run:

sudo apt install libssl-dev python3-dev

On Alpine Linux run:

sudo apk add openssl-dev python3-dev bsd-compat-headers libffi-dev

OS X

On OS X run:

brew install openssl

You will need to set some environment variables to link against OpenSSL:

export CFLAGS=-I$(brew --prefix openssl)/include
export LDFLAGS=-L$(brew --prefix openssl)/lib

Windows

On Windows the easiest way to install OpenSSL is to use Chocolatey.

choco install openssl

You will need to set some environment variables to link against OpenSSL:

$Env:INCLUDE = "C:\Progra~1\OpenSSL\include"
$Env:LIB = "C:\Progra~1\OpenSSL\lib"

Running the examples

aioquic comes with a number of examples illustrating various QUIC usecases.

You can browse these examples here: https://github.com/aiortc/aioquic/tree/main/examples

License

aioquic is released under the BSD license.

aioquic's People

Contributors

Stargazers

Watchers

Forkers

mohi7solanki tiran madjonr sysuljx priestd09 pgjones davidwangw blinkerbit rbramwell jlaine shuaizhao danilovazb hansroh tianyaqu alexeydevil mikehwang mbeacom the-rene rmarx keepsimple1 jamiejackherer volahtile yuki7070 moonfalir zverevm souvikghosh05 jvdrhoof ashleemikeyoung rheehot siliconcupcake i-am-justice mounikaprakash shubhayans mattyhsueh jagadeesh432 blandine11 alexfengcisco xgxzj mraavi432 5l1v3r1 lautarojayat qyysjtu aniketh01 ctr-226 prchander zhenruyan chen-yu-wang dnsun jeremybusk johnzhang27 stevennyman leokachun gale006 mishaantonkin mahdiyar-oraei veritaris fereyfang zoharzrihen halflernation ashsrao openwd bbhybh cavmelo maruibin alibaba-jack chaibot97 cs536-modbus-quic ag-python bw1109 jefrasa jalopezsilva rakbladsvalsen vranem1 josey-luan bashi arvind2222 jdoda comsys taizo danfu1020 godka cyberflamego kaz3w anusorn boogieling colleenkimball didimelli superlitong letemps fgtubs arthuredelstein yutakahirano priyansh2001here xirdigh hixio-mh guest271314 saying-yan carrotbeetrice manuelrosenthaler yunmingxiao

aioquic's Issues

Client ConnectionError on FreeBSD

Hi,

I am successfully able to run http3_server.py, but if I try to run the example client I get:

 Traceback (most recent call last):
  Fi "examples/http3_client.py", line 426, in <module>
  int_response=args.print_response,
  Fi "/usr/local/lib/python3.7/asyncio/base_events.py", line 583, in run_until_complete
  turn future.result()
  Fi "examples/http3_client.py", line 317, in run
  ssion_ticket_handler=save_session_ticket,
  Fi "/usr/local/lib/python3.7/contextlib.py", line 170, in __aenter__
  turn await self.gen.__anext__()
  Fi "/usr/home/vagrant/aiortc/aioquic/src/aioquic/asyncio/client.py", line 89, in connect
  ait protocol.wait_connected()
  Fi "/usr/home/vagrant/aiortc/aioquic/src/aioquic/asyncio/protocol.py", line 127, in wait_connected
await asyncio.shield(self._connected_waiter)
ConnectionError

I have tried creating a custom client and connecting to external servers e.g., https://cloudflare-quic.com/and https://quic.aiortc.org/ with both the example and my custom client and I get the same issue. I have checked that the ALPN protocols match from my client and the server and am running out of ideas :/ Any ideas?

P.S.
Before getting the server example running I had to update my openssl but I do not believe that to be the issue as it is now running fine...

Install fails on CentOS7

Here is the output when installing the aioquic project locally on a CentOS7 machine:

# pip3 install -e .                                                                                                                                                                                                
WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead.                                                                                                                            
Obtaining file:///home/aioquic                                                                                                                                                                                                                
Requirement already satisfied: cryptography>=2.5 in /usr/local/lib64/python3.6/site-packages (from aioquic==0.8.3)                                                                                                                            
Requirement already satisfied: pylsqpack<0.4.0,>=0.3.3 in /usr/local/lib64/python3.6/site-packages (from aioquic==0.8.3)                                                                                                                      
Requirement already satisfied: dataclasses in /usr/local/lib/python3.6/site-packages (from aioquic==0.8.3)                                                                                                                                    
Requirement already satisfied: six>=1.4.1 in /usr/local/lib/python3.6/site-packages (from cryptography>=2.5->aioquic==0.8.3)                                                                                                                  
Requirement already satisfied: cffi!=1.11.3,>=1.8 in /usr/local/lib64/python3.6/site-packages (from cryptography>=2.5->aioquic==0.8.3)                                                                                                        
Requirement already satisfied: pycparser in /usr/local/lib/python3.6/site-packages (from cffi!=1.11.3,>=1.8->cryptography>=2.5->aioquic==0.8.3)                                                                                               
Installing collected packages: aioquic                                                                                                                                                                                                        
  Running setup.py develop for aioquic                                                                                                                                                                                                        
    Complete output from command /usr/bin/python3 -c "import setuptools, tokenize;__file__='/home/aioquic/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'ex
ec'))" develop --no-deps:                                                                                                                                                                                                                     
    running develop                                                                                                                                                                                                                           
    running egg_info                                                                                                                                                                                                                          
    writing src/aioquic.egg-info/PKG-INFO                                                                                                                                                                                                     
    writing dependency_links to src/aioquic.egg-info/dependency_links.txt                                                                                                                                                                     
    writing requirements to src/aioquic.egg-info/requires.txt                                                                                                                                                                                 
    writing top-level names to src/aioquic.egg-info/top_level.txt                                                                                                                                                                             
    reading manifest file 'src/aioquic.egg-info/SOURCES.txt'                                                                                                                                                                                  
    reading manifest template 'MANIFEST.in'                                                                                                                                                                                                   
    writing manifest file 'src/aioquic.egg-info/SOURCES.txt'                                                                                                                                                                                  
    running build_ext                                                                                                                                                                                                                         
    building 'aioquic._buffer' extension                                                                                                                                                                                                      
    gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -
fPIC -I/usr/include/python3.6m -c src/aioquic/_buffer.c -o build/temp.linux-x86_64-3.6/src/aioquic/_buffer.o                                                                                                                                  
    creating build/lib.linux-x86_64-3.6                                                                                                                                                                                                       
    creating build/lib.linux-x86_64-3.6/aioquic                                                                                                                                                                                               
    gcc -pthread -shared -Wl,-z,relro -g build/temp.linux-x86_64-3.6/src/aioquic/_buffer.o -L/usr/lib64 -lpython3.6m -o build/lib.linux-x86_64-3.6/aioquic/_buffer.cpython-36m-x86_64-linux-gnu.so                                            
    building 'aioquic._crypto' extension                                                                                                                                                                                                      
    gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -
fPIC -I/usr/include/python3.6m -c src/aioquic/_crypto.c -o build/temp.linux-x86_64-3.6/src/aioquic/_crypto.o                                                                                                                                  
    src/aioquic/_crypto.c: In function 'AEAD_decrypt':                                                                                                                                                                                        
    src/aioquic/_crypto.c:126:5: error: 'for' loop initial declarations are only allowed in C99 mode                                                                                                                                          
         for (int i = 0; i < 8; ++i) {                                                                                                                                                                                                        
         ^                                                                                                                                                                                                                                    
    src/aioquic/_crypto.c:126:5: note: use option -std=c99 or -std=gnu99 to compile your code                                                                                                                                                 
    src/aioquic/_crypto.c: In function 'AEAD_encrypt':                                                                                                                                                                                        
    src/aioquic/_crypto.c:168:5: error: 'for' loop initial declarations are only allowed in C99 mode                                                                                                                                          
         for (int i = 0; i < 8; ++i) {                                                                                                                                                                                                        
         ^                                                                                                                                                                                                                                    
    src/aioquic/_crypto.c: In function 'HeaderProtection_apply':                                                                                                                                                                              
    src/aioquic/_crypto.c:326:5: error: 'for' loop initial declarations are only allowed in C99 mode                                                                                                                                          
         for (int i = 0; i < pn_length; ++i) {                                                                                                                                                                                                
         ^                                                                                                                                                                                                                                    
    src/aioquic/_crypto.c: In function 'HeaderProtection_remove':                                                                                                                                                                             
    src/aioquic/_crypto.c:356:5: error: 'for' loop initial declarations are only allowed in C99 mode                                                                                                                                          
         for (int i = 0; i < pn_length; ++i) {                                                                                                                                                                                                
         ^                                                                                                                                                                                                                                   
    error: command 'gcc' failed with exit status 1                                                                                                                                                                                           
                                                                                                                                                                                                                                              
    ----------------------------------------
Command "/usr/bin/python3 -c "import setuptools, tokenize;__file__='/home/aioquic/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" develop --no-deps
" failed with error code 1 in /home/aioquic/

I was able to get past the error by making the following modification (added the extra_compile_args bit) to the setup.py:

        setuptools.Extension("aioquic._buffer", sources=["src/aioquic/_buffer.c"], extra_compile_args=["-std=c99"]),
        setuptools.Extension(
            "aioquic._crypto", libraries=["crypto"], sources=["src/aioquic/_crypto.c"], extra_compile_args=["-std=c99"]
        ),

investigate performance and optimize

To have a consistent basis for testing, here is how I run my tests.

Launch the server

python examples/server.py -c tests/ssl_cert.pem -k tests/ssl_key.pem

Launch a client

aioquic

Download 50MB of data using aioquic:

python examples/client.py 127.0.0.1 4433 /50000000 > /dev/null

The download throughput will be displayed

picoquic

To only test the server it can be useful to use a different client, for instance picoquic which has good and consistent performance.

picoquicdemo -a hq-20 -v ff000014 127.0.0.1 4433 0:/50000000

Example -- Connect to google

Dear all,
I want to connect to google via QUIC protocol. I've installed aioquic and used the client such bellow:
python3 examples/http3_client.py --ca-certs tests/pycacert.pem https://www.google.com
I got this error:

2019-11-21 13:56:18,956 ERROR quic [838842fd97dd6a51] Could not find a common protocol version
Traceback (most recent call last):
  File "examples/http3_client.py", line 412, in <module>
    print_response=args.print_response,
  File "/usr/lib/python3.6/asyncio/base_events.py", line 484, in run_until_complete
    return future.result()
  File "examples/http3_client.py", line 307, in run
    session_ticket_handler=save_session_ticket,
  File "/home/aghamir/quick-test/aioquic/src/aioquic/asyncio/compat.py", line 17, in __aenter__
    return await self.gen.__anext__()
  File "/home/aghamir/quick-test/aioquic/src/aioquic/asyncio/client.py", line 77, in connect
    await protocol.wait_connected()
  File "/home/aghamir/quick-test/aioquic/src/aioquic/asyncio/protocol.py", line 127, in wait_connected
    await asyncio.shield(self._connected_waiter)
ConnectionError

I am newbie in QUIC protocol and also not sure about Google QUIC URL too. Can you help me into this?

Thanks a lot,

Certificate generation documentation?

Hi,

I am prototyping QUIC for a mobile application, thanks for making aioquic available.

Question: I am getting a certificate error even after hardcoding --host with the example http3 code, I am assuming that's because there is a mismatch between localhost and external server address I am communicating with (I'm not using localhost for development). Are there any documentation resources on how to generate the various certificates needed for that, if that is indeed the issue?

Here's the error:

2020-03-20 11:44:58,924 INFO quic [1dee7d339857a8fa] Connection close code 0x12A, reason hostname 'test.us-east-2.compute.amazonaws.com' doesn't match 'localhost'

Thanks in advance!

May the server send data proactively after the connection established between client and server?

May the server send data proactively after the connection established between client and server?
I want to use the aioquic.asyncio.server and how to let server try to send data to the client proactively?
because server has to send hello message in some protocols, So should I just use the event to manage with or without data from a protocol to send by writer in handle_stream() with a loop or other methods?

I am looking forward to your reply!

Streams should be deleted once they're no longer needed

I've tried writing some code that opened 100 of relatively short outgoing unidirectional streams per second. It worked fine at first, but eventually got to the point where it consumed 100% of CPU and a gigabyte of RAM in a single connection. A quick look at the code suggests that once a stream gets into QuicConnection._streams, it never gets removed, even when all of the data inside the stream has been acknowledged and there's no reason to keep it around.

Split the `cryptography` (and other?) typestubs into independent repo?

Following your comment in the pyca/cryptography issue tracker I was looking at merging your cryptography typestubs with the ones I've been incrementally generating for my own use. It would be handy if the typestubs were in their own git repository so that I could clone it locally and submit PRs against it and so on.

How to receive quic packet?

I use two ubuntu virtual machines(vmware), one set as server and the other as client.Then,I just use the example in the aioquic.
The code of client:
python3 examples/http3_py --ca-certs tests/pycacert.pem https://192.168.164.131:4433/
(PS:192.168.164.131 is the IP of my ens33 interface)

The code of server:
python3 examples/http3_server.py --ceificate tests/ssl_cert.pem --private-key tests/ssl_key.pem

At the same time,i run the "tcpdump -i ens33 port 4433 -w quic.pcap" on my server.
After that,I opened the quic.pcap document with wireshark.Unfortunately,all the packets i have received or sent are UDP packets.

What QPACK implementation should we use?

The initial choices which come to mind:

wrap an existing C implementation of QPACK, as I've attempted in https://github.com/aiortc/pylsqpack/
write a pure Python implementation, which would be more debugable but probably more work

pip install -e . fail on mac.

clang -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 
-Wall -I/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include 
-I/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include
-I/Users/karl/.pyenv/versions/3.8.0/include/python3.8 
-c src/aioquic/_crypto.c 
-o build/temp.macosx-10.14-x86_64-3.8/src/aioquic/_crypto.o -std=c99

    src/aioquic/_crypto.c:4:10: fatal error: 'openssl/err.h' file not found
    #include <openssl/err.h>
             ^~~~~~~~~~~~~~~
    1 error generated.

    error: command 'clang' failed with exit status 1

Openssl is installed on /usr/local/Cellar/[email protected]/1.1.1d by brew.

asyncio API example doesn't work

Hi,
I just tried asyncio API example from https://aioquic.readthedocs.io/en/latest/asyncio.html#asyncio-api and it failed for me.
The script

#!/usr/bin/env python

import asyncio
import sys
from aioquic.asyncio import connect

async def http_client(host, port):
    async with connect(host, port) as connection:
        reader, writer = await connection.create_stream()
        writer.write(b"GET /\r\n")
        writer.write_eof()

        response = await reader.read()
        sys.stdout.buffer.write(response)

asyncio.get_event_loop().run_until_complete(
    http_client("quic.aiortc.org", 4433))

error message

# python client.py
Traceback (most recent call last):
  File "client.py", line 17, in <module>
    http_client("quic.aiortc.org", 4433))
  File "/usr/lib/python3.6/asyncio/base_events.py", line 484, in run_until_complete
    return future.result()
  File "client.py", line 8, in http_client
    async with connect(host, port) as connection:
  File "/root/quic/aioquic/aioquic/asyncio/compat.py", line 17, in __aenter__
    return await self.gen.__anext__()
  File "/root/quic/aioquic/aioquic/asyncio/client.py", line 77, in connect
    await protocol.wait_connected()
  File "/root/quic/aioquic/aioquic/asyncio/protocol.py", line 122, in wait_connected
    await asyncio.shield(self._connected_waiter)
ConnectionError

i use python 3.6.5 on Ubuntu 18.04
I got same result with python 3.7.3 on Mac.

Any idea?

Close connection when idle timeout expires

Thoughts on making a sans-I/O implementation of QUIC

Would you be interested in implementing QUIC without I/O in order to support both synchronous and multiple async implementations?

aioquic performance test by example https_server and https_client

Is there any way to test the throughput by example https_server.py and https_client.py?

ImportError: /usr/local/lib/python3.7/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so: Undefined symbol "OPENSSL_no_config"

Hi,

Trying to run the aioquic H3 server on freeBSD 11.3 I get the error in the title of this issue. Seems like in openssl 1.1+ OPENSSL_no_config was replaced by OPENSSL_init.

Related libraries:
cffi==1.14.0
cryptography==2.9
six==1.14.0
wsproto==0.15.0

OS version:
FreeBSD 11.3-RELEASE

Openssl version:
OpenSSL 1.0.2s-freebsd 28 May 2019

Any ideas/help would be appreciated

Can server create stream?

Hi,
I am using aioquic to develop a software. I want to create a one-way stream from the server to transfer data in the existing session, but I don't seem to find a way to create a stream on the server.
Who can give me a solution?
Thanks.

Implement idle timeout

We need to enforce a local idle timeout.

Building compiled wheels for PyPI (Linux, Mac, Windows)

Just like the same name topic:
biopython/biopython#1296

Maybe this method will help: https://pypi.org/project/cibuildwheel/

Implement congestion control algorithm

To support congestion control we need the following.

Sender side

honour per-stream MAX_STREAM_DATA
honour connection-level MAX_DATA
implement congestion control algorithm

** Receiver side **

enforce per-stream MAX_STREAM_DATA
enforce connection-level MAX_DATA limit
raise MAX_DATA and/or MAX_STREAM_DATA as needed

how to run dos_x example?

Hi, :)

would you be kind to provide examples about how to run dns over QUIC?

Best,
SZ

Send DATA_BLOCKED and STREAM_DATA_BLOCKED?

These two frames only seem useful for debugging purposes, not sure whether we should implement sending them.

Example server & client teardown

When running the given example client and server, it does not appear that the connection teardown is done properly. This can be seen on the server side when running in verbose mode. Several loss detections are triggered and ping probe packets are sent, when the client have actually successfully finished. Any thoughts?

HTTP/3 header validation

I think aioquic should validate any headers it sends are valid (according to HTTP/3), for example you can currently send an empty :path header whereas RFC 7540 (HTTP/2 which I think applies here) states

This pseudo-header field MUST NOT be empty for "http" or "https"
URIs; "http" or "https" URIs that do not contain a path component
MUST include a value of '/'.

Honour preferred_address transport parameter

Servers can indicate a preferred address to be used once the handshake completes. Currently aioquic parses this parameter but does not switch to this address after handshake.

Sending images and other difficulties

Hello,
first of all thank you for your efforts and work. Im very interested in this project and quic (respectively http3) and would like to experiment a little bit, but i difficulties to continue. I have successfully installed the project and have been playing around with the examples. The http3_server is running and the http3_client can successfully communicate with the server.

As far as i understand the examples, the http3_server's response (by default) is determined by the demo.py file and the demo.py file specifies the index.html file to be send. By enabling the print-response option of the http3_client i can see, that the http3_server has send index.html file the http3_client and the content of the index.html file are printed in the terminal.

My difficulties begin here. Firstly, is it possible to open the index.html file, which was send from my http3_server to my http3_client, directly with a browser? For now, i can only see the respone in the terminal. Secondly, is there a possibility to send images together with html files?
My goal is to send webpages containing not only html text but also images.

Greetings,
Attila

Certificate and buffer capacity

Hi,

Thank you very much for developing this!

We are trying to use this as a test server in web-platform-tests, and I have one problem: the server throws with our certificates.

With web-platform.test.pem and web-platform.test.key in https://github.com/web-platform-tests/wpt/tree/master/tools/certs, it throws an BufferWriteError.

handle: <Handle _SelectorDatagramTransport._read_ready()>
Traceback (most recent call last):
  File "/usr/lib/python3.6/asyncio/events.py", line 145, in _run
    self._callback(*self._args)
  File "/usr/lib/python3.6/asyncio/selector_events.py", line 1068, in _read_ready
    self._protocol.datagram_received(data, addr)
  File "/home/yuta/work/aioquic/src/aioquic/asyncio/server.py", line 144, in datagram_received
    protocol.datagram_received(data, addr)
  File "/home/yuta/work/aioquic/src/aioquic/asyncio/protocol.py", line 135, in datagram_received
    self._quic.receive_datagram(cast(bytes, data), addr, now=self._loop.time())
  File "/home/yuta/work/aioquic/src/aioquic/quic/connection.py", line 870, in receive_datagram
    context, plain_payload
  File "/home/yuta/work/aioquic/src/aioquic/quic/connection.py", line 1929, in _payload_received
    frame_handler(context, frame_type, buf)
  File "/home/yuta/work/aioquic/src/aioquic/quic/connection.py", line 1308, in _handle_crypto_frame
    self.tls.handle_message(event.data, self._crypto_buffers)
  File "/home/yuta/work/aioquic/src/aioquic/tls.py", line 1339, in handle_message
    output_buf[Epoch.ONE_RTT],
  File "/home/yuta/work/aioquic/src/aioquic/tls.py", line 1831, in _server_handle_hello
    for x in [self.certificate] + self.certificate_chain
  File "/home/yuta/work/aioquic/src/aioquic/tls.py", line 921, in push_certificate
    buf, 3, partial(push_certificate_entry, buf), certificate.certificates
  File "/home/yuta/work/aioquic/src/aioquic/tls.py", line 461, in push_list
    func(value)
  File "/home/yuta/work/aioquic/src/aioquic/tls.py", line 917, in push_certificate_entry
    push_opaque(buf, 3, entry[0])
  File "/home/yuta/work/aioquic/src/aioquic/tls.py", line 480, in push_opaque
    buf.push_bytes(value)
aioquic._buffer.BufferWriteError: Write out of bounds
2020-04-30 19:49:35,467 INFO root QUIC event: <class 'aioquic.quic.events.ProtocolNegotiated'>

I found this can be workarounded by changing a buffer capacity in aioquic/quic/connection.py.

self._crypto_buffers = {
    tls.Epoch.INITIAL: Buffer(capacity=4096),
#  tls.Epoch.HANDSHAKE: Buffer(capacity=4096),
    tls.Epoch.HANDSHAKE: Buffer(capacity=16384),
    tls.Epoch.ONE_RTT: Buffer(capacity=4096),
}

Is this a known issue? Is there any clean solution?

Thank you!

building custorm Congestion Control policy on aioquic

Is there an example for demostrating how to use aioquic to implement a customized Congestion Control policy like PCC Vivace/BBR/ Hybla? It will be quite useful for college students to learn it.

https://quicwg.org/base-drafts/draft-ietf-quic-recovery.html#name-on-new-congestion-event
QPEP: A QUIC-Based Approach to Encrypted Performance Enhancing Proxies for High-Latency Satellite Broadband:
https://arxiv.org/abs/2002.05091
BBR:
https://github.com/google/bbr
PCC Vivace:
https://www.usenix.org/node/211246
https://github.com/KaiwenZha/PCC-Vivace

HTTP/3 ASGI server?

Is there a plan to implement a HTTP/3 ASGI server as part of this project? (something derives from the http3_server.py in tests/ ?)

client.py not working

Hi,
I'm not able to run the client-script...
When I start the Client, I get an AttributeError: X962
Traceback (most recent call last):
File "examples/client.py", line 106, in <module>
session_ticket_handler=save_session_ticket,
File "/usr/lib/python3.7/asyncio/base_events.py", line 584, in run_until_complete
return future.result()
File "examples/client.py", line 30, in run
async with connect(host, port, **kwargs) as connection:
File "/usr/lib/python3.7/contextlib.py", line 170, in __aenter__
return await self.gen.__anext__()
File "/home/ryan/Seafile/Masterarbeit/QUIC-Projekt-aioquic/aioquic/aioquic/asyncio/client.py", line 84, in connect
protocol.connect(addr, protocol_version)
File "/home/ryan/Seafile/Masterarbeit/QUIC-Projekt-aioquic/aioquic/aioquic/asyncio/protocol.py", line 61, in connect
addr, now=self._loop.time(), protocol_version=protocol_version
File "/home/ryan/Seafile/Masterarbeit/QUIC-Projekt-aioquic/aioquic/aioquic/connection.py", line 446, in connect
self._connect(now=now)
File "/home/ryan/Seafile/Masterarbeit/QUIC-Projekt-aioquic/aioquic/aioquic/connection.py", line 867, in _connect
self.tls.handle_message(b"", self._crypto_buffers)
File "/home/ryan/Seafile/Masterarbeit/QUIC-Projekt-aioquic/aioquic/aioquic/tls.py", line 1077, in handle_message
self._client_send_hello(output_buf[Epoch.INITIAL])
File "/home/ryan/Seafile/Masterarbeit/QUIC-Projekt-aioquic/aioquic/aioquic/tls.py", line 1192, in _client_send_hello
key_share.append(encode_public_key(self._ec_private_key.public_key()))
File "/home/ryan/Seafile/Masterarbeit/QUIC-Projekt-aioquic/aioquic/aioquic/tls.py", line 927, in encode_public_key
public_key.public_bytes(Encoding.X962, PublicFormat.UncompressedPoint),
File "/usr/lib/python3.7/enum.py", line 349, in __getattr__
raise AttributeError(name) from None
AttributeError: X962

After git clone ..., I run pip3 install -e . ... Am I missing something?

Error running examples: unable to get local issuer certificate

Running the examples results in an error related to the certificate found in tests/.

I have verified that the cert and key get loaded in during server startup. Handshake between client and server seems to go well until cert verification.

Any thoughts on how to resolve this issue?

Python version: 3.7.4

Verbose logging for http3_server.py:

python examples/http3_server.py --certificate tests/ssl_cert.pem --private-key tests/ssl_key.pem -v
2019-09-26 11:33:25,103 DEBUG asyncio Using selector: EpollSelector
2019-09-26 11:33:28,039 DEBUG quic [5ee0ffcad1c35b8f] Network path ('::1', 55095, 0, 0) discovered
2019-09-26 11:33:28,040 DEBUG quic [5ee0ffcad1c35b8f] QuicConnectionState.FIRSTFLIGHT -> QuicConnectionState.CONNECTED
2019-09-26 11:33:28,045 DEBUG quic [5ee0ffcad1c35b8f] TLS State.SERVER_EXPECT_CLIENT_HELLO -> State.SERVER_EXPECT_FINISHED
2019-09-26 11:33:28,047 DEBUG quic [5ee0ffcad1c35b8f] Discarding epoch Epoch.INITIAL
2019-09-26 11:33:28,048 DEBUG quic [5ee0ffcad1c35b8f] Network path ('::1', 55095, 0, 0) validated by handshake
2019-09-26 11:33:28,057 DEBUG quic [5ee0ffcad1c35b8f] Discarding epoch Epoch.INITIAL
2019-09-26 11:33:28,057 INFO quic [5ee0ffcad1c35b8f] Connection close code 0x12A, reason unable to get local issuer certificate
2019-09-26 11:33:28,057 DEBUG quic [5ee0ffcad1c35b8f] QuicConnectionState.CONNECTED -> QuicConnectionState.DRAINING
2019-09-26 11:33:28,145 DEBUG quic [5ee0ffcad1c35b8f] Discarding epoch Epoch.INITIAL
2019-09-26 11:33:28,146 DEBUG quic [5ee0ffcad1c35b8f] Discarding epoch Epoch.HANDSHAKE
2019-09-26 11:33:28,146 DEBUG quic [5ee0ffcad1c35b8f] Discarding epoch Epoch.ONE_RTT
2019-09-26 11:33:28,146 DEBUG quic [5ee0ffcad1c35b8f] QuicConnectionState.DRAINING -> QuicConnectionState.TERMINATED

Traceback produced from http3_client.py:

python examples/http3_client.py https://localhost:4433/ 
2019-09-26 11:33:28,056 WARNING quic [5ee0ffcad1c35b8f] Error: 298, reason: unable to get local issuer certificate, frame_type: QuicFrameType.CRYPTO
Traceback (most recent call last):
  File "examples/http3_client.py", line 358, in <module>
    run(configuration=configuration, url=args.url, data=args.data)
  File "/usr/lib64/python3.7/asyncio/base_events.py", line 579, in run_until_complete
    return future.result()
  File "examples/http3_client.py", line 250, in run
    session_ticket_handler=save_session_ticket,
  File "/usr/lib64/python3.7/contextlib.py", line 170, in __aenter__
    return await self.gen.__anext__()
  File "/home/supertuba/repos/aioquic/aioquic/asyncio/client.py", line 77, in connect
    await protocol.wait_connected()
  File "/home/supertuba/repos/aioquic/aioquic/asyncio/protocol.py", line 122, in wait_connected
    await asyncio.shield(self._connected_waiter)
ConnectionError

Unable to send a large amount of text in the http 0.9 example

Hi,

I have modified the server.py in the examples and instead of sending the template, I am reading a file and sending the value to the client. It is working perfectly fine for small amount of text, but as soon as I increase the text to around 10 mb, the transmission is still happening, but I am unable to see anything at the client side.

Can you please assist.

Thanks.

Can you see the details of the quic packet?

I want to see the details of quic's packet.

First, I ran the example with two vm.
http3.Server (ubuntu) http3.client (window10)

The packet I captured with wireshark (3.2) comes out as below.

I would like to see the packets in detail like the address below.
https://twitter.com/Lekensteyn/status/975894492392902656/photo/1

Should I set tls.keylog_file?

Implement retransmissions

Currently aioquic performs no retransmissions, meaning it cannot deal with packet loss.

See:
https://tools.ietf.org/html/draft-ietf-quic-transport-20#section-13.2

Implement retransmissions for INITIAL and HANDSHAKE epochs

Currently retransmissions are implemented for the 1-RTT epoch ("application data") but not for the handshake phases.

Understanding packet header construction

Hi,

I am working on a project, which requires modifying packet headers, especially for the ack packets.
Can you please help me understand the code flow via which I can modify the headers and add custom key values.

Thanks.

Performance impact in the case of variable latency

Hi,

I am currently testing aioquic in a simulation environment, where I have set a maximum bandwidth of 10 Mbps.

In a normal scenario, I am able to achieve a throughput of around 9.2 Mbps, but when I implement variable delay in the packets using traffic control (variable delay 1ms to 10ms , uniform distribution), I observe a severe impact in the performance. (The maximum throughput achieve in this case in around 3 Mbps)

Please assist what might be the problem.

WebSocket Server Error

Hi,
i tried to use the Websockets with the server and client example but the server throws an exception:

future: <Task finished coro=<WebSocketHandler.run_asgi() done, defined at /root/aioquic/examples/http3_server.py:162> exception=AssertionError()>
Traceback (most recent call last):
File "/root/aioquic/examples/http3_server.py", line 166, in run_asgi
await application(self.scope, self.receive, self.send)
File "/usr/local/lib/python3.7/dist-packages/starlette/applications.py", line 102, in call
await self.middleware_stack(scope, receive, send)
File "/usr/local/lib/python3.7/dist-packages/starlette/middleware/errors.py", line 146, in call
await self.app(scope, receive, send)
File "/usr/local/lib/python3.7/dist-packages/starlette/exceptions.py", line 58, in call
await self.app(scope, receive, send)
File "/usr/local/lib/python3.7/dist-packages/starlette/routing.py", line 550, in call
await route.handle(scope, receive, send)
File "/usr/local/lib/python3.7/dist-packages/starlette/routing.py", line 376, in handle
await self.app(scope, receive, send)
File "/usr/local/lib/python3.7/dist-packages/starlette/staticfiles.py", line 86, in call
assert scope["type"] == "http"
AssertionError

My calls to the server and client looks like this:

python3 /root/aioquic/examples/http3_server.py --certificate /root/aioquic/tests/ssl_cert.pem --private-key /root/aioquic/tests/ssl_key.pem -v

python3 /root/aioquic/examples/http3_client.py --ca-certs /root/aioquic/tests/pycacert.pem wss://[1005::5]:4433/ -v

Limit traffic on un-validated network paths

Now that we implement retransmissions we need to ensure we don't send more than 3x the received data until a network path is validated to avoid amplification attacks.

Many test cases failed under Windows

============================= test session starts =============================
platform win32 -- Python 3.8.2, pytest-5.4.1, py-1.8.1, pluggy-0.13.1 -- d:\ProgramData\Python38\python.exe
cachedir: .pytest_cache
rootdir: E:\data\aioquic-0.8.6
plugins: asyncio-0.10.0
collecting ... collected 300 items

tests/test_asyncio.py::HighLevelTest::test_change_connection_id FAILED   [  0%]
tests/test_asyncio.py::HighLevelTest::test_connect_and_serve FAILED      [  0%]
tests/test_asyncio.py::HighLevelTest::test_connect_and_serve_ec_certificate FAILED [  1%]
tests/test_asyncio.py::HighLevelTest::test_connect_and_serve_large FAILED [  1%]
tests/test_asyncio.py::HighLevelTest::test_connect_and_serve_with_packet_loss FAILED [  1%]
tests/test_asyncio.py::HighLevelTest::test_connect_and_serve_with_session_ticket FAILED [  2%]
tests/test_asyncio.py::HighLevelTest::test_connect_and_serve_with_sni FAILED [  2%]
tests/test_asyncio.py::HighLevelTest::test_connect_and_serve_with_stateless_retry FAILED [  2%]
tests/test_asyncio.py::HighLevelTest::test_connect_and_serve_with_stateless_retry_bad PASSED [  3%]
tests/test_asyncio.py::HighLevelTest::test_connect_and_serve_with_stateless_retry_bad_original_connection_id PASSED [  3%]
tests/test_asyncio.py::HighLevelTest::test_connect_and_serve_with_version_negotiation FAILED [  3%]
tests/test_asyncio.py::HighLevelTest::test_connect_and_serve_without_client_configuration PASSED [  4%]
tests/test_asyncio.py::HighLevelTest::test_connect_and_serve_writelines FAILED [  4%]
tests/test_asyncio.py::HighLevelTest::test_connect_timeout PASSED        [  4%]
tests/test_asyncio.py::HighLevelTest::test_connect_timeout_no_wait_connected PASSED [  5%]
tests/test_asyncio.py::HighLevelTest::test_key_update FAILED             [  5%]
tests/test_asyncio.py::HighLevelTest::test_ping FAILED                   [  5%]
tests/test_asyncio.py::HighLevelTest::test_ping_parallel FAILED          [  6%]
tests/test_asyncio.py::HighLevelTest::test_server_receives_garbage PASSED [  6%]
tests/test_asyncio_compat.py::AsyncioCompatTest::test_ok PASSED          [  6%]
tests/test_asyncio_compat.py::AsyncioCompatTest::test_raise_exception PASSED [  7%]
tests/test_asyncio_compat.py::AsyncioCompatTest::test_raise_exception_type PASSED [  7%]
tests/test_buffer.py::BufferTest::test_data_slice PASSED                 [  7%]
tests/test_buffer.py::BufferTest::test_pull_bytes PASSED                 [  8%]
tests/test_buffer.py::BufferTest::test_pull_bytes_negative PASSED        [  8%]
tests/test_buffer.py::BufferTest::test_pull_bytes_truncated PASSED       [  8%]
tests/test_buffer.py::BufferTest::test_pull_bytes_zero PASSED            [  9%]
tests/test_buffer.py::BufferTest::test_pull_uint16 PASSED                [  9%]
tests/test_buffer.py::BufferTest::test_pull_uint16_truncated PASSED      [  9%]
tests/test_buffer.py::BufferTest::test_pull_uint32 PASSED                [ 10%]
tests/test_buffer.py::BufferTest::test_pull_uint32_truncated PASSED      [ 10%]
tests/test_buffer.py::BufferTest::test_pull_uint64 PASSED                [ 10%]
tests/test_buffer.py::BufferTest::test_pull_uint64_truncated PASSED      [ 11%]
tests/test_buffer.py::BufferTest::test_pull_uint8 PASSED                 [ 11%]
tests/test_buffer.py::BufferTest::test_pull_uint8_truncated PASSED       [ 11%]
tests/test_buffer.py::BufferTest::test_push_bytes PASSED                 [ 12%]
tests/test_buffer.py::BufferTest::test_push_bytes_truncated PASSED       [ 12%]
tests/test_buffer.py::BufferTest::test_push_bytes_zero PASSED            [ 12%]
tests/test_buffer.py::BufferTest::test_push_uint16 PASSED                [ 13%]
tests/test_buffer.py::BufferTest::test_push_uint32 PASSED                [ 13%]
tests/test_buffer.py::BufferTest::test_push_uint64 PASSED                [ 13%]
tests/test_buffer.py::BufferTest::test_push_uint8 PASSED                 [ 14%]
tests/test_buffer.py::BufferTest::test_seek PASSED                       [ 14%]
tests/test_buffer.py::UintVarTest::test_pull_uint_var_truncated PASSED   [ 14%]
tests/test_buffer.py::UintVarTest::test_push_uint_var_too_big PASSED     [ 15%]
tests/test_buffer.py::UintVarTest::test_size_uint_var PASSED             [ 15%]
tests/test_buffer.py::UintVarTest::test_uint_var PASSED                  [ 15%]

Why didn't I see QUIC packets when I was using wireshark

I am using the examples of aioquic respectively on the client and server.However,when i was using wireshark,I found that no quic packets can i receive.What i recieve are just UDP packets.However,I can get the info that shows Congratulations, you loaded this page using HTTP/3!".Could you please tell me how this happens?

localhost address

It seems that by default localhost would be resolved into ipv6 address only (MacOS):

aioquic/src/aioquic/asyncio/client.py

Line 62 in 229c0f1

addr = infos[0][4]

infos[0] is ipv6 info (not dual-stack). Is this intended?

This is ok when running the server from this repo as it's binding to ipv6. But it will not work (by default) when connecting to a server that runs on "127.0.0.1:4433" or "0.0.0.0:4433".

Issues connecting via Cronet -> demo:app

Hi!

Thanks again for your work on this project. I've been setting up a test environment here, to experiment with Android + Cronet HTTP/3 libs for a mobile app.

I am simply using http3_server.py with the demo:app ASGI module, but cannot get Cronet to even negotiate a connection. Here is the error:

Exception in CronetUrlRequest: net::ERR_QUIC_PROTOCOL_ERROR, ErrorCode=10, InternalErrorCode=-356, Retryable=false, QuicDetailedErrorCode=25

Here's the relevant Cronet code:

private void callCronet() {
        CronetEngine.Builder myBuilder = new CronetEngine.Builder(this);
        cronetEngine = myBuilder
                .addQuicHint("quic.example.com", 8443, 8443)
                .enableHttpCache(CronetEngine.Builder.HTTP_CACHE_IN_MEMORY, 100 * 1024)
                .enableHttp2(true)
                .enableQuic(true)
                .build();

        Executor executor = Executors.newSingleThreadExecutor();
        UrlRequest.Callback callback = new MyUrlRequestCallback();
        UrlRequest.Builder builder = cronetEngine.newUrlRequestBuilder("https://quic.example.com:8443", callback, executor);
        UrlRequest urlRequest = builder.build();
        urlRequest.getStatus(new UrlRequest.StatusListener() {
            @Override
            public void onStatus(int status) {
                Log.e(TAG, status + " ");
            }
        });
        urlRequest.start();
    }

    private void callQuicCronet() {
        CronetEngine.Builder myBuilder = new CronetEngine.Builder(this);
        cronetEngine = myBuilder
                .enableHttpCache(CronetEngine.Builder.HTTP_CACHE_IN_MEMORY, 100 * 1024)
                .enableHttp2(true)
                .enableQuic(true)
                .addQuicHint("quic.tech", 8443, 8443)
                .build();

        Executor executor = Executors.newSingleThreadExecutor();
        UrlRequest.Callback callback = new MyUrlRequestCallback();
        UrlRequest.Builder builder = cronetEngine.newUrlRequestBuilder("https://quic.tech:8443", callback, executor);
        UrlRequest urlRequest = builder.build();
        urlRequest.getStatus(new UrlRequest.StatusListener() {
            @Override
            public void onStatus(int status) {
                Log.e(TAG, status + " ");
            }
        });
        urlRequest.start();
    }

We are able to connect without issue to the public QUIC endpoints above; for this local setup we have signed Let's Encrypt certs and are using the following command line to serve a one line index.html from examples/htdocs:

python3 examples/http3_server.py --certificate certs/fullchain1.pem --private-key certs/privkey1.pem --host quic.example.com --port 8443 -v -q quiclog "demo:app"

Any ideas?

Thanks in advance!

aioquic 0.8.6 build error under Win10/Python3.8

building 'aioquic._crypto' extension
C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\bin\HostX86\x64\cl.exe /c /nologo /Ox /W3 /GL /DNDEBUG /MD -IC:\Python38\include -IC:\Python38\include "-IC:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\ATLMFC\include" "-IC:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include" "-IC:\Program Files (x86)\Windows Kits\NETFXSDK\4.6.1\include\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\ucrt" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\shared" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\winrt" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\cppwinrt" /Tcsrc/aioquic/_crypto.c /Fobuild\temp.win-amd64-3.8\Release\src/aioquic/_crypto.obj
_crypto.c
src/aioquic/_crypto.c(93): warning C4244: “函数”: 从“Py_ssize_t”转换到“int”，可能丢失数据
src/aioquic/_crypto.c(96): warning C4244: “函数”: 从“Py_ssize_t”转换到“int”，可能丢失数据
src/aioquic/_crypto.c(130): error C2036: “void *”: 未知的大小
src/aioquic/_crypto.c(136): warning C4244: “函数”: 从“Py_ssize_t”转换到“int”，可能丢失数据
src/aioquic/_crypto.c(139): warning C4244: “函数”: 从“Py_ssize_t”转换到“int”，可能丢失数据
src/aioquic/_crypto.c(175): warning C4244: “函数”: 从“Py_ssize_t”转换到“int”，可能丢失数据
src/aioquic/_crypto.c(178): warning C4244: “函数”: 从“Py_ssize_t”转换到“int”，可能丢失数据
src/aioquic/_crypto.c(275): warning C4244: “函数”: 从“Py_ssize_t”转换到“int”，可能丢失数据
src/aioquic/_crypto.c(312): warning C4244: “初始化”: 从“Py_ssize_t”转换到“int”，可能丢失数据
error: command 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\Professional\\VC\\Tools\\MSVC\\14.24.28314\\bin\\HostX86\\x64\\cl.exe' failed with exit status 2

What is the protocol reference?

Hi, I'm discovering this nice project. I have hoping to see a reference for the protocol. I found https://tools.ietf.org/html/draft-tsvwg-quic-protocol-02 but I think this is not the same (for example, I don't think the "public flags" described there are the same as first_byte read in pull_quic_header).
Therefore, could you please make explicit what reference is followed?

loss detection not triggering

Hi,

I am currently using aioquic in a mobility simulation environment. There is a no connectivity zone for about 5 seconds, during which data transfer is still in place via aioquic.

In an ideal case, I am expecting packet losses when there is no connectivity for a few seconds during an ongoing transmission, but I am only getting sending probe messages and nothing else.
Please help with this issue.

Thanks.

Unable to install in Ubuntu Linux (open error, 'encoding is an invalid keyword argument for this function').

I'm using Ubuntu Linux under (1) Windows Subsystem for Linux (WSL) on Windows 10 and (2) ChromeOS, and am seeing this error in both places. Please note that this traceback is from "pip install aioquic", so installing from files.pythonhosted.org, but when I clone the repository and "pip install -e .", I get what looks, to me, to be the same error.

Perhaps I'm missing something obvious?

spencer@sdawkins-NB0:/mnt/c/Users/sdawkins/Documents/Github/aioquic$ pip install aioquic Collecting aioquic Using cached https://files.pythonhosted.org/packages/cb/de/e88c9ac83ec3f9caac691876fdc21233e9686887343b7bb815720efc0812/aioquic-0.8.7.tar.gz Complete output from command python setup.py egg_info: Traceback (most recent call last): File "", line 1, in File "/tmp/pip-build-zgfOZ9/aioquic/setup.py", line 10, in with open(about_file, encoding="utf-8") as fp: TypeError: 'encoding' is an invalid keyword argument for this function ---------------------------------------- Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-zgfOZ9/aioquic/

NAT traversal

Is there a recommended way of using this lib in the context of NAT traversal (UDP)? For example, if I want to mix in a signaling protocol (or some keep-alive packets) with HTTP/3 QUIC and wanted to go through NAT, is there an existing way in this lib?

or, can we run one instance being both HTTP/3 server and a normal UDP client/endpoint (via the same port)? Thanks.

Validate server TLS certificate

We will still need an option to disable this for testing.

Add support for NAT rebinding

Currently aioquic has no explicit support for rebinding, i.e. dealing with changes in the peer's (IP address, port) tuple.

The specs mention that we should perform path validation if we detect a network path change:

https://tools.ietf.org/html/draft-ietf-quic-transport-20#section-9

Open questions:

How should this interact with asyncio's transport / protocol scheme? Currently the client example creates a "connected" UDP socket which means we never pass the "addr" argument to sendto(). Likewise in the server example we store the peer's initial address
What does this mean if QUIC is layered on top of ICE? Who makes the decisions as to which network path to use: ICE or QUIC?
What is the behaviour while the new network path is being validated? Do we continue using the previous network path until path validation completes?

aiortc / aioquic Goto Github PK

aioquic's Introduction

aioquic

What is aioquic?

Why should I use aioquic?

Features

Installing

Building from source

Linux

OS X

Windows

Running the examples

License

aioquic's People

Contributors

Stargazers

Watchers

Forkers

aioquic's Issues

Launch the server

Launch a client

aioquic

picoquic

Recommend Projects

Recommend Topics

Recommend Org

What is `aioquic`?

Why should I use `aioquic`?