The `http_client_sync_ssl` example doesn't seem to work with Python's basic HTTPS server about beast HOT 13 CLOSED

@ashtum we should document this, where the Table of Contents entry for the paragraph says "stream_truncated error" so people can find it easily

from beast.

This is a known issue with some of the web servers in the wild; they don't perform a proper SSL shutdown procedure and close the TCP socket unexpectedly. For example, if you try connecting to google.com, you would get that error, but not when connecting to github.com.

By the way, because this error happens exactly when you want to close the SSL stream, you can safely ignore this error (as all internet browsers do):

beast::error_code ec;
stream.shutdown(ec);
if (ec == net::error::eof || ec == net::ssl::error::stream_truncated)
    ec = {};

from beast.

Thanks for the help!

I'm still struggling to figure out how to modify the example code to get it to print the "Hello" response from the server:

In the client I tried replacing:

http::read(stream, buffer, res);

with:

boost::beast::error_code myec;
http::read(stream, buffer, res, myec);
if(myec && myec != ssl::error::stream_truncated)
    throw beast::system_error{myec};

but this doesn't seem to do the trick. I still only get to see the HTTP/1.1 200 OK part, not the Hello

ee@vp:~/sand/beasttest % g++ main.cpp -lssl -lcrypto && ./a.out localhost 51733 /                                                                
HTTP/1.1 200 OK

Error: stream truncated
ee@vp:~/sand/beasttest % wget --no-check-certificate -q -O - https://localhost:51733                                                             
Hello

from beast.

@EelisVP In client mode, http::read(stream, buffer, res, myec) should complete successfully. The net::ssl::error::stream_truncated error only occurs during the call to stream.shutdown(ec).

What type of response body are you using? Could you please provide a minimal reproducible example of your issue?
I recommend first trying the http_client_sync_ssl.cpp example without any modifications to see what it prints.

from beast.

@ashtum As described in the opening post, the issue can be reproduced by taking http_client_sync_ssl.cpp and only changing ssl::verify_peer to ssl::verify_none (because the server's certificate was self-signed). With only this modification, I get this result:

ee@vp:~/sand/beasttest % g++ -I /home/eelis/apps/sources/boost_1_85_0 main.cpp -lssl -lcrypto && ./a.out localhost 51733 / 
Error: stream truncated [asio.ssl.stream:1]

from beast.

It seems you successfully get a response from the server, but it is just empty. Note that the target in the wget --no-check-certificate -q -O - https://localhost:51733 command is empty, while in ./a.out localhost 51733 /, the target is "/".
What happens if you try ./a.out localhost 51733 ""?

from beast.

Note that the target in the wget --no-check-certificate -q -O - https://localhost:51733 command is empty, while in ./a.out localhost 51733 /, the target is "/".

Nah, wget asks for / automatically if you don't provide a target. So wgetting https://localhost:51733 is the same as wgetting https://localhost:51733/.

What happens if you try ./a.out localhost 51733 ""?

Same result (because the server doesn't look at the target at all).

from beast.

Does http::read(stream, buffer, res, myec) complete without any errors, or are you getting net::ssl::error::stream_truncated on that line? Could you try setting req.keep_alive(true);?

from beast.

Does http::read(stream, buffer, res, myec) complete without any errors, or are you getting net::ssl::error::stream_truncated on that line?

If I use:

boost::beast::error_code myec;
http::read(stream, buffer, res, myec);
if(myec) std::cout << "oh there was an error: " << myec << std::endl;

then I do see oh there was an error: asio.ssl.stream:1 at runtime, so it's really the read that's reporting the error.

Could you try setting req.keep_alive(true);?

Sure. No effect.

from beast.

It seems we need to manually set the end of stream when parser completes with ssl::error::stream_truncated error, this should work:

http::response_parser<http::dynamic_body> p;
beast::error_code ec;
http::read(stream, buffer, p, ec);

if(ec == net::ssl::error::stream_truncated && p.need_eof())
    p.put_eof(ec);

std::cout << p.get() << std::endl;

from beast.

OK, the reason the Beast parser needs a manual EOF is that your Python server doesn't set the Content-Length header nor does it perform a proper SSL shutdown procedure to indicate the EOF.

If you change the Python code for the server to the following, it should work fine:

class SimpleHTTPRequestHandler(BaseHTTPRequestHandler):
    def do_GET(self):
        message = b'hello'
        self.send_response(200)
        self.send_header("Content-type", "text/plain")
        self.send_header('Content-Length', str(len(message)))
        self.end_headers()
        self.wfile.write(message)

from beast.

@ashtum Ah fantastic, with that change in the client, I do indeed get the hello! Thanks!

from beast.

Yes, the only way to get it to work in Beast is to manually set the EOF on the parser.

The reason this is not the default behavior is that it poses a security risk. Without a Content-Length header, the body length is determined by EOF, which allows an attacker to close the connection and truncate the body contents. Therefore, an HTTPS server should either perform a proper TLS shutdown or avoid relying on EOF by setting the Content-Length header.

If you try using curl on your server, you will see it also completes with an error.

from beast.