Git Product home page Git Product logo

httpc's Issues

User http headers

Hi, i would like to use this library to query data from influxDB.
To do this i need to add certain headers, but i don't see way to add them in this library.
Am I missing something or I need to modify this library?

Add context pointer to httpc_options

Would it be possible to add a void *context to the options struct? That would make it easier to use in C++ when you need to access class members. Example:

// Static method to enable calls from C.
int HttpClient::HttpOpen(void **socket, struct httpc_options *os, void *opts, const char *domain, unsigned short port, int use_ssl)
{
    HttpClient * self = reinterpret_cast<HttpClient *>(os->context);
    auto res = self->socketFactory.CreateSocket();
    ...
}

How to use re-use properly?

(sorry for the issue spam)

Do you have an example of how to do socket reuse properly? When I try it the first request seems to work fine:

info:1094 Program: Embeddable HTTP 1.0/1.1 client
info:1095 Version: 0.0.0
info:1096 Repo:    https://github.com/howerj/httpc
info:1097 Author:  Richard James Howe
info:1098 Email:   [email protected]
info:1099 Options: stk=128 tst=1 grw=1 log=1 cons=3 redirs=3 hmax=8192 sz=592
info:1103 License: The Unlicense (public domain)
info:521 domain:    212.183.159.230
info:522 port:      80
info:523 SSL:       false
info:526 path       /5MB.zip
debug:1312 state -- initial   -> open     
debug:1312 state -- open      -> send-head
debug:634 custom header 'Range: bytes=0-1023' added
info:642 GET  request complete
debug:1312 state -- send-head -> send-body
info:1035 no callback - nothing to do
debug:1312 state -- send-body -> recv-head
info:906 HEADER: HTTP/1.1 206 Partial Content/28
info:795 unknown field: Date: Fri, 22 Mar 2024 08:55:05 GMT
info:795 unknown field: Server: Apache
info:795 unknown field: Last-Modified: Mon, 02 Jun 2008 15:30:42 GMT
info:795 unknown field: ETag: "603fc-500000-44eb0adaf4c80"
info:739 Accept-Ranges: bytes
info:773 Content Length: 1024
info:795 unknown field: Cache-Control: no-tranform
info:795 unknown field: Content-Range: bytes 0-1023/5242880
info:795 unknown field: Keep-Alive: timeout=15, max=100
info:766 connection may be kept alive
info:795 unknown field: Content-Type: application/zip
info:922 header done
debug:1312 state -- recv-head -> recv-body
debug:1312 state -- recv-body -> done

Subsequent requests on the same client and socket seem to be thrown out of sync:

debug:1312 state -- open      -> send-head
debug:634 custom header 'Range: bytes=1024-2047' added
info:642 GET  request complete
debug:1312 state -- send-head -> send-body
info:1035 no callback - nothing to do
debug:1312 state -- send-body -> recv-head
info:906 HEADER: HTTP/1.1 206 Partial Content/28
info:795 unknown field: Date: Fri, 22 Mar 2024 08:55:05 GMT
info:795 unknown field: Server: Apache
info:795 unknown field: Last-Modified: Mon, 02 Jun 2008 15:30:42 GMT
info:795 unknown field: ETag: "603fc-500000-44eb0adaf4c80"
info:739 Accept-Ranges: bytes
info:773 Content Length: 1024
info:795 unknown field: Cache-Control: no-tranform
info:795 unknown field: Content-Range: bytes 1024-2047/5242880
info:795 unknown field: Keep-Alive: timeout=15, max=99
info:766 connection may be kept alive
info:795 unknown field: Content-Type: application/zip
info:922 header done
debug:1312 state -- recv-head -> recv-body
debug:1312 state -- recv-body -> done     
debug:1312 state -- open      -> send-head
debug:634 custom header 'Range: bytes=2048-3071' added
info:642 GET  request complete
debug:1312 state -- send-head -> send-body
info:1035 no callback - nothing to do
debug:1312 state -- send-body -> recv-head
debug:218 malloc 0x55c7dc393570/256
debug:225 realloc 0x55c7dc393570/512
error:818 Got '\r' with no '\n'
error:904 protocol error (could not read first line)
debug:1312 state -- recv-head -> back-off 
debug:1312 state -- back-off  -> sleeps   
info:663 backing off for 1000 ms, retried 1
debug:1312 state -- sleeps    -> open     
debug:1312 state -- open      -> send-head
debug:634 custom header 'Range: bytes=2048-3071' added
info:642 GET  request complete
debug:1312 state -- send-head -> send-body
info:1035 no callback - nothing to do
debug:1312 state -- send-body -> recv-head
info:906 HEADER: HTTP/1.1 206 Partial Content/28
info:795 unknown field: Date: Fri, 22 Mar 2024 08:55:06 GMT
info:795 unknown field: Server: Apache
info:795 unknown field: Last-Modified: Mon, 02 Jun 2008 15:30:42 GMT
info:795 unknown field: ETag: "603fc-500000-44eb0adaf4c80"
info:739 Accept-Ranges: bytes
info:773 Content Length: 1024
info:795 unknown field: Cache-Control: no-tranform
info:795 unknown field: Content-Range: bytes 2048-3071/5242880
info:795 unknown field: Keep-Alive: timeout=15, max=100
info:766 connection may be kept alive
info:795 unknown field: Content-Type: application/zip
info:922 header done
debug:1312 state -- recv-head -> recv-body
debug:1312 state -- recv-body -> done     
debug:1312 state -- open      -> send-head
debug:634 custom header 'Range: bytes=3072-4095' added
info:642 GET  request complete
debug:1312 state -- send-head -> send-body
info:1035 no callback - nothing to do
debug:1312 state -- send-body -> recv-head
info:906 HEADER: ��Ļ!��x�p�i�륯�-[(9�n���^��$$:$�����pz�p����O���8��J�{n�r��O�������e���J"<�Z����-]��+)v���bռ���N�P���?�9�b#8�'�u��V��:�S��6���R��)��}�Y�ƫ�x&�bMq/182
error:856 unknown HTTP protocol/version: ��Ļ!��x�p�i�륯�-[(9�n���^��$$:$�����pz�p����O���8��J�{n�r��O�������e���J"<�Z����-]��+)v���bռ���N�P���?�9�b#8�'�u��V��:�S��6���R��)��}�Y�ƫ�x&�bMq
error:909 start line parse failed
debug:1312 state -- recv-head -> back-off 
debug:1312 state -- back-off  -> sleeps   
info:663 backing off for 1000 ms, retried 1
^C

See provided sample file built with gcc -o http_test -g http_test.c unix.c httpc.c -lssl. I had to rename it to .txt due to github.
http_test.txt

Accessing Content-Length without manual parsing

It looks like the content length is parsed by httpc during header reception, but it seems to be tucked away in an opaque struct that's inaccessible to the caller. Is there any way to expose this after- or during a request is made, preferably without having to use a data callback and parse the headers again?

The reason it could be good is if you need to download a large file in smaller chunks. You'd then do a HEAD request to check how large the file is, then do a GET with range boundaries.

Error with responses other than OK

I get an error after receiving HTTP status = 201 "Created" after using PUT http method

info:1101 body generated
debug:1327 state -- send-body -> recv-head
info:920 HEADER: HTTP/1.1 201 Created/20
error:895 unexpected HTTP response: Created

Possible Bug on chunked request: missing crlf after chunk size

As per title, when sending a request with "Transfer-Encoding: chunked", the chunk size is currently sent along side the chunk data. It should be followed by '\r\n'. (https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Transfer-Encoding)

httpc/httpc.c

Lines 1015 to 1035 in ff16d71

if (chunky) {
char n[64 + 1] = { 0, };
assert(r < INT_MAX);
const unsigned l = num_to_str(n, r, 16);
if (httpc_network_write(h, (unsigned char*)n, l) < 0) {
r = error(h, "write failed");
break;
}
}
if (httpc_network_write(h, b0->buffer, r) < 0) {
r = error(h, "write failed");
break;
}
if (chunky) {
if (httpc_network_write(h, (unsigned char*)"\r\n", 2) < 0) {
r = error(h, "write failed");
break;
}
}

Here's a simple fix by copying crlf on n buffer (line 1015).

        if (chunky) {
            char n[64 + 1 + 3] = { 0, };
            assert(r < INT_MAX);
            const unsigned l = num_to_str(n, r, 16);
            // EDIT: added crlf after length of chunk
            memcpy(n+l,"\r\n\0",3);
            if (httpc_network_write(h, (unsigned char*)n, l+2) < 0) {
                r = error(h, "write failed");
                break;
            }
        }

Disable retry mechanism

Is there a way to disable retry mechanism?
Right now I see we can reduce connection attempt to a minimum 1 so we can't avoid retrying connection it at least once. If I try to set connection attempts to 0 then there is build error on BUILD_BUG_ON macro.

HTTP 1.0 range support?

The code provides a Range header only to HTTP 1.0 and not for HTTP 1.1. Is that for sure correct? Shouldn't be that reversed?

	if (op == HTTPC_GET && h->os->flags & HTTPC_OPT_HTTP_1_0 && h->position && h->accept_ranges) {
		char range[64 + 1] = { 0, };
		if (httpc_buffer_add_string(h, b0, "Range: bytes=") < 0)
			goto fail;
		httpc_num_to_str(range, h->position, 10);
		if (httpc_buffer_add_string(h, b0, range) < 0)
			goto fail;
		if (httpc_buffer_add_string(h, b0, "-\r\n") < 0)
			goto fail;
	}
		case FLD_ACCEPT_RANGES:
			if (httpc_case_insensitive_search(line, "bytes")) {
				h->accept_ranges = !!(h->os->flags & HTTPC_OPT_HTTP_1_0);
				return info(h, "Accept-Ranges: bytes");
			}

Also using modified at run time Range header during retrying previous transfer after network failure cause obtaining reduced Content-Length and because of that GET request ends somewhere in middle of file. The first assign of h->length should persist from the first Content-Length response header, and shouldn't be modified during retrying. Do you agree?

Separate HTTP status codes from function return values

Right now the return values from the coexist with the status code returned by the server.

enum {
	HTTPC_ERROR = -1, /* negated HTTP error codes also returned */
	HTTPC_OK    =  0, /* all operations completed successfully */
	HTTPC_YIELD =  1, /* call again later - the operation has not finished */
	HTTPC_REUSE =  2, /* operation complete, connection not closed */
};

This can make it tricky in cases where you want both fields. For example, when doing a GET request while using socket reuse the httpc_get call will return HTTPC_REUSE instead of 200 or, in my case, 206.

It could be a good idea to separate these two to make return value handling more intuitive. I don't know what the best option would be. Perhaps passing a struct to the requests, which would also solve #5 .

struct httpc_response_t {
   int16_t status_code;
   size_t content_length;
}

Content-Length set during PUT/POST method

Is there any way to set .length and .length-set fields before requesting header with PUT or POST methods?

I see there is a condition that in dependence of .length_set is setting header field but without any opportunity to modify the .length_set before it is forcing to use "Transfer-Encoding: chunked\r\n" anyway.

	if (op == HTTPC_PUT || op == HTTPC_POST) {
		if (h->length_set) {
			char content[64 + 1] = { 0, };
			if (httpc_buffer_add_string(h, b0, "Content-Length: ") < 0)
				goto fail;
			httpc_num_to_str(content, h->length, 10);
			if (httpc_buffer_add_string(h, b0, content) < 0)
				goto fail;
			if (httpc_buffer_add_string(h, b0, "\r\n") < 0)
				goto fail;
		} else { /* Attempt to send chunked encoding */
			if (httpc_buffer_add_string(h, b0, "Transfer-Encoding: chunked\r\n") < 0)
				goto fail;
		}
	}

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.