vibe-d / vibe-http Goto Github PK

Hello,

I would like to ask about the state of this project - how complete and reliable is it currently (yes, I expect it is still experimental, but still it can be either still lacking major features or relatively close to beta).

Are there any defined plans to release it on code.dlang.org, so that I could include it in my project in the same way I do with vibe.d and vibe-core?

Thank you for any information.

In continuation of our discussion here: vibe-d/vibe-core#48

baseline API with no dynamic per-request memory allocations

Well, it can't be completely avoided. I'd rather formulate in like this:

• all the memory required to hold one http request with all of it's lookup optimization structures (header collection and all that stuff) and various contexts should be allocated efficiently and released after the response is built and control returned to the framework code.
• all strings (header values etc.), contained in request, should span the request buffer

per-request memory must be safely managed (scoped and .dup'ed as soon as the user tries to escape it, either explicitly or implicitly)

I'm strongly against implicit copying. Marked with scope Non-copyable HTTPRequest and HTTPResponse structures that implement Stream interfaces themselves (no easily-accessable socket leakage this way, so no refcounting needed under the hood) as parameters to user callback, all "dangerous" fields and properties return something like

/// Doc-comment that warns the programmer about the lifetime of the memory behind m_val
struct RequestScoped(T)
{
    private T m_val;
    /// doc-comment that warns the programmer about the consequences
    ref T get() { return m_val; }
    T escape() { /* conveniently-written recursive duplication to GC-heap */ }
}

You cannot force user to write memory safe code. D does not support this. No, DIP1000 + "@safe" is not an enforcement. When something cannot be enforced, it is better to provide convenient and non-encumbering conventions, documentation and examples.

no stream proxy objects or dynamic dispatch by default - all combinations of encryption/encoding/compression streams must be instantiated as static types, need to keep in mind how to minimize the amount of template bloat this produces.

encoding - transport
encoding - compression - transport
encoding - compression - encryption - transport
Looks rather modest, templates should not be that terrible.

A drawback of this approach is that each API call would have to determine the static stream type all over again (e.g. ChunkedOutputStream(TLSStream(TCPConnection))).

Let's say dynamic dispatch between streams in the pipeline was collapsed, but this is the end. You need dynamic dispatch for HTTPInterchange.BodyReader\Writer, one way or another. Class will allow socket leak (again, only if you think a programmer is a baboon. Seriously, there is no hope if you escape a variable of heavily-documented type scope RequestScoped!BodyWriter). Or you can emulate VFT in a struct through function pointers, wich will keep uncopyability, but will look questionable.

I also don't think unified HTTPInterchange object is something good. Chained calls with staged callbacks:

• looks ugly, IMO.
• May perform worse, more indirection. On the freight trains of headers, sent by modern browsers, readHeaders may create more problems, than it solves. What should user do there, write a giant switch? How is switch over strings implemented by the compilers? Does it compare lengths first? Does it choose between an rb-tree and hash table, or does it build a state machine? I think an optimized and fine-tuned pre-built header lookup collection is more user-friendly and will be faster.
• Writing straight back to response body without even looking at request is perfectly viable behaviour. Why shoud I call three bethods that do nothing to get it done? I have to write code that means nothing.
• Body should already be read by vibe.d parser well before any user callback are invoked. Body start and length were found, CLRFs were found, all that rfc2616 stuff. All this validation and parsing should be already done before the client code. Client callbacks should not be ran if the request does not conform to HTTP standard. I don't see a benefit from forcing this specific order on the user.

I think the current form of HTTPServerRequestHandlerS callback is very nice.

Summary:

It seems as if URL decoding of URL path parameters happens before path segments are separated, leading the URLRouter to be unable to find matching routes if a path parameter value contains a URL encoded / character as %2F.

Reproduction Steps:

Consider the following example program:

import vibe.vibe;

void main()
{
  auto settings = new HTTPServerSettings;
  settings.port = 8090;
  settings.bindAddresses = ["::1", "127.0.0.1"];
  auto router = new URLRouter();
  router.registerRestInterface(new Api());
  auto listener = listenHTTP(settings, router);
  scope (exit) {
    listener.stopListening();
  }
  logInfo("Please open http://127.0.0.1:8090/ in your browser.");
  runApplication();
}

@path("/api")
interface ApiI {
  @path("/:name/age")
  string getAge(string _name) @safe;
}

class Api : ApiI {
  override
  string getAge(string _name) @safe {
    return _name ~ " is 3 years old.";
  }
}

Called with the :name parameter set to bob works as expected:

l$ curl http://localhost:8090/api/bob/age
"bob is 3 years old."

However, escaping a / character as %2F causes a 404 error. For example, URL encoding the value bob/ham results in bob%2Fham and the following interaction:

$ curl http://localhost:8090/api/bob%2Fham/age
404 - Not Found

Not Found

Internal error information:
No routes match path '/api/bob%2Fham/age'

Expected Result:

For the URL http://localhost:8090/api/bob%2Fham/age, the value bob%2Fham should have been recognized as being the path parameter :name, URL decoded, and then given as the function parameter _name with value bob/ham.

I tried to try switching from vibe-d:http to vibe-http in my application and give it a test run - however it fails to build:

/usr/include/dmd/phobos/std/range/primitives.d(175,38): Deprecation: alias byKeyValue this is deprecated - Iterate over .byKeyValue instead.
/usr/include/dmd/phobos/std/range/primitives.d(177,27): Deprecation: alias byKeyValue this is deprecated - Iterate over .byKeyValue instead.
/usr/include/dmd/phobos/std/range/primitives.d(175,38): Deprecation: alias byKeyValue this is deprecated - Iterate over .byKeyValue instead.
/usr/include/dmd/phobos/std/range/primitives.d(177,27): Deprecation: alias byKeyValue this is deprecated - Iterate over .byKeyValue instead.
../../../.dub/packages/vibe-core-1.9.3/vibe-core/source/vibe/core/task.d(683,3): Error: static assert:  "The arguments passed to run(Worker)Task must not exceed 128 bytes in total size: 144LU bytes"
../../../.dub/packages/vibe-core-1.9.3/vibe-core/source/vibe/core/core.d(331,46):        instantiated from here: set!(void function(TCPConnection, TCPConnection, HTTP2ServerContext) @safe, TCPConnection, TCPConnection, HTTP2ServerContext)
../../../.dub/packages/vibe-core-1.9.3/vibe-core/source/vibe/core/core.d(416,11):        instantiated from here: __lambda6!(TaskFuncInfo)
../../../.dub/packages/vibe-core-1.9.3/vibe-core/source/vibe/core/core.d(331,9):        instantiated from here: runTask_internal!((ref tfi)
{
tfi.set(task, args);
}
)
../../../.dub/packages/vibe-http-0.1.0/vibe-http/source/vibe/http/internal/http2/http2.d(241,16):        ... (1 instantiations, -v to show) ...
../../../.dub/packages/vibe-http-0.1.0/vibe-http/source/vibe/http/internal/http2/http2.d(219,23):        instantiated from here: handleHTTP2FrameChain!(TCPConnection)
../../../.dub/packages/vibe-http-0.1.0/vibe-http/source/vibe/http/internal/http1.d(243,25):        instantiated from here: handleHTTP2Connection!(TCPConnection)

While testing ALPN with HTTP/2 I ran into the following AssertError:

core.exception.AssertError@/home/fra/_progs/saoc/vibe-core/source/vibe/internal/interfaceproxy.d(132):    
Extraction of wrong type from InterfaceProxy.    
----------------                                              
??:? _d_assert_msg [0x60187dc2]                                             
??:? nothrow @trusted vibe.internal.freelistref.FreeListRef!(vibe.stream.openssl.OpenSSLStream, true).F    
reeListRef vibe.internal.interfaceproxy.InterfaceProxy!(vibe.core.stream.Stream).InterfaceProxy.extract    
!(vibe.internal.freelistref.FreeListRef!(vibe.stream.openssl.OpenSSLStream, true).FreeListRef).extract(    
) [0xe2f1ed8b]                                                                                             
??:? @safe bool vibe.http.internal.http1.originalHandleRequest(vibe.internal.interfaceproxy.InterfacePr    
oxy!(vibe.core.stream.Stream).InterfaceProxy, vibe.core.net.TCPConnection, vibe.http.server.HTTPServerC    
ontext, vibe.http.server.HTTPServerSettings, ref bool, scope stdx.allocator.IAllocator) [0xe2f7f2cd]    
??:? @trusted void vibe.http.internal.http1.handleHTTP1Request!(vibe.core.net.TCPConnection).handleHTTP    
1Request(vibe.core.net.TCPConnection, vibe.http.server.HTTPServerContext).__lambda3() [0xe2f82451]    
??:? @safe void vibe.http.internal.http1.handleHTTP1Request!(vibe.core.net.TCPConnection).handleHTTP1Re    
quest(vibe.core.net.TCPConnection, vibe.http.server.HTTPServerContext) [0xe2f82292]    
??:? @safe void vibe.http.internal.http1.handleHTTP1RequestChain!(vibe.core.net.TCPConnection).handleHT    
TP1RequestChain(vibe.core.net.TCPConnection, vibe.http.server.HTTPServerContext) [0xe2f82000]    
??:? void vibe.core.task.TaskFuncInfo.set!(void function(vibe.core.net.TCPConnection, vibe.http.server.    
HTTPServerContext) @safe*, vibe.core.net.TCPConnection, vibe.http.server.HTTPServerContext).set(ref voi    
d function(vibe.core.net.TCPConnection, vibe.http.server.HTTPServerContext) @safe*, ref vibe.core.net.T    
CPConnection, ref vibe.http.server.HTTPServerContext).callDelegate(ref vibe.core.task.TaskFuncInfo) [0x    
e2f603bb]    
??:? void vibe.core.task.TaskFuncInfo.call() [0xe3042f99]    
??:? nothrow void vibe.core.task.TaskFiber.run() [0xe30425a1]    
??:? void core.thread.Fiber.run() [0x6018ca57]    
??:? fiber_entryPoint [0x6018be22]    
??:? [0xffffffff]    
Program exited with code -6    
    
[Process exited 2]

The error above regards the following snipped of code in vibe.http.internal.http1:

166     if (req.tls) {
167         version (HaveNoTLS) assert(false);
168         else {    
169             static if (is(InterfaceProxy!ConnectionStream == ConnectionStream))
170                 req.clientCertificate = (cast(TLSStream)http_stream).peerCertificate;
171             else
172                 req.clientCertificate = http_stream.extract!TLSStreamType.peerCertificate;
173                 assert(false);
174         }       
175     } 

To reproduce the error, it is sufficient to initialize an HTTPS server and send a request. The assertion error occours using both HTTP/1.1 and HTTP/2, without difference. It is to be noted that the program never enters the static if clause (line 169) and the else clause (line 172) fails before reaching assert(false).

It looks like a type check which is never true (the assert is thrown at runtime after a typeid check in vibe.internal.interfaceproxy:132), but removing the compile time checks to allow for a direct cast of http_stream to type TLSStream does not work either, so I guess we should change how the http_stream is declared / managed in case of a TLS connection.

Note: This issue affects only the use of client certificates, therefore it is not critical to HTTP/2 introduction.

Is it currently possible to use HTTPClientRequest to send requests with content-type multipart/form-data? An example use-case would be to send a request to an endpoint that takes a file as an upload.

The protocol is described here in RFC 2388: https://datatracker.ietf.org/doc/html/rfc2388

An example endpoint that uses this protocol would be GitLab's POST /api/v4/projects/{id}/uploads endpoint: https://docs.gitlab.com/ee/api/projects.html#upload-a-file

Is this an area where contributions are desired? And if so, what is the intended interface for using this capability? I suspect the HTTPClientRequest::writePart() function is meant to be involved, but I'm not super clear what a simple invocation would look like.

OS: Windows 10 x64
Lib. version: 0.3.1
Compiler: LDC2 1.28.0
Compilation error:
dub\packages\vibe-http-0.3.1\vibe-http\source\vibe\http\websockets.d(41,15): Error: module exception is in file 'vibe\internal\exception.d' which cannot be read