mirage / ocaml-conduit Goto Github PK

Dereference URIs into communication channels for Async or Lwt

License: ISC License

Makefile 0.33% OCaml 99.00% Shell 0.67%

ocaml-conduit's Introduction

conduit -- an OCaml network connection establishment library

The conduit library takes care of establishing and listening for TCP and SSL/TLS connections for the Lwt and Async libraries.

The reason this library exists is to provide a degree of abstraction from the precise SSL library used, since there are a variety of ways to bind to a library (e.g. the C FFI, or the Ctypes library), as well as well as which library is used (just OpenSSL for now).

By default, OpenSSL is used as the preferred connection library, but you can force the use of the pure OCaml TLS stack by setting the environment variable CONDUIT_TLS=native when starting your program.

The opam packages available are:

conduit: the main Conduit module
conduit-lwt: the portable Lwt implementation
conduit-lwt-unix: the Lwt/Unix implementation
conduit-async the Jane Street Async implementation
conduit-mirage: the MirageOS compatible implementation

Debugging

Some of the Lwt_unix-based modules use a non-empty CONDUIT_DEBUG environment variable to output debugging information to standard error. Just set this variable when running the program to see what URIs are being resolved to.

Further Informartion

API Docs: https://mirage.github.io/ocaml-conduit/
WWW: https://github.com/mirage/ocaml-conduit
E-mail: [email protected]
Bugs: https://github.com/mirage/ocaml-conduit/issues

ocaml-conduit's People

Contributors

Stargazers

Watchers

ocaml-conduit's Issues

Bug: 'init' in Conduit_lwt_unix_ssl.Server ignores custom context when calling 'accept'

In line 100 of conduit_lwt_unix_ssl.ml the call to accept doesn't use a custom ssl context if provided.

As a quick fix, this:

(fun () -> accept s >>= process_accept ~timeout callback)

should become

(fun () -> accept ?ctx s >>= process_accept ~timeout callback)

Getting IP address from Client that connected.

I'm trying to get the IP address of a client that connected to a conduit powered server.

I have a Conduit_lwt_unix.flow object, which I've tried working with:

  let address = match ch with
    | Conduit_lwt_unix.TCP g -> Some (g.Conduit_lwt_unix.ip |> Ipaddr.to_string)
    | _ -> None

but this string appears to be the address of the server. I've also tried playing with the numerous Conduit_lwt_unix.sexp_* functions but some of these just hang and others don't return fruitful information. Any simple self-contained code sample is much appreciated.

Documentation misses some doc-comments

The comments on the variants (like the difference between TLS`, TLS_native`, etc.) are not propagated there:
http://mirage.github.io/ocaml-conduit/Conduit_lwt_unix.html#TYPEclient
(but they are there: https://github.com/mirage/ocaml-conduit/blob/master/lib/conduit_lwt_unix.mli#L34)

Can't properly stop lwt server

There appear to be two ways to stop a server started with Conduit_lwt_unix.serve, and neither seems to work correctly.

If I pass ~stop and wake up the stop thread, the server will accept the next connection and then exit its loop. However, the server thread never returns and the listening socket remains open.
If I Lwt.cancel the server thread, the server immediately stops accepting connections and closes all existing connections. The server thread returns, but the listening socket remains open.

In both cases, I'm left with a listening socket that never accepts connections and prevents me from starting a new server on that port.

I think the correct behaviour in both cases is to close the listening socket when the accept loop exits, but I'm not sure why there are two mechanisms for this or what the intended differences are.

Conduit_mirage combinators are order dependent

Tcp must be done before tls, the other way will give you an error message. It doesn't seem to be documented, so I guess it's not completely on purpose.

(It was very annoying to debug)

Binding on Unix to particular IP not working

Using cohttp-server-lwt it appears to bind to all interfaces even with a specific -s

Conduit_async.serve silentily fails when when OpenSSL is selected but no certificates are provided

Still need to check on this to be completely sure, just reporting this as I really think this is what happens.

support setting cipher_list

The call would be Ssl.set_cipher_list, but how to expose this in server_tls_config without breaking existing applications that don't want to set a cipherlist?

Reason for setting cipherlist :by default apparently cohttp-lwt-server accepts TLS_RSA_WITH_DES_CBC_SHA (0x9) too.

Huge GC pressure caused by I/O in Conduit_lwt_unix_ssl

Conduit_lwt_unix_ssl uses Lwt_ssl.in_channel_of_descr and out_channel_of_descr to create the Lwt_io channels it uses. Unfortunately, these use under the hood Lwt_ssl.read/write_bytes, which allocate a string the size of the data being read/written.

Adding insult to injury, the default buffer size is 4 KB, well over the 256-word limit for blocks in the minor heap. So the same amount of data being read/written is allocated in the major heap, to become garbage right away. GC pressure increases, lots of mark&sweeping ensues, and on top of it the more frequent minor collections cause even more soon-to-become-garbage data to be promoted.

Refer to ocsigen/ocsigenserver#49 for the consequences of such allocation. See also mirage/ocaml-cohttp#207.

Have a Channel backend for Cohttp with raw Request/Response

Requested by @talex5 and @pqwy :

It might be nice if mirage would let you configure an HTTP server without using conduit. Resolving URLs needs to be dynamic, but when providing a service you usually know statically which transport you want (http, https or vchan).

Conduit_async_ssl. ssl_listen Leaks Pipes

See mirage/ocaml-cohttp#287

It appears that Conduit_async_ssl.ssl_listen creates a couple of pipes that are never closed. See the above ticket for reproduction of this bug.

unify accept loop and process accept in lwt (conduit_lwt_unix,conduit_lwt_tls, conduit_lwt_unix_ssl)

bugfixes/changes would have to be applied 3 times otherwise, e.g. #111
To avoid creating circular dependencies perhaps should be a conduit_lwt.ml for shared code.

More types of conduits

There are many more types of conduits that the ones currently exposed by that library. That would be great to have a common home to put all of them together. See for instance https://github.com/docker/datakit/blob/master/src/datakit_conduit.ml which abstract socket listening over:

Unix domain sockets
HyperV sockets
Windows named pipes

Currently conduit is not in shape to welcome this new backends as I am not very happy with the current API (note: I wrote parts of that API, so I am certainly to blame). Would be great if we could have a better design using some new features of OCaml, such as open types. WDYT?

Add Ipv6 support for Unix conduit

Right now it's only an IPv4 bind.

Expose conduit destination

As a security for a webservice I'm writing, I have to log the IP address of the requester. Unix conduit exposes this indirectly via sexp_of_flow, but for Mirage conduit, that information is hidden. It would be nice to have this available in a uniform way. There may be a need to change to more generally change behavior base on the remote endpoint (security, logging, quotas, etc).

I've attached a primitive implementation I threw together for my current projects, but it doesn't cover Vchan flows. (I've attached the specific files here as well).
https://github.com/juanchanco/ocaml-conduit/tree/Flow.get_dest

files.zip

Development very annoying due to ppx

Using my editor, I can't jump to errors because the file referenced is the temporary source file generated by the ppx extension. We should really fix this :(

Documentation and examples on using conduit

Over on mirage/mirage#474 @wmhaddad has asked for some docs (and examples) on how to use conduit. From the last MirageOS call, this (and vchan) are important components of the work he's doing.

If anyone can help out or point to existing material, that would be useful.

Install .mli

Please install the .mli files — and intro.html.

`Resolver_mirage.register` should not overwrite the resolver's service

Currently, Resolver_mirage.register do some stuff to register the DNS functions and then it calls:

      Resolver_lwt.set_service ~f:static_service res;

which reset the services of res. This is annoying if you are in an unikernel and want to add some services which are not very common (and thus are not defined in the static Uri_services).

with vchan results in surprising symbol error on Unix

conduit 0.8.3 and vchan 2.0.2:

# #require "conduit.lwt-unix";;
Error: The external function `stub_evtchn_virq_dom_exc' is not available

This error occurs when trying to use any dependent library (and does not have an obvious cause) and is problematic for no-switch development of Mirage applications. Some kind of configurations, dummy stub, or stub-with-an-error-message-name would be helpful.

TLS authentication trust anchors

should default to exceptional behaviour (empty trust anchors), instead of not validating the certificate chain, see

ocaml-conduit/lib/conduit_lwt_tls.ml

Line 31 in 78b113f

X509_lwt.authenticator `No_authentication_I'M_STUPID >>= fun authenticator ->

and likely similar cases for other front/backends; further discussion at mirage/ocaml-cohttp#479 (comment)

Cannot install cohttp with latest Core due to conduit

There seems to be a conflict with "conduit" with the latest versions of other packages. This prevents installing the cohttp package (which relies on conduit) with the latest version of other things in opam.

This is preventing installing dependencies for the Real World OCaml book, per instructions here: https://github.com/realworldocaml/book/wiki/Installation-Instructions which is very confusing.

Is there some way you can modify your conduit not to give huge numbers of scary downgrade warnings? For now, I'll simply not install cohttp & conduit. I have installed every other dependency of cohttp without any downgrade warnings.

This seems to be related to the commit here: 4f876e5 but it is not explained in the commit notes WHY this restriction was applied.

Many thanks!

$ opam install conduit
The following actions will be performed:
  ⊘  remove    ppx_jane           113.24.01               [uses ppx_expect]
  ∗  install   camlp4             4.02+system             [required by async, cstruct]
  ∗  install   ocplib-endian      0.8                     [required by cstruct]
  ⊘  remove    ppx_sexp_message   113.24.00               [conflicts with ppx_core]
  ⊘  remove    ppx_let            113.24.00               [conflicts with ppx_core]
  ⊘  remove    ppx_expect         113.24.00               [conflicts with fieldslib]
  ∗  install   type_conv          113.00.02               [required by conduit]
  ∗  install   pipebang           113.00.00               [required by async_extra]
  ∗  install   herelib            112.35.00               [required by async_kernel]
  ↻  recompile lwt                2.5.1                   [uses camlp4]
  ↘  downgrade ppx_core           113.24.00 to 113.09.00  [upstream changes]
  ↘  downgrade variantslib        113.24.00 to 109.15.03  [uses camlp4]
  ↘  downgrade sexplib            113.24.00 to 113.00.00  [required by conduit]
  ∗  install   pa_ounit           113.00.00               [required by async_extra]
  ↘  downgrade fieldslib          113.24.00 to 113.00.00  [required by async_extra]
  ∗  install   enumerate          111.08.00               [required by core]
  ∗  install   comparelib         113.00.00               [required by async_rpc_kernel]
  ↘  downgrade bin_prot           113.24.00 to 113.00.00  [required by async_extra]
  ↻  recompile lambda-term        1.9                     [uses lwt]
  ↘  downgrade ppx_optcomp        113.24.00 to 113.09.00  [uses ppx_core]
  ∗  install   pa_structural_sexp 113.00.00               [required by core]
  ∗  install   pa_bench           113.00.00               [required by core]
  ∗  install   custom_printf      113.00.00               [required by core]
  ∗  install   pa_test            112.24.00               [required by async_kernel]
  ↘  downgrade typerep            113.24.00 to 113.00.00  [uses bin_prot]
  ↻  recompile utop               1.18.1                  [uses camlp4]
  ↘  downgrade ppx_driver         113.24.00 to 113.09.00  [upstream changes]
  ↘  downgrade core_kernel        113.24.00 to 113.00.00  [required by async_kernel]
  ↘  downgrade ppx_type_conv      113.24.00 to 113.09.00  [required by ppx_compare]
  ↘  downgrade ppx_pipebang       113.24.00 to 113.09.00  [upstream changes]
  ↘  downgrade ppx_here           113.24.00 to 113.09.00  [upstream changes]
  ↘  downgrade core               113.24.01 to 113.00.00  [required by async_extra]
  ↘  downgrade async_kernel       113.24.00 to 113.00.00  [required by async]
  ↘  downgrade ppx_variants_conv  113.24.00 to 113.09.00  [upstream changes]
  ↘  downgrade ppx_typerep_conv   113.24.00 to 113.09.00  [upstream changes]
  ↘  downgrade ppx_sexp_conv      113.24.00 to 113.09.00  [required by ipaddr]
  ↘  downgrade ppx_inline_test    113.24.00 to 113.09.00  [upstream changes]
  ↘  downgrade ppx_fields_conv    113.24.00 to 113.09.00  [upstream changes]
  ↘  downgrade ppx_enumerate      113.24.00 to 113.09.00  [upstream changes]
  ↘  downgrade ppx_compare        113.24.00 to 113.09.00  [upstream changes]
  ↘  downgrade ppx_bin_prot       113.24.01 to 113.09.00  [upstream changes]
  ↘  downgrade ppx_fail           113.24.00 to 113.09.00  [upstream changes]
  ↘  downgrade textutils          113.24.00 to 112.17.00  [uses camlp4]
  ↘  downgrade re2                113.24.00 to 113.00.00  [uses bin_prot]
  ↘  downgrade async_unix         113.24.00 to 113.00.00  [required by async]
  ↘  downgrade async_rpc_kernel   113.24.00 to 113.00.00  [required by async_extra]
  ↘  downgrade ppx_sexp_value     113.24.00 to 113.09.00  [upstream changes]
  ↘  downgrade ppx_custom_printf  113.24.00 to 113.09.00  [upstream changes]
  ∗  install   ipaddr             2.7.0                   [required by conduit]
  ↘  downgrade ppx_bench          113.24.00 to 113.09.00  [upstream changes]
  ↘  downgrade ppx_assert         113.24.00 to 113.09.00  [upstream changes]
  ↘  downgrade core_bench         113.24.00 to 112.35.00  [uses bin_prot]
  ↘  downgrade core_extended      113.24.00 to 113.00.00  [uses bin_prot]
  ↘  downgrade async_extra        113.24.00 to 113.00.00  [required by async]
  ↘  downgrade async              113.24.00 to 113.00.00  [required by conduit]
  ∗  install   cstruct            1.9.0                   [required by conduit]
  ↻  recompile async_graphics     0.5.1                   [uses async]
  ∗  install   conduit            0.10.0                
===== ∗  15   ↻  4   ↘  35   ⊘  4 =====
Do you want to continue ? [Y/n] n


$ opam list
# Installed packages for system:
alcotest               0.4.7  Alcotest is a lightweight and colourful test framework.
async              113.24.00  Monadic concurrency library
async_extra        113.24.00  Monadic concurrency library
async_graphics         0.5.1  Async wrapper for the OCaml Graphics library
async_kernel       113.24.00  Monadic concurrency library
async_rpc_kernel   113.24.00  Platform-independent core of Async RPC library
async_unix         113.24.00  Monadic concurrency library
base-bigarray           base  Bigarray library distributed with the OCaml compiler
base-bytes              base  Bytes library distributed with the OCaml compiler
base-ocamlbuild         base  OCamlbuild binary and libraries distributed with the OCaml compiler
base-threads            base  Threads library distributed with the OCaml compiler
base-unix               base  Unix library distributed with the OCaml compiler
base64                 2.0.0  Base64 encoding and decoding library
bin_prot           113.24.00  A binary protocol generator
biniou                 1.0.9  Binary data format designed for speed, safety, ease of use and backwar
camomile               0.8.5  A comprehensive Unicode library
cmdliner               0.9.8  Declarative definition of command line interfaces for OCaml
core               113.24.01  Industrial strength alternative to OCaml's standard library
core_bench         113.24.00  Benchmarking library
core_extended      113.24.00  Extra components that are not as closely vetted or as stable as Core
core_kernel        113.24.00  Industrial strength alternative to OCaml's standard library
cppo                   1.3.1  Equivalent of the C preprocessor for OCaml programs
cryptokit               1.10  Cryptographic primitives library. 
easy-format            1.2.0  High-level and functional interface to the Format module of the OCaml 
fieldslib          113.24.00  Syntax extension to define first class values representing record fiel
lambda-term              1.9  Terminal manipulation library for OCaml
lwt                    2.5.1  A cooperative threads library for OCaml
magic-mime             1.0.0  Convert file extensions to MIME types
menhir              20151112  LR(1) parser generator
ocamlbuild                 0  Build system distributed with the OCaml compiler since OCaml 3.10.0
ocamlfind              1.5.6  A library manager for OCaml
ounit                  2.0.0  Unit testing framework loosely based on HUnit. It is similar to JUnit,
ppx_assert         113.24.00  Assert-like extension nodes that raise useful errors on failure
ppx_bench          113.24.00  Syntax extension for writing in-line benchmarks in ocaml code
ppx_bin_prot       113.24.01  Generation of bin_prot readers and writers from types
ppx_compare        113.24.00  Generation of comparison functions from types
ppx_core           113.24.00  Standard library for ppx rewriters
ppx_custom_printf  113.24.00  Printf-style format-strings for user-defined string conversion
ppx_deriving             3.1  Type-driven code generation for OCaml >=4.02
ppx_driver         113.24.00  Feature-full driver for OCaml AST transformers
ppx_enumerate      113.24.00  Generate a list containing all values of a finite type
ppx_expect         113.24.00  Cram like framework for OCaml
ppx_fail           113.24.00  Add location to calls to failwiths
ppx_fields_conv    113.24.00  Generation of accessor and iteration functions for ocaml records
ppx_here           113.24.00  Expands [%here] into its location
ppx_inline_test    113.24.00  Syntax extension for writing in-line tests in ocaml code
ppx_jane           113.24.01  Standard Jane Street ppx rewriters
ppx_let            113.24.00  Monadic let-bindings
ppx_optcomp        113.24.00  Optional compilation for OCaml
ppx_pipebang       113.24.00  A ppx rewriter that inlines reverse application operators `|>` and `|!
ppx_sexp_conv      113.24.00  Generation of S-expression conversion functions from type definitions
ppx_sexp_message   113.24.00  A ppx rewriter for easy construction of s-expressions
ppx_sexp_value     113.24.00  A ppx rewriter that simplifies building s-expressions from ocaml value
ppx_tools             0.99.3  Tools for authors of ppx rewriters and other syntactic tools
ppx_type_conv      113.24.00  Support Library for type-driven code generators
ppx_typerep_conv   113.24.00  Generation of runtime types from type declarations
ppx_variants_conv  113.24.00  Generation of accessor and iteration functions for ocaml variant types
re                     1.5.0  RE is a regular expression library for OCaml
re2                113.24.00  OCaml bindings for RE2, Google's regular expression library
react                  1.2.0  Declarative events and signals for OCaml
result                   1.1  Compatibility Result module
sexplib            113.24.00  Library for serializing OCaml values to and from S-expressions
stringext              1.4.0  Extra string functions for OCaml
textutils          113.24.00  Text output utilities
typerep            113.24.00  typerep is a library for runtime types.
uri                   1.3.12  RFC3986 URI/URL parsing library
utop                  1.18.1  Universal toplevel for OCaml
variantslib        113.24.00  Part of Jane Street's Core library
yojson                 1.3.1  Yojson is an optimized parsing and printing library for the JSON forma
zed                      1.4  Abstract engine for text edition in OCaml


$ opam switch
--     -- 3.07    Official 3.07 release
--     -- 3.08.0  Official 3.08.0 release
--     -- 3.08.1  Official 3.08.1 release
--     -- 3.08.2  Official 3.08.2 release
--     -- 3.08.3  Official 3.08.3 release
--     -- 3.08.4  Official 3.08.4 release
--     -- 3.09.0  Official 3.09.0 release
--     -- 3.09.1  Official 3.09.1 release
--     -- 3.09.2  Official 3.09.2 release
--     -- 3.09.3  Official 3.09.3 release
--     -- 3.10.0  Official 3.10.0 release
--     -- 3.10.1  Official 3.10.1 release
--     -- 3.10.2  Official 3.10.2 release
--     -- 3.11.0  Official 3.11.0 release
--     -- 3.11.1  Official 3.11.1 release
--     -- 3.11.2  Official 3.11.2 release
--     -- 3.12.0  Official 3.12.0 release
--     -- 3.12.1  Official 3.12.1 release
--     -- 4.00.0  Official 4.00.0 release
--     -- 4.00.1  Official 4.00.1 release
--     -- 4.01.0  Official 4.01.0 release
--     -- 4.02.0  Official 4.02.0 release
--     -- 4.02.1  Official 4.02.1 release
--     -- 4.02.2  Official 4.02.2 release
--     -- 4.02.3  Official 4.02.3 release
system  C system  System compiler (4.02.3)
#121 more patched or experimental compilers, use '--all' to show

channels hang when stream raises exception

When a body stream raises an exception the connection is kept open (probably forever).

Since the headers were already sent we don't have much choice but to close the channel, with chunked encoding curl would report a proper error then:

curl: (18) transfer closed with outstanding read data remaining

I was testing a fix like this

-    Lwt.ignore_result (Lwt.pick events >>= fun () -> close (ic,oc))
+    Lwt.finalize (fun () -> Lwt.pick events) (fun () -> close (ic,oc)) |>
+    Lwt.ignore_result

Not sure if Lwt.finalize or Lwt.on_termination would be preferred here (finalize creates an extra closure?).

License file is missing

While the source indicates that this is MIT licensed, without the license file its hard to tell at a glance. For companies with a legal department it could also be an issue of which files are open and which are not. If you add a license file that just removes all the confusion.

Conduit_async.connect should be able to use an Uri as well as an IP

Currently, I can't use it because I have an Uri and I don't want to resolve it myself.

limit maximum number of active sockets

When Conduit runs out of sockets in Conduit_lwt_unix_ssl it loops consuming 100% CPU due to ignoring all exceptions: https://github.com/mirage/ocaml-conduit/blob/master/lib/conduit_lwt_unix_ssl.ml#L103

accept(3, 0x7ffc3bd61d50, [112])        = -1 EMFILE (Too many open files)
accept(3, 0x7ffc3bd61d50, [112])        = -1 EMFILE (Too many open files)
accept(3, 0x7ffc3bd61d50, [112])        = -1 EMFILE (Too many open files)
accept(3, 0x7ffc3bd61d50, [112])        = -1 EMFILE (Too many open files)

The behaviour is different from Conduit_lwt_unix which would apparently just quit the accept loop.

It would be better if Conduit would count the number of globally accepted connections and throttle (wait) until the number of connections drops below a maximum.
FWIW Ocsigen uses a maximum number of connections = 350 by default.

Since this would be a global setting (if you have multiple conduits listening on different ports they should use the same shared counter) would it make sense to have the counter in Conduit.mli and have all conduit implementions use that?
Obviously throttling itself has to be implemented differently in Lwt and Async.
For now I've added an ugly hack on my branch, but this maximum should also be configurable.

Conduit_lwt_{ssl/tls} serializes on Ssl accept

When connecting to a remote server the SSL handshake can take e.g. 150ms, and due to the code below and no new connections were accepted until previous one finished the SSL handshake:
accept config s >|= process_accept ~timeout callback

It'd be better to perform the accept and handle the SSL handshake asynchronously like the callback itself. This can be implemented nicely by unifying the accept logic into Conduit_lwt_unix as in #112.

Mirage Nocrypto initialise should be done within Lwt main loop

let _ = Nocrypto_entropy_lwt.initialize ()

the result shouldnt be ignored. See #64 for discussion.

Handle connection to IPv6 link local addresses

For a project I need to connect to link-local IPv6 addresses.
As this is not possible under normal OCaml Unix, I wrote another binding to the necessary unix primitives using Ctypes, here: http://github.com/vbmithr/ocaml-sockopt.

Now I'd like irminsule to fetch data from an IPv6 link local address, and irminsule uses cohttp for that, which in turn uses ocaml-conduit.

So I'm asking what is in your opinion the best/cleanest way to achieve this.
The first thing I'm thinking about is to hack ocaml-conduit to plumb it into my ocaml-sockopt library but if I'm working on that, I'd better have in integrated. So what do you think ?

Give access to TLS connection

From this issue: mirage/ocaml-cohttp#509, reading cohttp/conduit code led to post my problem here.

In a cohttp server, I need to access the TLS session (a Tls_lwt.t). The IO.conn parameter of callback function of HTTP server (from Cohttp_lwt_unix_io) is a Conduit_lwt_unix.flow. I can retrieve the file descriptor, since the flow is a TCP but I cannot retrieve the Tls session to perform operations on it.

I don't have a solution to make this TLS session available without changing the Conduit_lwt_unix.flow type, for example by adding a TLS constructor, something like:

type flow =
 ...
| Tls of Tls_lwt.Unix.t * flow

compatibility of 0.6.0 with async_ssl needs declaration in opam file

https://travis-ci.org/mirage/ocaml-cohttp/jobs/58886818#L2638

mirage resolver fails with "name resolution failed" when given ip:port based uri

This may be one of the corner cases, but I tried issuing HTTP requests using ip:port generated uri.

While targeting to unix and using resolver_unix_system everything works fine, but switching to xen and resolver_dns gives

Fatal error: exception (Failure "name resolution failed: unknown endpoint type")

I think this exception comes from this function: dns_stub_resolver. Before referring to DNS.gethostbyname, if the host part of uri is an ip address, maybe we could just return it and the port. If this is the case, I could submit a patch later.

Release 0.8.3 on opam

I see that @samoht tagged 0.8.3 a few weeks ago but the release hasn't made it's way to opam.

Anything holding it up?

Error while building mirage-www on 4.02.1

Not sure if this issue belongs here or if it's specific to mirage-www. Note the line that says Error: Unbound module Conduit_mirage.No_TLS. I'm rebuilding my switches so will try this again later (I normally use 4.01).

$ make
ocamlbuild -use-ocamlfind -pkgs lwt.syntax,conduit.mirage,cow.syntax,cowabloga,cstruct,io-page,io-page.unix,lwt,mirage-clock-unix,mirage-console.unix,mirage-fs-unix,mirage-http,mirage-net-unix,mirage-types,mirage-types.lwt,mirage.runtime,tcpip.stack-direct -tags "syntax(camlp4o),annot,bin_annot,strict_sequence,principal" -tag-line "<static*.*>: -syntax(camlp4o)" -cflag -g -lflags -g,-linkpkg main.native
+ ocamlfind ocamlc -c -g -annot -bin-annot -principal -strict-sequence -package tcpip.stack-direct -package mirage.runtime -package mirage-types.lwt -package mirage-types -package mirage-net-unix -package mirage-http -package mirage-fs-unix -package mirage-console.unix -package mirage-clock-unix -package lwt -package io-page.unix -package io-page -package cstruct -package cowabloga -package cow.syntax -package conduit.mirage -package lwt.syntax -syntax camlp4o -o main.cmo main.ml
File "main.ml", line 89, characters 18-39:
Error: Unbound module Conduit_mirage.No_TLS
Command exited with code 2.
Hint: Recursive traversal of subdirectories was not enabled for this build,
  as the working directory does not look like an ocamlbuild project (no
  '_tags' or 'myocamlbuild.ml' file). If you have modules in subdirectories,
  you should add the option "-r" or create an empty '_tags' file.

  To enable recursive traversal for some subdirectories only, you can use the
  following '_tags' file:

      true: -traverse
      <dir1> or <dir2>: traverse

Compilation unsuccessful after building 20 targets (0 cached) in 00:00:02.
make: *** [main.native] Error 10

Handle failed lwt threads in server

Currently failing lwt threads are swallowed (I checked Conduit_lwt_unix.serve, but maybe there are other places). I think the error should be reported somehow, or at least this behaviour should be documented. See example below.

let () =
  let ctx = Conduit_lwt_unix.default_ctx in
  let mode = `TCP (`Port 1234) in
  let main = Conduit_lwt_unix.serve ~ctx ~mode (fun flow ic oc -> Lwt.fail_with "oh no") in
  Lwt_main.run main

Conduit is build 2 times

make install causes everything to be rebuilt. So when doing an opam install conduit, the compilation is actually performed 2 times.

Cannot install conduit

File "lib/conduit_async.ml", line 18, characters 1-29:
optcomp: cannot open "conduit_config.mlh": conduit_config.mlh: No such file or directory
File "lib/conduit_async.ml", line 1:
Error: Error while running external preprocessor
Command line: /home/vb/code/ocaml-conduit/ppx 'lib/conduit_async.ml' > /tmp/ocamlpp89a66e

Getting an endp for a ws:// uri

Hello,

I don't understand how to get an endp value for an uri with scheme ws://. Could you provide an example ?

Using Resolver_lwt_unix.system (Uri.of_string "ws://localhost:8080"), I get a Failure("resolution failed: unknown scheme").

Docs not at given URL

README says http://anil-code.recoil.org/ocaml-conduit has the ocamldoc docs, but that seems to give a 404

Conduit need a lot of RAM to build

1Go is not enough. Even core and async compile fine with this amount of RAM. Why?

Conduit_mirage: make TLS dependency optional

From @talex5:

It would be good to make the config type abstract, so that conduit doesn't bring in dependencies on the TLS libraries.

Conduit_lwt_unix_ssl leaks file descriptors

Lwt_io.close on the channel obtained fromLwt_ssl.out_channel_of_descr leaks the underlying FD if ssl_shutdown fails. This can occur if the client has disconnected uncleanly (e.g. TCP RST, or simply gone in the case of a mobile client, etc.).
As a consequence my cohttp.lwt server died every few weeks due to running out of file descriptors.

Pull request coming up

Conduit_mirage silently skips certificate validation

config.ml:

open Mirage

let stackv4 = generic_stackv4 default_console tap0
let tls_conduit = conduit_direct ~tls:true stackv4
let main = foreign "Unikernel.Main" (resolver @-> conduit @-> job)

let () =
  register "client" [main $ resolver_dns stackv4 $ tls_conduit]

unikernel.ml:

open Lwt.Infix

let test_server = "https://example.com"
(* A web-site with a self-signed or otherwise untrusted X.509 certificate. *)

module Main(R : Resolver_lwt.S)(C : Conduit_mirage.S) = struct
  let start resolver conduit =
    R.resolve_uri resolver ~uri:(Uri.of_string test_server) >>= fun endp ->
(*     Logs.info (fun f -> f "Resolved to address: %s" (Sexplib.Std.string_of_sexp (Conduit.sexp_of_endp endp))); *)
    Conduit_mirage.client endp >>= fun client ->
    C.connect conduit client >>= fun flow ->
    Conduit_mirage.Flow.write flow (Cstruct.of_string "GET / HTTP/1.0\r\n\r\n") >>= function
    | `Eof | `Error _ -> assert false
    | `Ok () ->
      let rec loop () =
        Conduit_mirage.Flow.read flow >>= function
        | `Error _ -> failwith "Error!"
        | `Eof ->
          Logs.info (fun f -> f "End of file");
          Lwt.return_unit
        | `Ok data ->
          Logs.info (fun f -> f "Got: %s" (Cstruct.to_string data));
          loop ()
      in
      loop ()
end

Set test_server to a web server with a self-signed X.509 certificate (one that a web-brower would reject). Test with:

mirage configure --net=socket --no-opam
make
./main.native

Expected outcome:

Conduit refuses to connect because site cannot be authenticated; OR
The code does not compile because no authenticator is specified

Actual outcome:

Conduit connects and returns the contents of the web-page

This is related to #31, but the bug here is that it silently disables the security check, which I think is a separate and more important issue than security checks not being available. i.e. this bug could be closed if it errored out instead of pretending to work.

(aside: the commented-out debug line for some reason crashes the process with Failure "string_of_sexp: atom needed", but that's another bug...)

Different behavior between nqsb-tls and openssl

Maybe I'm missing something obivous, but with CONDUIT_TLS=native the HTTP requests seems to be handled sequentially.

This is my server:

#use "topfind";;
#require "cohttp.lwt";;
let say fmt = Printf.(ksprintf (printf "%s\n%!") fmt)

let mode =
  let certfile = "_test_env/test-cert.pem" in
  let keyfile =  "_test_env/test-key.pem" in
  let port = 4242 in
  `TLS (
    (* `TLS means that conduit will do:
       match Sys.getenv "CONDUIT_TLS" with
       | "native" | "Native" | "NATIVE" -> Native
       | _ -> OpenSSL *)
    `Crt_file_path certfile,
    `Key_file_path keyfile,
    `No_password, `Port port)
open Lwt
let req_callback (_, conn_id) request body =
  let id = Cohttp.Connection.to_string conn_id in
  say "Connection %s, request %s" id
    (Cohttp.Request.sexp_of_t request |> Sexplib.Sexp.to_string);
  match Uri.path (Cohttp.Request.uri request) with
  | "/sleep" ->
    Lwt_unix.sleep 10.
    >>= fun () ->
    Cohttp_lwt_unix.Server.respond_string ~status:`OK  ~body:"slept" ()
  | other ->
    Cohttp_lwt_unix.Server.respond_string ~status:`OK  ~body:"hello" ()

let () =
  Lwt_main.run
    Cohttp_lwt_unix.Server.(
      create ~mode (make ~callback:req_callback ()))

OpenSSL

If I run CONDUIT_TLS=openssl ocaml server.ml

Then I can run curl -k https://127.0.0.1:4242/sleep on one terminal, and many times curl -k https://127.0.0.1:4242/hello on another terminal, everything is fine.

OCaml-TLS

When running CONDUIT_TLS=native ocaml server.ml, if I have curl -k https://127.0.0.1:4242/sleep on one shell, any call to curl -k https://127.0.0.1:4242/hello on another terminal will block until the /sleep is done.

trust anchors - verification of server certificate

as discussed in #23, conduit should deliver some 'default' trust anchors. how is the best way to do so? It is a list of pem-encoded certificates -- should it install such a trust anchor file (if so, how do I tell the build system to install a resource and how do I access it at run-time (how do I find out what the complete path is)?
or should we inline these pem-encoded certificates into a .ml file and compile them into the binary/shared object?

SSL Client Authentication

I would like to ask for a way to supply a certificate and key for the cohttp client (lwt), as I use SSL client authentication. I used Cohttp_lwt_unix_net.Ssl_client.sslctx up til now, but I can't find an alternative with the latest version. I can see the SSL code has been moved here, but depending on how it's solved, Cohttp may need adjustments too.

The simplest fix may be to re-expose the SSL context as a global.

An alternative to avoid modifying globals after program initialization:

Add an initialization function which picks up defaults for the global context from environment variables (e.g. CONDUIT_C_CERT, CONDUIT_C_CERTKEY, CONDUIT_S_CERT, CONDUIT_S_CERTKEY, CONDUIT_S_CACERT_FILE, ...). This can be implemented by client code if the context is exposed, which might be preferable for applications complex enough to have it's own configuration file and environment.
Add parameters to override these. I could be a single record of SSL parameters passed as ~tls:{tls_defaults with ...}. If these/this parameter is present, create a new context, otherwise use the global one.

But the simple fix suffice for my use.

support TLSv1.1 and TLSv1.2 with openssl backend

Currently ocaml-conduit explicitly requests only TLSv1, would be nice if it supported TLSv1.1, v1.2 and whatever new protocols there will be.
To do that you apparently have to request SSLv23 and explicitly disable SSLv2 and SSLv3 then.

Access to the socket file-descriptor

It would be useful to be able to set socket options; or more generally to have access (also from cohttp) to the underliying Lwt_unix.file_descr that is used with bind and listen
(like here: https://github.com/mirage/ocaml-conduit/blob/master/lib/conduit_lwt_unix.ml#L198)

PS: my use-case is about adding Lwt_unix.set_close_on_exec, if people think it's useful I can create a PR that adds it to all the "listening" file-descriptors.

inconsistent signatures (of server and client)

as a reminder from #23

why is type client a polyvar of tuples, and type server a polyvar of polyvar-tuples (esp: int vs `Port of int)?

Inconsistency; can rev that before conduit.1.0 to fit one style (unsure if polyvar-tuples or a simple record is better)

missing `vchan` to the requirements of building `Conduit_mirage`

Otherwise:

# File "lib/conduit_lwt_unix.ml", line 385, characters 19-779:
# Warning 27: unused variable ctx.
# ocamlfind ocamlc -a -thread -syntax camlp4o -package uri.services -package dns.mirage -package mirage-types -package ipaddr.unix -package lwt.unix -package lwt -package stringext -package uri -package cstruct -package ipaddr -package sexplib -package sexplib.syntax -package camlp4.macro lib/conduit_lwt_unix.cmo lib/resolver_lwt_unix.cmo -o lib/conduit-lwt-unix.cma
# ocamlfind ocamldep -ppopt -DHAVE_MIRAGE -syntax camlp4o -package uri.services -package dns.mirage -package mirage-types -package ipaddr.unix -package lwt.unix -package lwt -package stringext -package uri -package cstruct -package ipaddr -package sexplib -package sexplib.syntax -package camlp4.macro -modules lib/conduit_mirage.ml > lib/conduit_mirage.ml.depends
# ocamlfind ocamldep -ppopt -DHAVE_MIRAGE -syntax camlp4o -package uri.services -package dns.mirage -package mirage-types -package ipaddr.unix -package lwt.unix -package lwt -package stringext -package uri -package cstruct -package ipaddr -package sexplib -package sexplib.syntax -package camlp4.macro -modules lib/conduit_mirage.mli > lib/conduit_mirage.mli.depends
# ocamlfind ocamlc -c -w A-4-33-40-41-42-43-34-44 -annot -bin-annot -short-paths -principal -thread -strict-sequence -ppopt -DHAVE_MIRAGE -syntax camlp4o -package uri.services -package dns.mirage -package mirage-types -package ipaddr.unix -package lwt.unix -package lwt -package stringext -package uri -package cstruct -package ipaddr -package sexplib -package sexplib.syntax -package camlp4.macro -I lib -o lib/conduit_mirage.cmi lib/conduit_mirage.mli
# + ocamlfind ocamlc -c -w A-4-33-40-41-42-43-34-44 -annot -bin-annot -short-paths -principal -thread -strict-sequence -ppopt -DHAVE_MIRAGE -syntax camlp4o -package uri.services -package dns.mirage -package mirage-types -package ipaddr.unix -package lwt.unix -package lwt -package stringext -package uri -package cstruct -package ipaddr -package sexplib -package sexplib.syntax -package camlp4.macro -I lib -o lib/conduit_mirage.cmi lib/conduit_mirage.mli
# File "lib/conduit_mirage.mli", line 28, characters 18-30:
# Error: Unbound module Vchan
# Command exited with code 2.