inhabitedtype / faraday Goto Github PK

View Code? Open in Web Editor NEW

133.0 133.0 20.0 279 KB

Serialization library built for speed and memory efficiency

License: Other

Makefile 0.59% OCaml 99.41%

ocaml serialization

faraday's People

Contributors

Stargazers

Watchers

faraday's Issues

How to serialise a user-defined type?

Could you expand the README to explain the scope of the serialiser? Looking at the interface it is not obvious how to serialise user-defined types. Maybe this is not even a design goal. In any case, I think the README could address this.

Eliminate dependency on ocplib-endian

Faraday already requires an OCaml version after 4.02.0, which includes all the bigarray intrinsics that are needed to efficiently implement little- and big-endian blitting operations. Eliminate the dependency and just use the intrinsics directly.

The uint48 functions are both broken

(reported against Eio in ocaml-multicore/eio#418, which contains fixes and tests)

faraday/lib/faraday.ml

Lines 290 to 297 in c66fc02

 let write_uint48 t i = 

 writable_exn t; 

 ensure_space t 6; 

 Bigstringaf.unsafe_set_int32_be t.buffer t.write_pos 

 Int64.(to_int32 (shift_right_logical i 4)); 

 Bigstringaf.unsafe_set_int16_be t.buffer (t.write_pos + 2) 

 Int64.(to_int i); 

 t.write_pos <- t.write_pos + 6

@adatario noticed that the shifts should be in bits, not bytes.

Also, the BE version gets the sizes of the chunks wrong (should be 16 then 32).

Assertion when scheduling several strings to a small buffer

I'm still trying to minimize the steps and narrow down what's going on, but from a utop session the following will fail with Exception: Assert_failure ("lib/faraday.ml", 477, 4). using the faraday 0.3.0 from opam:

let f = Faraday.create 1;;
Faraday.schedule_string f "hello world";;
Faraday.schedule_string f "hello world";;
Faraday.schedule_string f "hello world";;
Faraday.schedule_string f "hello world";;
Faraday.serialize_to_string f;;

The real error came from a larger buffer with a lot more input. I may also be using the library very incorrectly.

is_empty

Hi - is there a simple way of testing if a Faraday.t is empty? Thanks

Writing more bytes than the buffer can hold raises Invalid_argument

Comes from the blit call in write_gen.

Writing can raise `Failure _`

When writing to a closed Faraday.t an exception Failure "cannot write to closed writer" is raised. I am observing this exception from Httpaf.Body.write_string, and I'm wary of catching all Failures. Could a more precise exception be used?

faraday/lib/faraday.ml

Lines 193 to 195 in c66fc02

 let writable_exn t = 

 if t.closed then 

 failwith "cannot write to closed writer"

assert failure faraday.ml:377 using Async

I'm trying to use faraday with Async like so:

open Async

let handle fd =
  let ic = Reader.create fd in
  let writev = Faraday_async.writev_of_fd fd in
  let output_buffer = Faraday.create 16 in
  let write_output x =
    Faraday_async.serialize output_buffer
      ~yield:(fun buffer -> my_serialize_function buffer x |> return)
      ~writev
  in
  Angstrom_async.parse_many parse_function
    (fun parsed -> parsed |> my_processing_function |> write_output)
    ic
  >>| Result.get_ok

let create_server port =
  Tcp.Server.create_sock_inet ~on_handler_error:`Raise
    (Tcp.Where_to_listen.of_port port) (fun _address socket ->
      handle (Socket.fd socket))

but I get this error any time I try to connect:

((monitor.ml.Error "Assert_failure lib/faraday.ml:377:4"
    ("Raised at file \"async/faraday_async.ml\", line 30, characters 6-15"
     "Called from file \"src/deferred1.ml\", line 17, characters 40-45"
     "Called from file \"src/job_queue.ml\", line 167, characters 6-47"
     "Caught by monitor Tcp.Server.create_sock_inet"))

Any idea why this might be happening?

Double precision floating point values encoded as single precision

Small typo - PR incoming.

Don't allocate a new buffer on every flush

Introduced in #19 to fix a pretty bad bug. Before the next release, eliminate the allocation on every flush, allowing the library to use additional free space in the buffer across flushes. As part of this fix, ensure that write_pos and schedule_pos are reset to 0 when they're equal and the flush queue is empty.

Faraday in Eio (some bugs)

Hi, I needed a buffered writer in Eio, and I used Faraday as the basis of that (ocaml-multicore/eio#235). While reviewing the code, I fixed a few things that might want fixing here too.

Overflow

There is a comment in shift_flushes saying that it avoids overflow problems by subtracting both sides:

faraday/lib/faraday.ml

Lines 392 to 410 in 3ea082b

 | (threshold, f) as flush -> 

 (* Edited notes from @dinosaure: 

  * 

  * The quantities [t.bytes_written] and [threshold] are always going to be 

  * positive integers. Therefore, we can treat them as unsinged integers for 

  * the purposes of comparision. Doing so allows us to handle overflows in 

  * either quantity as long as they're both within one overflow of each other. 

  * We can accomplish this by subracting [min_int] from both quantities before 

  * comparision. This shift a quantity that has not overflowed into the 

  * negative integer range while shifting a quantity that has overflow into 

  * the positive integer range. 

  * 

  * This effectively restablishes the relative difference when an overflow 

  * has occurred, and otherwise just compares numbers that haven't 

  * overflowed as similarly, just shifted down a bit. 

  *) 

 if t.bytes_written - min_int >= threshold - min_int 

 then begin f reason; shift_flushes t ~reason end 

 else Flushes.enqueue_front flush t.flushed

This doesn't work; it just shifts the problem elsewhere (consider bytes_written = -2 and threshold = 8, for example). I believe the test should be t.bytes_written - threshold >= 0. /cc @dinosaure

Safety

write_string and write_bytes both perform unsafe blits using offsets provided by the user:

faraday/lib/faraday.ml

Lines 250 to 258 in 3ea082b

 let write_string = 

 let length = String.length in 

 let blit = Bigstringaf.unsafe_blit_from_string in 

 fun t ?off ?len a -> write_gen t ~length ~blit ?off ?len a 

 let write_bytes = 

 let length = Bytes.length in 

 let blit = Bigstringaf.unsafe_blit_from_bytes in 

 fun t ?off ?len a -> write_gen t ~length ~blit ?off ?len a

Serialize loop

If the user's writev function returns Closed then it can't consume anything and just calls it again, forever:

faraday/lib/faraday.ml

Lines 436 to 444 in 3ea082b

 let rec serialize t writev = 

 match operation t with 

 | `Writev iovecs -> 

 begin match writev iovecs with 

 | `Ok n -> shift t n; if not (Buffers.is_empty t.scheduled) then yield t 

 | `Closed -> close t 

 end; 

 serialize t writev 

 | (`Close|`Yield) as next -> next

License

BTW, Faraday uses the BSD-3-clause license, whereas Eio uses ISC. I included your headers and license in Eio, but if you're willing to let us use it as ISC it would simplify things slightly.

dune-project is missing

May be a bug in dune but it seems that if we use faraday as a vendor (available into a dune workspace of another project), dune is not able to compile anything due to a missing dune-project.

As angstrom, distribution of faraday should have a dune-project to be able to use it as a vendor library.

PS: I don't really know the real issue, hwever :/

	let write_uint48 t i =
	writable_exn t;
	ensure_space t 6;
	Bigstringaf.unsafe_set_int32_be t.buffer t.write_pos
	Int64.(to_int32 (shift_right_logical i 4));
	Bigstringaf.unsafe_set_int16_be t.buffer (t.write_pos + 2)
	Int64.(to_int i);
	t.write_pos <- t.write_pos + 6

	let writable_exn t =
	if t.closed then
	failwith "cannot write to closed writer"

	\| (threshold, f) as flush ->
	(* Edited notes from @dinosaure:
	*
	* The quantities [t.bytes_written] and [threshold] are always going to be
	* positive integers. Therefore, we can treat them as unsinged integers for
	* the purposes of comparision. Doing so allows us to handle overflows in
	* either quantity as long as they're both within one overflow of each other.
	* We can accomplish this by subracting [min_int] from both quantities before
	* comparision. This shift a quantity that has not overflowed into the
	* negative integer range while shifting a quantity that has overflow into
	* the positive integer range.
	*
	* This effectively restablishes the relative difference when an overflow
	* has occurred, and otherwise just compares numbers that haven't
	* overflowed as similarly, just shifted down a bit.
	*)
	if t.bytes_written - min_int >= threshold - min_int
	then begin f reason; shift_flushes t ~reason end
	else Flushes.enqueue_front flush t.flushed

	let write_string =
	let length = String.length in
	let blit = Bigstringaf.unsafe_blit_from_string in
	fun t ?off ?len a -> write_gen t ~length ~blit ?off ?len a

	let write_bytes =
	let length = Bytes.length in
	let blit = Bigstringaf.unsafe_blit_from_bytes in
	fun t ?off ?len a -> write_gen t ~length ~blit ?off ?len a

	let rec serialize t writev =
	match operation t with
	\| `Writev iovecs ->
	begin match writev iovecs with
	\| `Ok n -> shift t n; if not (Buffers.is_empty t.scheduled) then yield t
	\| `Closed -> close t
	end;
	serialize t writev
	\| (`Close\|`Yield) as next -> next

inhabitedtype / faraday Goto Github PK

faraday's People

Contributors

Stargazers

Watchers

Forkers

faraday's Issues

Overflow

Safety

Serialize loop

License

Recommend Projects

Recommend Topics

Recommend Org