odis-labs / streaming Goto Github PK

version: 0.8.0

https://odis-labs.github.io/streaming/streaming/Streaming/Stream/index.html

val partition : int -> 'a t -> 'a t t
partition n partitions the stream into sub-streams of size n.

#require "streaming";;
open Streaming.Stream;;

of_array [|0; 1; 2; 3; 4; 5; 6; 7; 8; 9|]
|> partition 10
|> map (fun x -> x |> to_list)
|> to_list;;
- : int list list = []

of_array [|0; 1; 2; 3; 4; 5; 6; 7; 8; 9|]
|> partition 9
|> map (fun x -> x |> to_list)
|> to_list;;
- : int list list = [[0; 1; 2; 3; 4; 5; 6; 7; 8]; [9]]

of_array ([|0; 1; 2; 3; 4; 5; 6; 7; 8; 9; 10; 11; 12; 13; 14; 15; 16; 17; 18; 19|])
|> partition 10
|> map (fun x -> x |> to_list)
|> to_list;;
- : int list list = [[0; 1; 2; 3; 4; 5; 6; 7; 8; 9]]

How would you feel about having a filter_map: ('a -> 'b option) -> 'a t -> 'b t function in Stream and Flow ? It's not really difficult, something along the line of :

let filter_map f =
  let flow (Sink k) =
    let push r x = match f x with Some x -> k.push r x 
                                | None -> r in
    Sink { k with push } in
  { flow }

(Based on Flow.filter implementation).

I really like this function, sometimes it can avoid some awkward assumptions when composing filter and map - for example I find that :

let map_inverse l =  List.filter ((<>) 0.) l |> List.map (fun x -> 1. /. x)

less elegant than that :

let map_inverse = List.filter_map (fun x -> if x <> 0. then Some (1. /. x) else None)

Because in the first version, we have to assume that map takes a list that contains only non-zero number.

How about renaming of_file to something like file_lines (which conveys its meaning more explicitly), and letting of_file return just chunks read by input to enable other segmentations/primary parsing?

https://odis.io/docs/streaming/Streaming/Flow/index.html#val-into

Flow "through" better reflects the fact that the provided sink will be run multiple times.