documentation, what is missing ? about missionary HOT 25 OPEN

Hi,

I have added a walkthrough "à la core.async" focused on Missionary Clojure implementation. The last example about flow is an implementation of the example from the core.async walkthrough using alts!!.

I have added a cheatsheet "à la specter", for the moment only about tasks and flows (I dont understand yet how to properly use dfv, mbx, sem, etc)

from missionary.

This partially relates to documentation and partially to naming, but I find that runes and acronyms make it harder to grok the code, examples and documentation. Until I'm fluent, I have to keep reminding myself, sp is sequential process, ap is asyncrhonous-process, not to mention all the variations on pulling and forking. If they had clearer names I think it would go a long way towards making the code more approachable to new arrivals.
There are also some fundamental concepts such as forking which could be clarified.
Also, for people who learn more by following things up from first principles than by example, it could be beneficial to create a continuity between the task and flow specification to the primitives in the library.
This issue might be partially related, too leonoel/flow#1

from missionary.

AFAIK the creator of that documentation structure moved it to https://diataxis.fr/

from missionary.

Thank you all, lots of relevant ideas here.

Regarding the overall structure of documentation for any software project, I've come across this pattern which has been applied to several high profile projects: https://documentation.divio.com/ (The video is worth watching). I believe adopting this pattern would help both beginners and experienced users well.

AFAIK the creator of that documentation structure moved it to https://diataxis.fr/

A cookbook (called how-to guides above) is what I miss the most. The current 3 guides are nice, but more, shorter examples of common patterns would help me personally.

How about we start with a couple of wiki pages? Specter has this nice list of navigators that I visit often and find valuable and short explanations with simple examples. A second page could be a cookbook of patterns.

diataxis is a good framework and I want to endorse it, here are my thoughts :

• tutorials : I think the current tutorials are decent but we need more.
• how-to guides : I like the idea of using the github wiki as a cookbook. I've moved the 3 existing guides and set public write access, feel free to contribute !
• reference : I think the docstrings are the right place for it, especially in combination with cljdoc to improve reading experience. The contents need refreshing and elaboration, I'm going to make a global pass at some point.
• discussions / explanations : we have none yet but I plan to write some.

IMHO something like this would suffice, touching on all the features available and how to use them.
https://github.com/clojure/core.async/blob/master/examples/walkthrough.clj

Some of the primitives (e.g. in happy eyeballs race of dfv) are not part of the documentation, which makes happy eyeballs harder to understand.

I'm not sure if it's a good fit for a tutorial, it would probably be rather long and I see more value in a synthetic reference. I'd like to have some kind of cheatsheet, however it's still unclear to me what it should look like.

A walkthrough of some common workflows with ClojureScript, e.g. subscribe to DOM events, start/end flows etc.

How to create a Task and maybe an example on how to interop with a js callback api.

Converting core.async channel to flow

Converting promise to task

Consuming a BlockingQueue

CompletableFuture to task

File IO

HTTP

To build off this point a bit, I think it can be generalized to the "edges of the system".

How can things be turned to tasks and flows?

How do I start a flow which does some IO?

These are definitely good fits for the cookbook.

However, for someone who feels reasonably proficient in core.async, it's like colliding with a wall. Everything is different, and it's a little hard to translate things, to missionary.

I still struggle understanding how will my code run, most importantly when will it block and what to do when I don't want a task to block. I think it's typically with tasks, flows are easier on this front.

There are also some fundamental concepts such as forking which could be clarified.

forking

discrete vs. continuous flow

back pressure

My current plan for explanation pages, roughly :

• imperative vs functional : switching mindset
• execution model : which thread runs my code
• forking : how it works, how to reason about it
• discrete vs continuous : back pressure and lazy sampling

reactor

Yes. I have a WIP tutorial on the reactor.

from missionary.

A great thing to add to the documentation is what is functional effect and streaming system? why use functional effects?
It could be on separate page or inlined at the top of the main page. These questions overlap with the documentation of tasks maybe all that is needed is a bridge from the words functional effect and task and a bit more why.

My answer would be something like below.
A side effect is when a function relies on, or modifies, something outside its parameters to do something.
For example: (println "Hello world!")
A functional effect is value that describes an effect, without actually executing it. In missionary we represent function effects using tasks.
For example creating a sequential process for the effect hello world (def hello-world-task (sp (println "Hello world!")))
now will not print to the console until the task is ran
(m/? hello-world-task)
will run the task hello-world-task which will print "Hello world!" to the console.
Using functional effects yields many benefits including improvement to testability and composability.

from missionary.

Some ideas:

• A walkthrough of some common workflows with ClojureScript, e.g. subscribe to DOM events, start/end flows etc.
• A list of the various macros m/? m/ap and so on with descriptions of what they do, CLJ/S compatibility, and a "how to read them", i.e. what words to use to describe them

IMHO something like this would suffice, touching on all the features available and how to use them.
https://github.com/clojure/core.async/blob/master/examples/walkthrough.clj

There are some tools (like transcriptor) to turn REPL sessions into tests, maybe that'd be a good format for this.

from missionary.

Regarding the overall structure of documentation for any software project, I've come across this pattern which has been applied to several high profile projects: https://documentation.divio.com/ (The video is worth watching). I believe adopting this pattern would help both beginners and experienced users well.

from missionary.

The documentation is good, but only for someone who catches the concept.

However, for someone who feels reasonably proficient in core.async, it's like colliding with a wall. Everything is different, and it's a little hard to translate things, to missionary.

I don't have a problem with reading tests, but in this case they are not always readable either, especially since you first have to understand how the underlying macros work.

IMHO something like this would suffice, touching on all the features available and how to use them.
https://github.com/clojure/core.async/blob/master/examples/walkthrough.clj

from missionary.

A cookbook (called how-to guides above) is what I miss the most. The current 3 guides are nice, but more, shorter examples of common patterns would help me personally.

@leonoel any reason you're asking for suggestions? Did someone complain or do you find the documentation lacking?

from missionary.

any reason you're asking for suggestions? Did someone complain or do you find the documentation lacking?

Both. A significant part of the library is not covered at all by current docs, several users have asked for more resources, and the learning curve feels steep. Also I'd like to understand better what newcomers struggle with.

from missionary.

I still struggle understanding how will my code run, most importantly when will it block and what to do when I don't want a task to block. I think it's typically with tasks, flows are easier on this front.

Some of the primitives (e.g. in happy eyeballs race of dfv) are not part of the documentation, which makes happy eyeballs harder to understand.

How about we start with a couple of wiki pages? Specter has this nice list of navigators that I visit often and find valuable and short explanations with simple examples. A second page could be a cookbook of patterns.

from missionary.

How to create a Task and maybe an example on how to interop with a js callback api.

from missionary.

I think it would be useful to have more examples of how to integrate missionary with existing libraries and tools (in addition to the current documentation on integrating with RxJava):

Some examples:

• Converting core.async channel to flow
• Converting promise to task
• Consuming a BlockingQueue
• CompletableFuture to task
• File IO
• HTTP

It might be worth adding a wiki or something similar for managing and working on these types of snippets and examples.

from missionary.

I think it would be useful to have more examples of how to integrate missionary with existing libraries and tools (in addition to the current documentation on integrating with RxJava):

Some examples:

• Converting core.async channel to flow
• Converting promise to task
• Consuming a BlockingQueue
• CompletableFuture to task
• File IO
• HTTP

It might be worth adding a wiki or something similar for managing and working on these types of snippets and examples.

To build off this point a bit, I think it can be generalized to the "edges of the system".
How can things be turned to tasks and flows? How do I start a flow which does some IO?

This example from a slack thread is a helpful learning aid, not just useful for my particular case.

(defn poll [^KafkaConsumer consumer]
  (m/via m/blk (.poll consumer Long/MAX_VALUE)))

(defn forever [task]
  (m/ap (m/? (m/?> (m/seed (repeat task))))))

(->> (forever (poll consumer))
  (m/eduction (comp cat (map -value)))
  (m/reduce (fn [_ x] (println 'consumed x)) nil)
  m/?)

from missionary.

Trying to see if I've been paying attention in class, CompletableFuture to task should look something like:

(import '(java.util.concurrent Executor CompletableFuture)
        '(missionary.impl Thunk))

(defn cf->task
  [^CompletableFuture cf]
  (fn [success failure]
    (.handleAsync
     cf
     (reify java.util.function.BiFunction
       (apply [_ r e]
         (if (instance? Exception e)
           (failure e)
           (success r))))
     Thunk/cpu)
    (fn [] (.cancel cf true))))

(let [cf (CompletableFuture.)
      t (cf->task cf)
      success! (fn [res] (println 'yay! res))
      fail! (fn [e] (println 'Error! (ex-message e)))]
  (t success! fail!)
  (.complete cf 2))


(let [cf (CompletableFuture.)
      t (cf->task cf)
      success! (fn [res] (println 'yay! res))
      fail! (fn [e] (println 'Error! (ex-message e)))]
  (t success! fail!)
  (.completeExceptionally cf (Exception. "failed!")))

from missionary.

Other terms which are unclear after rereading the documentation and specification, or which just aren't clear how one should work with

• discrete vs. continuous flow
• back pressure
• forking
• reactor
• introducing concurrency into a flow

from missionary.

A list of the various macros m/? m/ap and so on with descriptions of what they do, CLJ/S compatibility, and a "how to read them", i.e. what words to use to describe them

This partially relates to documentation and partially to naming, but I find that runes and acronyms make it harder to grok the code, examples and documentation. Until I'm fluent, I have to keep reminding myself, sp is sequential process, ap is asyncrhonous-process, not to mention all the variations on pulling and forking. If they had clearer names I think it would go a long way towards making the code more approachable to new arrivals.

I opened a separate issue, #44

from missionary.

@leonoel

I have a WIP tutorial on the reactor

could you share it even in its raw state?

from missionary.

@kawas44 Thank you, these resources are highly valuable. I have some remarks about the walkthrough :

• (m/ap (for [i (range 4)] (m/?> i))) this is invalid code (because i is not a flow, and because m/?> is not allowed in closures)
• in the last example, the ap block is actually fully synchronous and single threaded. We could make it async using via, the execution would be closer to what happens with core.async.

(let [begin (System/currentTimeMillis)
      ;; create a flow of values generated by asynchronous tasks
      inputs (repeat 1000 (m/via m/cpu "hi")) ;; a task has no identity, it can be reused
      values (m/ap
               (let [flow (m/seed inputs)     ;; create a flow of tasks to execute
                     task (m/?= flow)]        ;; from here, fork on every task in **parallel**
                 (m/? task)))                 ;; get a forked task value when available

      ;; drain the flow of values and count them
      n (m/? ;; tasks are executed, and flow is consume here!!
         (m/reduce (fn [acc v]
                     (assert (= "hi" v))
                     (inc acc))
                   0 values))]
  (println "Read" n "msgs in" (- (System/currentTimeMillis) begin) "ms"))

from missionary.

• (m/ap (for [i (range 4)] (m/?> i))) this is invalid code (because i is not a flow, and because m/?> is not allowed in closures)

Oups, didn't try to reduce it 😬
I can fix it with amb> which I understood while writing the cheatsheet :)

(m/ap (println (m/seed [1 2])))

EDIT: Changed the m/ap snippet above to use println instead of m/amb> to avoid introducing this macro early in the walkthrough.

• in the last example, the ap block is actually fully synchronous and single threaded. We could make it async using via, the execution would be closer to what happens with core.async.

Yes I knew it run on the same thread. I am still a little bit lost on the vocabulary. Maybe "concurrent" would have been a better word. Anyway, if you dont mind, I will copy/paste your version.

from missionary.

IMO any/all tutorials need to work in ClojureScript (so m/? outside process block should not be in the tutorial, instead describe the task interface and how to run it directly)

(def task (fn [success! failure!] (success! 1) (fn cancel! [])))
(def cancel! (task #(prn ::success %) #(prn ::failure %)))

also see https://github.com/leonoel/task

from missionary.

@dustingetz makes sense to me, it also prevents a potential confusion between parking m/? and blocking m/?.

from missionary.

hello task suffers the same issue btw. It even mentions the task works in cljs (which it does), but the blocking m/? call is misleading there too

from missionary.

@xificurC

could you share it even in its raw state?

I'd like to make a step-by-step guide to solve the petrol pump problem popularized by the sodiumFRP team.

My current implementation relies on unreleased features, you'll need to compile the master branch locally.

from missionary.

(tests
  "Missionary signals"
  (def !x (atom 0))                                         ; atoms model variable inputs
  (def >x (m/watch !x))                                     ; "recipe" for a signal derived from atom

  ; a signal is a "continuous flow" in Missionary jargon
  ; signals (flows) are recipes, like Haskell IO actions.
  ; Like IO actions, they are pure function values (thunks)
  ; and do not perform side effects until you run them.
  (fn? >x) := true                                          ; thunk (implementation detail)
  ; The atom has not been subscribed to yet, because >x is a pure value

  ; Flow thunks concretely have the structure (fn [notify! terminate!] !iterator),
  ; see https://github.com/leonoel/flow#specification
  (def !it (>x (fn [] (! ::notify))
               (fn [] (! ::terminate))))
  % := ::notify                                             ; lazy flow is ready to be sampled
  @!it := 0                                                 ; sample
  (swap! !x inc)                                            ; trigger a change
  % := ::notify                                             ; flow is ready again
  @!it := 1                                                 ; sample
  )

from missionary.