timeit loop? about spork HOT 13 CLOSED

True, there is some room for some more functionality in the ev library here - both the ability to Send interrupts to other threads as well as just call pthread_cancel on them. Unfortunately this has some issues with memory management so we may not support pthread_cancel directly

from spork.

We can have both implementations in the spork, as someone can use the current version.

from spork.

Tried it out a bit:

  (do
    (def ds @{})
    (timeit-loop [:repeat 100_000] :label
      (put ds :a 1))
    ds)
  # printed: label 0.008s, 0.08157µs/body
  # =>
  @{:a 1}

  (do
    (def ds @{})
    (timeit-loop [:repeat 100_000] :label
      (set (ds :a) 1))
    ds)
  # printed: label 0.007s, 0.07186µs/body
  # =>
  @{:a 1}

  (timeit-loop [:timeout 1] :label
    (get {:a 1} :a))
  # printed: label 1.000s, 0.06712µs/body
  # =>
  nil

Looks like it's working :)

seems like it could be avoided using an os/sleep in a separate thread?

I'm not sure, but may be at 0.03µs, it's better to keep the implementation simple at least initially?

from spork.

Cleaned up slightly:

(defmacro timeit-loop
  ``Similar to `loop`, but outputs performance statistics after completion.``
  [head & body]
  (def tag "Elapsed time:")
  (with-syms [c head2 start elapsed per-body _]
    (def head2 @[;head])
    (when (def i (index-of :timeout head2))
      (array/insert head2 i _)
      (set (head2 (+ i 1)) :iterate)
      (set (head2 (+ i 2)) ~(< (- (os/clock) ,start) ,(in head2 (+ i 2)))))
    ~(do
       (var ,c 0)
       (def ,start (os/clock))
       (loop ,head2 (++ ,c) ,;body)
       (def ,elapsed (- (os/clock) ,start))
       (def ,per-body (/ ,elapsed ,c))
       (cond
         (< ,per-body 1e-3) (printf "%s %fs, %.4gµs/body" ,tag ,elapsed (* ,per-body 1_000_000))
         (< ,per-body 1) (printf "%s %fs, %.4gms/body" ,tag ,elapsed (* ,per-body 1_000))
         (printf "%s %fs, %.4gs/body" ,tag ,elapsed ,per-body)))))


(def ind (range 1000))
(timeit-loop [:timeout 2]
  (take -500 ind))
# ->
Elapsed time: 2.000049s, 3.275µs/body

Two things I'd like to modify:

• add a :timeout verb to prematurely break after a given number of seconds has elapsed
• make tag somehow configurable, as it is with timeit

I'm not quite sure what the best approach is to achieve either of these. Suggestions welcome.

from spork.

No good ideas about :timeout.

Some hacky ideas for tag:

1. Dynamic variable
2. Scan first part of body for a keyword to use as a tag
3. Similar to previous idea, but use dictionary instead -- it can contain a key-value pair indicating a tag

For 2. and 3., having keywords or dictionaries at the beginning of a body of code usually doesn't mean anything (unless in the dictionary there is something side-effecty).

from spork.

No good ideas about :timeout.

Patching it for an :iterate verb seems to work.

I suppose if the first form of the body is a bytes?, it could be assumed to be the tag?

from spork.

Patching it for an :iterate verb seems to work.

Nice.

I suppose if the first form of the body is a bytes?, it could be assumed to be the tag?

That sounds like it might work [1].

If you ever wanted to support more / other things, a struct (no point in it being a table I guess) could be checked for perhaps.

[1] This kind of approach has been working for me in other places. One example is for "tagging" jpm tasks. There I look for a keyword followed by a tuple -- after (task "name" [...]. timeit-loop's case is safer because you control it :)

from spork.

That sounds like it might work [1].

I think it works great.

(defmacro timeit-loop
  ``Similar to `loop`, but outputs performance statistics after completion.
  Additionally defines a `:timeout` verb to iterate continuously for a given
  number of seconds. If the first form of `body` is a bytes, it will be taken
  as a custom tag.``
  [head & body]
  (var tag "Elapsed time:")
  (def head2 @[;head])
  (def body2 @[;body])
  (with-syms [c start elapsed per-body]
    (when (def i (index-of :timeout head2))
      (array/insert head2 i [])
      (set (head2 (+ i 1)) :iterate)
      (set (head2 (+ i 2)) ~(< (- (os/clock) ,start) ,(in head2 (+ i 2)))))
    (when (bytes? (get body2 0))
      (set tag (in body2 0))
      (array/remove body2 0))
    ~(do
       (var ,c 0)
       (def ,start (os/clock))
       (loop ,head2 (++ ,c) ,;body2)
       (def ,elapsed (- (os/clock) ,start))
       (def ,per-body (/ ,elapsed ,c))
       (cond
         (< ,per-body 1e-3) (printf "%s %.3fs, %.4gµs/body" ,tag ,elapsed (* ,per-body 1_000_000))
         (< ,per-body 1) (printf "%s %.3fs, %.4gms/body" ,tag ,elapsed (* ,per-body 1_000))
         (printf "%s %.3fs, %.4gs/body" ,tag ,elapsed ,per-body)))))

(timeit-loop [:repeat 200_000_000] ":repeat ")
(timeit-loop [:timeout 1] ":timeout")
# ->
:repeat  1.043s, 0.005214µs/body
:timeout 1.000s, 0.03572µs/body

About the :timeout, the os/clock adds about 0.03µs of overhead, which probably isn't a big deal for the purpose of comparison, but also seems like it could be avoided using an os/sleep in a separate thread?

from spork.

I'm not sure, but may be at 0.03µs, it's better to keep the implementation simple at least initially?

Actually, ev/deadline or ev/with-deadline seem like they were intended for precisely this sort of thing. I can't figure out the proper usage, though.

(ev/with-deadline 1 (forever nil))

This never terminates.

from spork.

Similar situation here.

Replacing (forever nil) with (os/sleep 3) seems to terminate though not sure what the responsible mechanism is there.

Admittedly, not so helpful for the intended use-case.

from spork.

This almost does the job:

(var c 0)
(def f (ev/spawn (forever (++ c))))
(os/sleep 1)
(ev/cancel f :timeout)

the counting var from f would need to be propagated somehow, though.

from spork.

Janet has a way to interrupt the interpreter, but it's not accessible from Janet, only from C with janet_interrupt. You could insert (ev/sleep 0) every turn of the loop but that would have unacceptable perf cost for timing.

Ideal we could spawn a new thread, sleep for some time, and then set the interrupt flag in the original interpreter.

from spork.

The task doesn't seem so complicated, though.

1. Start a thread.
2. Terminate it after n seconds.

Everything I tried either fails to halt the thread, or terminates with a seemingly unrecoverable error.

from spork.