Default Buffer Size, iota Choking about iota HOT 5 CLOSED

Hi Tom,

Thank you for your curiosity and providing such detailed information above. You are correct that iota/seq's performance without reducers is quite poor, and the iota readme does not reflect that. When the readme states "Tuned for Clojure's reducers", I meant to imply that it's coded for the benefit of multi-threaded code via clojure's reducers library.
If Clojure's reducers aren't being used, I suspect Clojure's line-seq will work much better than iota/seq.

Using line-seq as a baseline;

(time (count (line-seq (clojure.java.io/reader  testfile))))
"Elapsed time: 16685.486074 msecs"
4281565

With iota/seq (on 8 core desktop);

 (time (->> (iota/seq testfile)
      (clojure.core.reducers/map  (fn [_] 1))
      (clojure.core.reducers/fold +)))
"Elapsed time: 8925.41084 msecs"
4281565

With iota/seq (above code, but simulating 2 cores);

$ taskset 0x3 java -jar target/validator-0.1.0-SNAPSHOT-standalone.jar
"Elapsed time: 10687.774363 msecs"

With iota/seq (above code, but simulating 1 core);

$ taskset 0x1 java -jar target/validator-0.1.0-SNAPSHOT-standalone.jar
"Elapsed time: 17309.757736 msecs"

The above was using a test file created via the following;

(defn records->file [n filename]
  (with-open [fh (clojure.java.io/writer  filename)]
    (binding [*out* fh]
      (doseq [rec (take n (repeatedly fake-record))]
        (->> (fake-record)
             vals
             (map str)
             (interpose "\t")
             (apply str)
             println)))))

(records->file 4281565 testfile)

I hope this helps, and I'll update the readme shortly.

from iota.

Got it. Thanks for clarifying. Out of curiosity, what size heap were you using in your tests?

from iota.

Didn't use "-Xmx1g" on tests, per above, but I get the same results with it.

$ taskset 0x3 java -Xmx1g -jar target/validator-0.1.0-SNAPSHOT-standalone.jar
"Elapsed time: 9813.673496 msecs"

from iota.

That's interesting. I'm running your fold example and I've gone well over the ten minute mark (on my paltry laptop). Looks like I'm in gc purgatory. Like before, I can get the fold to complete if I reduce the buffer size to 1024 *4, on the order of several minutes.

from iota.

I'm probably dealing with a specific constraints and maybe the overhead introduced by fork/join. I'll take it that I'm an outlier :) I think for the vast majority of use cases, iota is probably fine as is. Again, thanks for publishing iota.

from iota.