samtools parser error when stdout output into samtools view about streammd HOT 2 CLOSED

Not observed when writing to file; a SAM file created by streammd is well-formed according to samtools view.
Also not observed when running with a single worker (consumer) process.

from streammd.

The issue was a combination of finite buffer size (Linux system default=8192) and having each of the worker processes writing to stdout simultaneously. When the size of the output chunk exceeds the buffer size (hence the dependence on batchsize) samtools view gets an incomplete line in the input. With multiple writers to stdout there's no guarantee that what follows is the remainder of the first incomplete chunk. (But when output is redirected to a file the parallel writes are correctly sequenced, and all is well... the kernel is clever?)
Some possible solutions:

1. increase buffer size so that it's larger than any potential batch size
2. use an output queue

1. is less work but more fragile and smelly
2. is a little more work, smells better, query on performance

from streammd.