Feature suggestion: real time command output using streams about zx HOT 12 CLOSED

@biels Yes. It would be a tree of streams that efficiently pipe data from stdout to stdin. That is what Node Streams is patterned after, pipes, pehaps even UNIX pipes. There is this wonderful opportunity here to parse the command line and make it come to life within node.

The main argument against the brilliant $`command1 | command2` is the same argument that is made for Node Streams, read chunk at a time, instead of cramming the whole thing into memory.

I think antonmedv did an amazing job, this is not a critique, I must admit, I think this is outside of the scope of this program.

I am simply saying parse the commands with peg.js build a child_process stream tree and allow for incredible complexity. Complexity and functionality currently obscured by node just sending a string to /bin/sh and then sitting there and waiting until a potentially massive chunk of text is returned.

So it is not just a single child_proces that wraps the entire command, it is a thing that interprets the command line, and creates a child_proces tree. It will work for simple commands like $`tar -cvf bork.tar bork` but also for freakishly complex things that reroute output streams, run as a daemon, respond to something like UNIX's nice(1) command, and enhance the operating system with node.

await $`cat package.json | grep name`

Will actually start two
child_processes one is cat and the other grep, cat will send things to grep chunk at a time, by relying on Node Streams.

The next step to this would be /bin/zxsh a zx based Bash replacement. Which will not be possible to make if we just send strings at /bin/sh

And the step after that would be using Electron as a replacement for X Windows, perhaps with https://nextapps-de.github.io/winbox/

and then antonmedv could release a https://www.linuxfromscratch.org/ called ANTONUX

And we would all end up using it.

My instinct says that

let name = await $`cat package.json | grep name`;

obscures powerful things, that if we can get access too we can create an alternative to https://www.npmjs.com/package/pm2 alternative to Bash, and even mess around with electron as a GUI something that could potentially be as tiny as http://www.damnsmalllinux.org/

The sending a string to /bin/sh and waiting for one back obscures powerful functionality.

antonmedv your work is perfect, zx is awesome, this is not a critique, but a discussion of a potential future of this program.

You started something, that could develop legs.

Way to go!

from zx.

@catpea What about:

await $`cat package.json`.pipe($`grep name`)

let name = await $`cat package.json`.pipe($`grep name`); // works since `.pipe` returns its argument

The main idea is to transform Promise<ProcessOutput> into DuplexStream & PromiseLike<ProcessOutput> and rely on .pipe. This is significantly easier to implement than using pegjs, while increasing the awesomeness of pipes:

const r = fs.createReadStream('file.txt');
const w = fs.createWriteStream('file.txt.gz');
await r.pipe($`gzip`).pipe(w);

Pros:

• Easy to implement (redirect duplex input to stdin and stdout to duplex output, and add a .then function for await to work)
• User-friendly
• Might not break the traditional | (await $`cat package.json | grep name`;)
• Seamless integration with any node library using streams.
• Improve support for Windows

Cons:

• May affect performance (but the user can always fall back on | in this case)
• Node specific feature (but fs and os are already node only)

from zx.

Now zx support real time output!

from zx.

This is excellent! A lot of other great things have been added since I opened this issue as well. Thank you for bringing this library to such a high standard.

from zx.

@catpea What you are suggesting would be quite involved to implement but I think you are on to something.

from zx.

I think this is out of the scope of the zx project. Closing.

UPDATE 2022-05-31

Now zx supports real time output as well.

Pipes can be used to show real-time output of programs:

$.verbose = false

await $`echo 1; sleep 1; echo 2; sleep 1; echo 3;`
  .pipe(process.stdout)

Or like this:

let { stdin, stdout } = $`npm init`

for await (let chunk of stdout) {
  if (chunk.includes('package name:')) stdin.write('test\n')
}

from zx.

Why not just use nodejs API directly for this?

from zx.

Sorry if I'm misunderstanding the way this works. I thought the promise of $some-command only resolved after the command exited? In this way it would not be possible to get the output of a command as it is happening before it terminated.

from zx.

Yes, only then the command finished.

from zx.

I am worried about this.

zx is a great idea and $`` is wonderfully lightweight - you are wonderful!

But my feline instincts, are saying, get https://pegjs.org/ in here, and instead of shelling everything out to /bin/sh get node to take over.

This will not be lightweight, but it will unfurl what is going on between $`a | b` and grant access to all the output streams of all the commands being executed.

Now you will be able to get the stream of a's stderr for example, as it stdout into b's stdin. This is what programming is about, beautiful great things that keep you awake at night... that rustle your jimmies... that use node to make running commands even more powerful...

Instead of being clever and lightweight, which is very noble, you become a leader, you become legendary, which is even better.

I think we forget not to $`` all to often, here's Brian to remind us, what it means to hold a legendary idea:
https://www.youtube.com/watch?v=tc4ROCJYbm0&t=249s

Make flow control a first class citizen via https://www.sweetjs.org/ and instead of $`a | b` which is "what it is", allow for something even more brilliant like $ cat purr.txt | wc -l

(I leave the DSL sweetness up to you as you are excellent at it)

Make $`commands` into real commands, not something that is just thrown at /bin/sh, but something that comes to life in node.

@yvesnrb is on the right path, we want to keep getting the data, we want to $.stream`tail -f logs/flarp` or read sockets.

pingStream.on('data', () => {
console.log('ping wrote to stdout');
});

This is an opportunity to change the world,
you made something special, and @yvesnrb has the right idea to make it better.

from zx.

@catpea If I understand correctly are you talking about executing executables using child_process directly instead of executing sh commands?

from zx.

@catpea For those looking for the pipeline feature, I made bazx

For instance (#35 (comment))

console.log(await collect($`find ${dir} -type f -print0`.pipe($`xargs -0 grep foo`).pipe($`wc -l`)))

// $ find /tmp/bazx/test -type f -print0 | xargs -0 grep foo | wc -l
// 3
// { success: true, code: 0, stdout: "3", stderr: "" }

https://github.com/Minigugus/bazx/blob/023496cae0a53c0ed9b9c4f72f799a3ca36a6703/test/debug.js#L5-L13

from zx.