smonn / globcat Goto Github PK

globcat makes combining files super easy, appreciate it 👍

One feature I'd like to see added is sourcemap generation, since I'm using it to concat JS files currently.

Thanks again!

Hi, this is a really useful library, so thank you!

I've hit an issue when concatenating large files, whereby the file contents get split up and mixed in with the content of other files.

Issue reproduction

I've created a codesandbox to reproduce the issue:

https://codesandbox.io/p/sandbox/clever-heisenberg-fkfxrt

The two input files in ./data are a.txt (containing thousands of lines of "a") and b.txt (containing a single "b"). There's a build script which runs globcat and outputs to ./dist/output.txt.

Expected outcome - I expect output.txt to contain all the content from a.txt followed by the content from b.txt.
Actual outcome - The content from b.txt is in the middle if the output, with thousands of "a"s before and after it.

Possible cause

I have limited experience with Node streams but I think the issue might lie here: https://github.com/smonn/globcat/blob/main/src/lib/files-to-stream.ts. Specifically this bit (shortened for clarity):

for (const file of files) {
  const stream = fs.createReadStream(file)
  passthrough = stream.pipe(passthrough, { end: false })
}

As I understand it, because this happens within a loop, it means that files are being streamed to passthrough simultaneously, so their order is not guaranteed. I imagine this is fine as long as each file is below the highWaterMark (64KiB by default) and is therefore processed as a single chunk.

However, files larger than this will be broken into chunks, and those chunks may be delayed and processed after chunks from a different file, making it possible for file content to be jumbled up as in my reproduction.

Possible fix

I've fixed this locally by writing chunks synchronously. As I say, I'm no expert on Node streams so I don't know whether there are any issues with this solution, but I've included it in case it's helpful:

for (const file of files) {
  const stats = await fsPromises.stat(file)
  if (stats.isFile()) {
    const stream = fs.createReadStream(file)
    stream.once('end', () => {
      queueSize--
      if (queueSize === 0) {
        passthrough.end()
      }
    })
    for await (const chunk of stream) {
      passthrough.write(chunk)
    }
  } else {
    throw new Error('Not a file: ' + file)
  }
}

Thanks again,

Pete

It's currently not pushing the updated package.json to the main branch. See build output for more details:

https://github.com/smonn/globcat/actions/runs/4209293690/jobs/7306163261