Node module to sniff the start of a stream (non-destructively) to detect the file type and encoding when you don't have the luxury of being able to restart the stream again.

It does so by using buffer-peek-stream to get the first 16KB of the stream then send that to mmmagic (which uses libmagic). Before it's finished the peek stream will unshift the bytes it's received back onto the origin stream thereby making it appear as if the origin stream was new.

npm install stream-mmmagic

var magic = require('stream-mmmagic');

var input = fs.createReadStream('somefile.csv');

magic(input, function (err, mime, output) {
  if (err) throw err;

  console.log('TYPE:', mime.type);
  console.log('ENCODING:', mime.encoding);

  // will print the *whole* file
  output.pipe(process.stdout);
});

//- TYPE: text/plain
//- ENCODING: us-ascii
//- <the file content>

A magic file is bundled with the mmmagic npm module but if you want to use your own then set the path to the file on the magicFile option.

const magicFile = '/usr/share/magic';
magic(input, {magicFile}, callback);

Use {splitMime: false} option to get back the original mime string instead of a split object.

magic(input, {splitMime: false}, (err, mime, output) => {
  console.log(mime);
});
//- text/plain; charset=us-ascii

As the input stream starts to get data the first 16KB is buffered and sent to libmagic for analysis to get file type and encoding. 1KB is more than enough for detecting file type with a standard magicFile but the reliabilty of getting the correct encoding is increased the more bytes are buffered. The tradeoff is performance and memory use.

Set peekBytes to the number of bytes you want buffered and sent to libmagic. For best results do not set below 256 bytes.

// somefile.txt is a utf8 file where the first doublebyte char is after the first 1KB of the file
var input = fs.createReadStream('somefile.txt');

magic(input, {peekBytes: 1024}, (err, mime, output) => {
  console.log(mime);
});
// not detected as utf8 because the first doublebyte char wasn't until later in the stream
//- text/plain; charset=us-ascii

magic(input, {peekBytes: 16384}, (err, mime, output) => {
  console.log(mime);
});
// now we're peeking 16KB into the file libmagic gets that first doublebyte char and knows it's utf8
//- text/plain; charset=utf8

MIT