Is there a way to cleanup when the application is closing. I am currently calling the writable.end() method but that doesn't handle the compression of the current stream?

Hi!

I was looking at an issue in a new deployment which was crashing on startup and the only error present in the stderr logs was ERR_MULTIPLE_CALLBACK from rotating-file-stream.

After patching the utils.js file with an error log to print out where I suspected the error was occurring, I uncovered the real error:

ERROR: { [Error: EACCES: permission denied, mkdir 'log']
  [stack]: 'Error: EACCES: permission denied, mkdir \'log\'',
  [message]: 'EACCES: permission denied, mkdir \'log\'',
  errno: -13,
  code: 'EACCES',
  syscall: 'mkdir',
  path: 'log' }
_stream_writable.js:456
    throw new ERR_MULTIPLE_CALLBACK();
    ^
Error [ERR_MULTIPLE_CALLBACK]: Callback called multiple times
    at onwrite (_stream_writable.js:456:11)
    at RotatingFileStream._rewrite (/opt/service/node_modules/rotating-file-stream/index.js:66:66)
    at RotatingFileStream._write (/opt/service/node_modules/rotating-file-stream/index.js:100:7)
    at doWrite (_stream_writable.js:415:12)
    at clearBuffer (_stream_writable.js:545:7)
    at onwrite (_stream_writable.js:470:7)
    at RotatingFileStream._rewrite (/opt/service/node_modules/rotating-file-stream/index.js:66:66)
    at RotatingFileStream.end (/opt/service/node_modules/rotating-file-stream/index.js:127:12)
    at RotatingFileStream.<anonymous> (/opt/service/node_modules/rotating-file-stream/utils.js:216:8)
    at Object.onceWrapper (events.js:286:20)

Furthermore, the error event was not fired. This is on nodejs v10.16.0. I will try to make a PR with a failing test to replicate this behavior.

Thanks for your module!

Node version: 8.11.3
OS: 64 bit Windows 10

I am using rotating-file-stream and morgan in my application to log all HTTP requests into daily rotated file. I have followed closely with the way that the docs of morgan recommended to use RFS:
https://github.com/expressjs/morgan#log-file-rotation

Code:
app.js

// define the log directory
let logDirectory = path.join(__dirname, 'log');

// ensure log directory exists
fs.existsSync(logDirectory) || fs.mkdirSync(logDirectory);

// create a rotating write stream
let writeStream = rfs((time) => {
    if (!time) {
        return 'access.log';
    }
    let t = moment(time);
    return `${t.format('MM-DD-YYYY')}.log`;
}, {
    interval: '1d', // daily rotate
    path: logDirectory,
    rotationTime: false,
});


// set the logger, logging every request to the console
app.use(morgan('common', {
    stream: writeStream,
}));

Error message:

events.js:183
      throw er; // Unhandled 'error' event
      ^

Error: write after end
    at writeAfterEnd (_stream_writable.js:236:12)
    at RotatingFileStream.Writable.write (_stream_writable.js:287:5)
    at Array.logRequest (C:\Haoyang\source_code\myapp\server\node_modules\morgan\index.js:130:14)
    at listener (C:\Haoyang\source_code\myapp\server\node_modules\on-finished\index.js:169:15)
    at onFinish (C:\Haoyang\source_code\myapp\server\node_modules\on-finished\index.js:100:5)
    at callback (C:\Haoyang\source_code\myapp\server\node_modules\ee-first\index.js:55:10)
    at ServerResponse.onevent (C:\Haoyang\source_code\myapp\server\node_modules\ee-first\index.js:93:5)
    at emitNone (events.js:111:20)
    at ServerResponse.emit (events.js:208:7)
    at onFinish (_http_outgoing.js:720:10)

My goal is quite simple: I just want to let morgan direct all its log into the rotating write stream created via RFS, with filename to be in MM-DD-YYYY.log format for each day. I do a check to return a fixed file name access.log for the case when the time parameter is null. However, the error still persists

We have an Apache Flume instance set up to poll the directory containing rotated files once per second and grab/process any new files that appear.

Under RFS v1.x, this setup worked perfectly fine.

RFS 2.x appears to create an empty file at the intended destination prior to moving the actual unrotated file to that destination. Due to this change, we sometimes encounter a race condition in which Apache Flume grabs/processes the empty file instead of the rotated one. Flume then ignores the rotated file because it thinks it's already processed that file (due to its name being identical), and the rotated file never gets processed.

Is there any way under RFS v2.x to force the v1.x behavior of transferring each file to its rotated destination atomically?

Hi,

I found the example:

It works but it remains a log file without ".gz".
And I found there is a closed issue: #5
It suggests to do "rm -f " + dest in the compress function.
I think you can update the document to let the user do the thing right in the first place.

Best regards,
Powen

Hello,
Very thanks for this module. It is very useful for me to logs my REST APIs.

However I am facing some issue in rotation job. The logs files are lost after one day. Can any one home me to findout and fix the issue. Below is my code snippet.

function formatDate() {
  var d = new Date(),
    month = '' + (d.getMonth() + 1),
    day = '' + d.getDate(),
    year = d.getFullYear();
    hour = d.getHours();

  if (month.length < 2)
    month = '0' + month;
  if (day.length < 2)
    day = '0' + day;
  if (hour.length < 2)
    hour = '0' + hour

  return [year, month, day, hour].join('-');
}

let log_directory = '/../accesslogs/';
if(process.env.NODE_ENV == 'app'){
  log_directory = '/../../accesslogs-app/'
}

let log_date = formatDate()
let accessLogStream = rfs.createStream('access.log', {
  size: "200M",
  interval: "1d",
  path: path.join(__dirname,log_directory,log_date)
})

Below are the log directory and files created

2020-05-11-14/
    20200512-0000-01-access.log  ->  file-size - 3.5MB
    20200512-0000-02-access.log  ->  file-size - 3.6MB
    20200512-0000-03-access.log  ->  file-size - 3.3MB
    access.log                                 ->   file-size - 3.3MB

There should be log file with 20200511-0000-01-access.log file name as i run the project on 2020-05-11. Next when i have a look at the log directory I didn't find the first day logs.

Can you guys help me out to findout the reason and how to fix this issue?

Many thanks.

I noticed that events in subject may cross over if rotation job takes a time longer than the time for next rotation event.
This should happen only during tests, where rotation events are so close each other... anyway it could be a good idea to fix this

Hi, I want to use the package to create a rotating log file with time, I did it, but the first log file doesn't have the time stamp as the following. I have seen the package documentation, but I can't understand if there is this option inside the package.

This is my configuration:

Hello! 👋 Thanks for making this great module. It's really useful.

I was happily using the 1.x version of this module with the Hapi framework's https://hapi.dev/family/good/api/?v=9.0.0 plugin. Basically, that boils down to:

{
  module: 'rotating-file-system',
  args: [
    'filename'
    {
      ...options...
    }
  ]
}

in the plugin's configuration. Setting 'module' to a string which is a package name results in the plugin requiring the package and assuming the default export is a constructor it will call new on in order to get a new stream.Writable instance. This worked well with version 1.x...

With version 2.x, however, it's broken:

1. The default export isn't the class anymore.
2. While the class itself is exported, and could be provided as the module option in the plugin's config, the class now requires a "generator" function as its first parameter instead of a file name string.
3. Looking inside the rotating-file-stream package, passing a string as the filename parameter to the also-exported createStream() function internally calls createGenerator() (because I don't set the rotate option to non-zero), but createGenerator() isn't exported, so must be copied.

If I copy createGenerator(), then I can do this to get things working again:

const rfs = require('rotating-file-stream');

const createGenerator = function(filename) {
  // ...verbatim from rotating-file-stream... //
}

class RotatingFileStream extends rfs.RotatingFileStream {
  constructor(filename, options) {
    super(createGenerator(filename), options);
  }
}
...
// plugin config
{
  module: RotatingFileStream,
  args: [...]
}

But the problem with that is that it circumvents the call to the also-internal checkOpts() method that massages the given options object before passing it to the RotatingFileStream constructor internally. So I'd have to copy that out as well to be really "complete", but that's now getting into internal logic, which is even messier.

What I would propose, to stay as much as possible inline with the direction this module took in 2.x and yet remain backward-compatible here, is:

1. Expose createGenerator() as a utility to create a default generator to pass to the constructor (could also point this out in the README where the "complex generator function" example is given, since this is the same implementation).
2. Move the call to checkOpts() out of createStream() and into the RotatingFileStream constructor since createStream() doesn't even use that function's output (it just passes it to the constructor and instead still uses the original options.rotate).

The "rotation" functionality of this module is missing. It will correctly track time/size changes to determine when the current file stream needs to be swapped out, but instead of rotating the list of previous files it adds to it.

As an example, I've created a stream using the filename function:

rotatingFileStream(
    function(time, index) { return 'access.log' + (index ? ('.' + index) : ''); },
    {
        path: '/tmp',
        size: '500B'
    }
);

Expected behaviour:

1. Write 500 bytes to the stream
2. access.log gets rotated to access.log.1
3. Write 500 bytes to the stream
4. access.log.1 gets rotate to access.log.2
5. access.log gets rotated to access.log.1

Observed behaviour:

1. Write 500 bytes to the stream
2. access.log gets rotated to access.log.1
3. Write 500 bytes to the stream
4. access.log gets rotate to access.log.2

Ideally this should let me, as an example, delete logs that have been rotated 5 times by deleting files matching access.log.N where N > 5 (even better, this could be a feature of the module but that's not important now). Instead, I need to first check how many extra log files I have and start deleting from access.log.1 onward. This also means that the next time it tries to do a rotation it will see that access.log.1 is available.

I've got a lot of empty .gz files, the following is my Typescript code:

import rfs = require("rotating-file-stream")
import * as stream from "stream"

export default class RotatedFile {
  private static pad(num: number): string {
    return (num > 9 ? "" : "0") + num
  }

  private static generator(time: Date, index: number) {
    if (!time) {
      return "file.log"
    }

    const month = time.getFullYear() + RotatedFile.pad(time.getMonth() + 1)
    const day = RotatedFile.pad(time.getDate())
    const hour = RotatedFile.pad(time.getHours())
    const minute = RotatedFile.pad(time.getMinutes())

    return month + day + "/" + month + day + "-" + hour + minute + "-" + index + ".json.gz"
  }

  private stream: stream.Writable

  constructor(rootDir: string) {
    this.stream = rfs(RotatedFile.generator, {
      compress: "gzip",
      interval: "1d", // rotate daily
      path: rootDir,
      size: "256M" // rotate every 256 MegaBytes written
    })
  }

  public write(data: string): void {
    this.stream.write(data)
  }
}

Is there any misconfiguration in my code? Thanks!

Cannot invoke an expression whose type lacks a call signature. Type 'typeof import("C:/project/node_modules/rotating-file-stream/index")' has no compatible call signatures.

I think the definition file is wrong. I would also consider not using a default export.

Hi guys @jorgemsrs @iccicci
I just found an issue in which if the log file don't have a write permission.
The error will occur if the log file is read-only that was modified by an another node app or intentionally change the file permission.

It will return a TypeError like this which is confusing to the end-user:

node app.js
Express server listening on port 3333
TypeError: this.destroy is not a function
    at destroy (/home/armano/_node/webinnov_project/node_modules/rotating-file-stream/index.js:71:18)
    at rewrite (/home/armano/_node/webinnov_project/node_modules/rotating-file-stream/index.js:78:24)
    at fsStat (/home/armano/_node/webinnov_project/node_modules/rotating-file-stream/index.js:102:28)
    at FSReqWrap.oncomplete (fs.js:111:15)

To solve this TypeError I have modified the destroy() function to put in a try-catch. For example:

  private rewrite(chunk: Chunk, callback: Callback): void {
    const destroy = (error: Error): void => {
      try {
        this.destroy();
      } catch(error) {
        return callback(error);
      }
    };
...
...

Here's my snippets using the rfs:

this._debuggerStream = rfs.createStream("debug.log", {
  size: '50M',
  interval: '1d',
  immutable: true,
  path: this._directory,
  mode: 0o777
})

// Since we have the try-catch, the exception will now catch here instead of displaying
// TypeError: this.destroy is not a function --which is confusing to the end-user
this._debuggerStream.on('error', function (err) {
  console.log('MyWebApp Debug Error - %s', err)
})

What do you think of my fix? Or do you have idea on how to catch the rfs exception in onError event? Your help will be greatly appreciated.
Thanks!

Please add a index.d.ts for typescript

When rotating by interval filename generator is supposed to return not-rotated file name when there is no time parameter initially and writing will happen to this file.

When rotation happens filename generator is called with time parameter being the beginning of the just ended rotation interval and the not-rotated file name will be moved to be named per returned value.

With 1h rotation interval and filename generator returning current.log when there is no time parameter imagine the sequence:

• start app at 12:01 => writes appended to current.log
• app crashes / is killed 12:59
• start app at 13:01 => writes appended to current.log
• rotation comes up at 14:00 => current.log moved to 20170917T13 and new current.log created

Now 20170917T13 contains data that should have been in log file 20170917T12.

https://gist.github.com/tkurki/3eb2fc0ce563985c57952b5a52ef5424 can demonstrate this with minute resolution.

Is there an easy way to fix things so that data would always be in a correctly timestamped file with this package or should I look elsewhere?

I noticed that when I create a rotating file stream on a readonly file system it will fail and assign this.error.

The error from this.error is only used to destroy the stream on the first write. This leads to the following code hanging indefinitely

    return new Promise((resolve) => {
      self._stream = rfs.createStream(
        self._logFileLocation,
        {
          size: MAX_LOG_FILE_SIZE,
          maxFiles: MAX_LOG_FILES,
          compress: 'gzip'
        }
      )
      self._stream.once('open', onOpen)
      self._stream.once('error', onError)

      function onOpen () {
        self._stream.removeListener('error', onError)

        self.hasOpened = true
        resolve({})
      }

      /** @param {Error} err */
      function onError (err) {
        self._stream.removeListener('open', onOpen)

        resolve({ err: err })
      }
    })

I believe a reasonable fix for this would be to emit an error event from the init() callback so that the user gets feedback about the stream without having to write() to it.

I see the following in my application's logs:

events.js:160
      throw er; // Unhandled 'error' event
      ^

Error: write after end
    at writeAfterEnd (_stream_writable.js:193:12)
    at RotatingFileStream.Writable.write (_stream_writable.js:240:5)
    at Array.logRequest (/usr/local/dcos-universe-browser/node_modules/morgan/index.js:130:14)
    at listener (/usr/local/dcos-universe-browser/node_modules/on-finished/index.js:169:15)
    at onFinish (/usr/local/dcos-universe-browser/node_modules/on-finished/index.js:100:5)
    at callback (/usr/local/dcos-universe-browser/node_modules/ee-first/index.js:55:10)
    at ServerResponse.onevent (/usr/local/dcos-universe-browser/node_modules/ee-first/index.js:93:5)
    at emitNone (events.js:91:20)
    at ServerResponse.emit (events.js:185:7)
    at finish (_http_outgoing.js:596:10)

This occurs every night at 00:00 UTC. I use it like this:

// Create a rotating write stream
var accessLogStream = rotatingFileStream(function(time, index) {
    if (!time) time = new Date();
    return 'access-' + time.toISOString().substr(0,10).split('-').join('') + '.log'
}, {
    interval: '1d',  // rotate daily
    path: logDirectory
});

Is there a hint where the problem can arise from? Thanks!

I switched down from node 14 to node 12 and all the tests pass.

Something about the order and frequency of events in node14 as well as the string constants in error messages causes the entire test suite to fail.

Hi @iccicci ,

I have my stream config as follow:
{ interval: '10s', compress: (source, dest) => 'cat ${source} | gzip -c9 > ${dest}.gz', maxFiles: 3 }

I was monitoring the log file with tail -F,
when it comes to the rotation time,
it did not generate the .gz file and the writing stream was stopped

I have received the error message as follow:
{ [Error: EPERM: operation not permitted, stat 'C:\git\***\log\access.log'] errno: -4048, code: 'EPERM', syscall: 'stat', path: 'C:\\git\\***\\log\\access.log' }

Error [ERR_STREAM_DESTROYED]: Cannot call write after a stream was destroyed
at doWrite (_stream_writable.js:411:19)
at writeOrBuffer (_stream_writable.js:399:5)
at RotatingFileStream.Writable.write (_stream_writable.js:299:11)

On certain target machines, when set for a daily rotation, the file will rotate 1 second prior to 0:00 UTC, and then again at 0:00 UTC
This would seem to occur on target machines with internal clocks that run slightly fast.
The rotation time is set using setTimeout with a 24 hour timer.
If due to the internal clock, this triggers slightly early the rotation takes place and is completed prior to 0:00 UTC, and the scheduler then calls setTimeout again to trigger at 0:00 UTC.

If i create a config object like this:

var config = {
    size: '10M',
    interval: '30s',
    rotate: 5
};

and reuse it on two calls to rotating-file-stream, the second call will fail with:

Error: Don't know how to handle 'options.size' type: number

The type number seems to come from expanding the original object from 10M to 10485760.

My workaround is to create a new object with exact same properties on each call to rotating-file-stream, but in my opinion, it should be up to the module to clone it and not impose this non documented restriction on the user.

Hello, thank you for this package.

I noticed that the files stay with .log extension after compression.
It would be nice to have the option to change the file extension to .gz after compression, so it's more clear that this is a gzipped file.

My deployment environment is linux, however I am working on development in Windows; my code might work in linux, but it's certainly not working on Windows and having unusual behavior.

I tried setting compress to true, and had problems.. but then I realized that perhaps my Windows installation didn't have the compression binaries that rotating-file-stream is expecting, so I overrode it with my own function and having the same problem:

let compressFunction = (source, dest) => 'tar -czf "' + dest + '" "' + source + '"';

if (accessLogging) {
  let accessLogStream = rfs(accessLogName, {
    interval: '60s',
    compress: compressFunction,
    path: path.join(accessLogPath, accessLogName)
  })
  app.use(morgan(accessLogFormat, { stream: accessLogStream }))
}

When compress isn't enabled, it rotates as expected. However with compress specified as above, the files that are written do appear to be blank (pre-compression) and then aren't actually compressed.

It appears when it's trying to compress the files, it's executing the temporary file itself, as I get a prompt to "Choose app to open SUCHANDSUCH.tmp" at the interval about on par with what I have specified.

this is the configuration that I used:

const expressLogStream = rfs('express.log', {
  interval: '1m',
  compress: 'gzip',
  maxFiles: 100,
  path: path.join(__dirname, '..', 'public', 'logs'),
})

and I get this error that ends node process:

/Users/m0hammad/pheebs/episode-server/node_modules/rotating-file-stream/compress.js:216
        for(var i in files) files[i].once("error", callback);
                                     ^

TypeError: files[i].once is not a function
    at RotatingFileStream.gzip (/Users/m0hammad/pheebs/episode-server/node_modules/rotating-file-stream/compress.js:216:31)
    at /Users/m0hammad/pheebs/episode-server/node_modules/rotating-file-stream/compress.js:102:14
    at FSReqCallback.oncomplete (fs.js:137:23)

Hi
I'm trying to rotate my logs by 1day. But it seems that it not good at different timezones.

e.g. I am in Beijing which located at East 8 (+8:00) zone.
I set my time zone by:

process.env.TZ = 'Asia/Beijing'

and rotate logs by:

var access200LogStream = rfs('access200.log', {
  interval: '1d', // rotate daily
  path: path.join(__dirname, 'logs')
})

It seems that it cut the logs according the +0:00 timezone.

when I want to see my logs of 2020-08-06 of Beijing timezone,
I have to look into the file of 20200805-0000-01-access200.log
and grep out the logs later than 05/Aug/2020:16:00:00 +0000

than look into the file named 20200806-0000-01-access200.log
and grep out the logs before than 06/Aug/2020:16:00:00 +0000

that is my logs of Beijing day 2020-08-06.

Did I miss some important statements in my code the declare the timezone?

I have this in my code -

// create a rotating write stream
var accessLogStream = rfs(logFileName, {
  interval: '5m', 
  path: logDirectory
});

The log files created every 5 minutes are -
! /tmp: ls -ltr access
-rw-rw-r-- 1 dev develop 0 Nov 28 09:50 20181128-0950-01-access.log
-rw-rw-r-- 1 dev develop 0 Nov 28 09:55 20181128-0955-01-access.log
-rw-rw-r-- 1 dev develop 782 Nov 28 09:58 20181128-0945-01-access.log
-rw-rw-r-- 1 dev develop 0 Nov 28 10:00 20181128-1000-01-access.log
-rw-rw-r-- 1 dev develop 0 Nov 28 10:05 20181128-1005-01-access.log
-rw-rw-r-- 1 dev develop 0 Nov 28 10:10 access.log

The original file created when the server started is access,log, which was renamed to 20181128-0945-01-access.log. The server still keeps logging to 20181128-0945-01-access.log file instead of logging it to access.log

I got empty log file when use interval option, I need logs file some periods like every 30 minutes, but when using interval option it will save logs although stream data is empty, how can i check it save only not empty file?

code as belows:-
`

var rfs = require('rotating-file-stream');

function pad(num) {
return (num > 9 ? "" : "0") + num;
}

function logRFS(logPath) {
return rfs((time, index)=>{
function pad(num) {
return (num > 9 ? "" : "0") + num;
}
if (!time)
return "file.log";

    let year = time.getFullYear();
    let month = pad(time.getMonth() + 1);
    let date = pad(time.getDate());
    let hour = pad(time.getHours());
    let minute = pad(time.getMinutes());

    return year+"-"+month+"-"+date+"/"+year+"-"+month+"-"+date+"_"+hour+minute+"_"+index+"_"+"file.log";
}, {
    path: logPath,
    interval: "5s",		// rotate every 5seconds
    // compress: "gzip"	// compress rotated files
});

}

var stream = logRFS("myLogs");

setInterval(()=>{
console.log("test....");
},1000);

`

Example of file with current module:

var fs = require('fs');
var i = 0;
var logFileName = 'app.log';
var handler = function(err) {
    if (err) {
        console.log(err.stack());
        process.abort();
    }
};

setInterval(function() {
    var data = ++i;
    fs.appendFile(logFileName, data + '\n', handler);
});

var rfs = require('rotating-file-stream');
var stream = rfs(logFileName, {
    size: '1K',
    interval: '1d'
});

When I start it in first time:

sudo node app.js

I receive one file - app.log. On second start (if file app.js is bigger than 1 kilobyte already after first start) I receive another file:
NaN0NaN0NaN-0NaN0NaN-01-app.log

It seems like incorrect date parsing or something else.

I have this current setup for rotating-file-stream writing logs whenever a request comes in.

rotating-file-stream: 1.3.9, 1.4.1
node: v6.9.0, v10.4.0 (same as my windows)
OS: Ubuntu 16.04

let filenameBase = 'some base file name'
let directory = 'directory'
function makeGenerator (lastIndex) {
    let logIndex = 0
    if (Util.isNumber(lastIndex)) {
      logIndex = lastIndex
    }

    return function generator (time, index) {
      logIndex++
      return _filenameBase + '-' + logIndex + '.log'
    }
  }

  let stream = rfs(makeGenerator(lastIndex), {
    size: '1M',
    interval: '1d',
    immutable: true,
    path: directory,
    mode: 0o777
  })

It works currently but if I remove the current active file filenameBase-logIndex.log, the log being written are not flushed out to the file (expecting it to be recreated - which is currently working in windows) and only creates a new file when it has rotated (the size being recorded maxes out or reaches the maxSize threshold).

Not receiving any errors/warnings during the process.

I don't know if this is the current behavior but please advise.

Edit: Tried to create a PoC for the fs.createWriteStream writing entries and removing the file halfway does not throw any error on the stream and just keeps writing until stream.end() is called.

followed the given example, and I add the "maxFiles" opts to 10, but number of the rotated files still continue to rise by larger than 10, did i miss something in the doc? hope to hear back, thanks!

Have i issue with:

const stream = rfs.createStream("./folder/file.log", {...);

and

const stream = rfs.createStream(path.join(__dirname, "./folder/file.log") {...});

• If i use ./ relative path or folder the name of rotate folder finish with "."
• Full path not generate folder.

I want to use the generator along with the rotate option. Using the normal file name without the generator function, it works perfectly.

I have the following code:

const pad = num => (num > 9 ? '' : '0') + num;
const fileNameGenerator = (time, index) => {
  if (!time) return `app.log`;

  var month = time.getFullYear() + "" + pad(time.getMonth() + 1);
  var day = pad(time.getDate());
  var hour = pad(time.getHours());

  return `app.log.${month}-${day}-${hour}.${index}`;
};

logStream = createStream(fileNameGenerator, {
    path: '/var/log/app',
    interval: '10s',
    rotate: 7 // keep 7 rotated logs
  });

Expected behavior:
This should generate 7 rotated log files in 10 second intervals with the file name app.log.${month}-${day}-${hour}.${index}

Observed behavior
Only app.log is generated and stops logging after 10 seconds

The example use of a generator function in the documentation fails with "write after end" error:

Simple program to reproduce this is:

var rfs    = require('rotating-file-stream');

function pad(num) {
    return (num > 9 ? "" : "0") + num;
}

function generator(time, index) {
    if(! time)
        return "file.log";

    var month  = time.getFullYear() + "" + pad(time.getMonth() + 1);
    var day    = pad(time.getDate());
    var hour   = pad(time.getHours());
    var minute = pad(time.getMinutes());

    return "/storage/" + month + "/" + month +
        day + "-" + hour + minute + "-" + index + "-file.log";
}

var stream = rfs(generator, {
    size:     '1M',
    interval: '30m'
});

setInterval(function () {
    stream.write('xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx');
}, 10);

Using node 8.4.0 on Bash on Windows 10.

A nice option to add would be totalFiles. This could limit the total number of files on disk at any moment. This can be useful if disk is limited.

I think a good way to implement this would be to handle the 'rotated' event and keep a queue of filenames. Then if the totalFiles >= queue.length, we could delete the oldest file. Does this seem a reasonable way to implement?

Would the firstOpen be a good place to wire up the event handling?

// index.js

// compress.js

// compress.js

Don't use compress option

Didn't send tmp(options.compress = null) parameter to findName, findName will return the result of this.generator, like 20170531-1617-01-/Users/someone/.projectName/logs/projectName.error.log.

Then move function use fs.rename move /Users/someone/.projectName/logs/projectName.error.log to ${process.cwd()}/20170531-1617-01-/Users/someone/.projectName/logs/projectName.error.log.

Because of options.compress is false, so self.compress(name) in callback will not run.

Use gzip compress option

Sent tmp(options.compress = "gzip") parameter to findName, findName will return name like /Users/someone/.projectName/logs/projectName.error.log.1.rfs.tmp.

Then move function use fs.rename move /Users/someone/.projectName/logs/projectName.error.log to /Users/someone/.projectName/logs/projectName.error.log.1.rfs.tmp.

Because of options.compress is true, so self.compress(name) in callback will run.

In compress function will call findName again and didn't send tmp(options.compress = null), so get the result name like ${process.cwd()}/20170531-1617-01-/Users/someone/.projectName/logs/projectName.error.log.

Then self.touch → self.gzip also create folder like ${process.cwd()}20170531-1617-01-/Users/someone/.projectName/logs/projectName.error.log.

Conclusion

I don't knwo if I understand correctly, the result is the folder like 20170531-1617-01- will always be created. I don't know it is a feature or bug. Please give me help.

I am trying to use rotating-file-stream in express with morgan. Everything is coded just as the tutorial of morgan but got this error: Argument of type '"combined"' is not assignable to parameter of type 'FormatFn'. I load a screenshot as following.

Version info:

[email protected]
[email protected]
@types/[email protected]
[email protected]
@types/[email protected]

rotating-file-stream v1.3.10
node v10.14.1
OS windows10

Issue

I tried this settings for at least monthly logs since my app is not expected to produce so much log

rfs('foo.log', {
  size: '10M', // rotate every 10 MegaBytes written
  interval: '30d', // rotate 30 days
  compress: 'gzip', // compress rotated files
});

and Node spills out millions of

(node:1684) TimeoutOverflowWarning: 2155129676 does not fit into a 32-bit signed integer.
Timeout duration was set to 1.
(node:1684) TimeoutOverflowWarning: 2155129670 does not fit into a 32-bit signed integer.
Timeout duration was set to 1.
(node:1684) TimeoutOverflowWarning: 2155129667 does not fit into a 32-bit signed integer.
Timeout duration was set to 1.
(node:1684) TimeoutOverflowWarning: 2155129659 does not fit into a 32-bit signed integer.
Timeout duration was set to 1.
(node:1684) TimeoutOverflowWarning: 2155129636 does not fit into a 32-bit signed integer.
Timeout duration was set to 1.
(node:1684) TimeoutOverflowWarning: 2155129634 does not fit into a 32-bit signed integer.
Timeout duration was set to 1.

and a bazillion files are being created (just for illustration)

       LastWriteTime Length Name
       ------------- ------ ----
06.01.2019     01:21     20 20190106-0000-01-foo.log
06.01.2019     01:21     20 20190106-0000-01-banana.log
06.01.2019     01:21     20 20190106-0000-01-bar.log
06.01.2019     01:21     20 20190106-0000-01-apple.log
06.01.2019     01:21     20 20190106-0000-02-foo.log
06.01.2019     01:21     20 20190106-0000-02-banana.log
06.01.2019     01:21     20 20190106-0000-02-bar.log
06.01.2019     01:21     20 20190106-0000-02-apple.log
06.01.2019     01:21     20 20190106-0000-03-foo.log
06.01.2019     01:21     20 20190106-0000-03-banana.log
06.01.2019     01:21     20 20190106-0000-03-bar.log
06.01.2019     01:21     20 20190106-0000-03-apple.log
06.01.2019     01:21     20 20190106-0000-04-foo.log
06.01.2019     01:21     20 20190106-0000-04-banana.log
06.01.2019     01:21     20 20190106-0000-04-bar.log
06.01.2019     01:21     20 20190106-0000-04-apple.log
06.01.2019     01:21     20 20190106-0000-05-foo.log
06.01.2019     01:21     20 20190106-0000-05-banana.log
06.01.2019     01:21     20 20190106-0000-05-bar.log
06.01.2019     01:21     20 20190106-0000-05-apple.log
and so on

Cause

After a bit fiddling I figured it out ...
30 days is probably a very big value one sane developer would ever use 😱
and the node error looked fishy already

(node:1684) TimeoutOverflowWarning: 2155129450 does not fit into a 32-bit signed integer.
Timeout duration was set to 1.

2,147,483,647 is the maximum positive value for a 32-bit signed binary integer in computing

So quick math here for how many d you could set option interal to
2,147,483,647 / (60 * 60 * 24) = 24,855
Seems a bit much days for crashing with 30 already ...

Throw in a divide by 1000 for good measure
(I am quite sure it is due to setInterval() being in milliseconds 😜)
and you end up at a maximum of 24.8d as value

Trying with interval: '24d' everything works as expected ✅
and interval: '25d' explodes 💥 😏

Maybe check the value and throw error 👍 but better do not let it run wild!

I use rotating-file-stream as a dependency in a custom logging framework, but I have not tested the rolling file behavior much because we send a good amount of log data to Splunk. I cannot reproduce the following in test cases for our framework, but when I add our framework to a real application, the following errors from rotating-file-stream start appearing.

Any ideas?

Using package version 1.2.2, with Node 7.9.0

TypeError: this.timer.unref is not a function
    at RotatingFileStream.interval (/Users/mrapczynski/Development/Node/Applications/BSIS Next Gen/node_modules/fhda-pubsub-logging/node_modules/rotating-file-stream/interval.js:56:13)
    at Object.onceWrapper (events.js:293:19)
    at emitNone (events.js:86:13)
    at RotatingFileStream.emit (events.js:188:7)
    at WriteStream.<anonymous> (/Users/mrapczynski/Development/Node/Applications/BSIS Next Gen/node_modules/fhda-pubsub-logging/node_modules/rotating-file-stream/index.js:252:8)
    at Object.onceWrapper (events.js:293:19)
    at emitOne (events.js:96:13)
    at WriteStream.emit (events.js:191:7)
    at WriteStream.<anonymous> (fs.js:2043:10)
    at FSReqWrap.oncomplete (fs.js:114:15)

Hello,

I have an issue with an express API when multiples request are processed in the same time, there are not all logged (rotating-file-stream: 2.1.4).

Here is a sample code:

  const accessLogStream2 = rfs.createStream('access-test.log', {
    interval: '1d',
    path: path.join(__dirname, 'log'),
    teeToStdout: true,
  })

  api.use((req, res, next) => {
    console.log(`console ${Date.now()} ** ${req.url}`)
    accessLogStream2.write(`rfs ${Date.now()} ** ${req.url}\n`)
    next()
  })

And here is the output:

console 1612715099666 ** /api/states?module=srm&logicalId=devices
console 1612715099667 ** /api/states?module=srm&logicalId=eth0-upload
console 1612715099669 ** /api/states?module=srm&logicalId=lbr0-upload
console 1612715099670 ** /api/states?module=srm&logicalId=eth0-download
console 1612715099671 ** /api/states?module=srm&logicalId=lbr0-download
rfs 1612715099666 ** /api/states?module=srm&logicalId=devices
rfs 1612715099667 ** /api/states?module=srm&logicalId=eth0-upload
...
console 1612715099691 ** /api/states/165
console 1612715099692 ** /api/states/168
console 1612715099692 ** /api/states/166
console 1612715099693 ** /api/states/169
console 1612715099694 ** /api/states/167
rfs 1612715099691 ** /api/states/165
rfs 1612715099692 ** /api/states/168

It seems that only 2 logs are logged 😞
Any idea?

Hello,

I am playing with this plugin but can't get the time parameter or index parameter.
time is null and index is undefined.
I am providing an interval and a path option.

Looking at the code:

• Here the value of this.generator https://github.com/iccicci/rotating-file-stream/blob/master/index.js#L31
• It is used here https://github.com/iccicci/rotating-file-stream/blob/master/index.js#L130

I understand now why time is null and index is undefined. Is this normal?

I can't get the rotating file stream to work with the immutable option set:

const rfs = require('rotating-file-stream');

const stream = rfs('yolo.log', {
    interval: '1m',
    immutable: true,
});

stream.write('abc\n');

stream.end();

The write immediately throws an exception:

events.js:141
      throw er; // Unhandled 'error' event
      ^

Error: write after end
    at writeAfterEnd (_stream_writable.js:167:12)
    at RotatingFileStream.Writable.write (_stream_writable.js:212:5)
    at Object.<anonymous> (/projectdir/yolo.js:8:8)
    at Module._compile (module.js:409:26)
    at Object.Module._extensions..js (module.js:416:10)
    at Module.load (module.js:343:32)
    at Function.Module._load (module.js:300:12)
    at Function.Module.runMain (module.js:441:10)
    at startup (node.js:140:18)
    at node.js:1043:3

Do you have any idea what is happening here?
Thanks

While wrote own implementation on TS found that

https://github.com/iccicci/rotating-file-stream/blob/master/compress.js#L246

does not close handle of file, that cause test failing when removing previous results
due EPERM with directory is not empty

First of all thank you for this awesome lib 🙏🏼

I'm experiencing some weird issue with disappearing logs.
This is my production config of pino logger.

import path from 'path';
import pino from 'pino';
import { createStream } from 'rotating-file-stream';

function getISODate(date) {
  const clone = new Date(date);
  clone.setHours(12);
  return clone.toISOString().split('T').shift();
}

const generator = (time) => {
  if (!time) return `${filename}.log`;

  return `${filename}-${getISODate(time)}.log`;
};

const stream = createStream(generator, {
  interval: '1d', // rotate daily
  path: path.join(process.env.DEPLOY_BASE_PATH, `/logs/${filename}/`),
});

const logger = pino({ ...defaultConfig, name: filename }, stream);

If I check log in real time, I can see the logs without any issue. but if I check the log from "yesterday" (by the date) i see only last operation that occurred via cronjob at 23:30, rest of the day logs are not there.

I suspect that it is somehow related to my config of rotating-file-stream, but I don't see misconfiguration.
Maybe you can spot the issue?

My app runs on Node v12

I am trying to use rotating-file-stream with Hapi.js and good.

My trouble is that the compresses files don't have a .gz extention and that makes most Linux utilities confuzed.
If I change the compress string to add '.gz', the file with the same name is not deleted but it's size is set to zero !!! I expected the file to be deleted.
BTW: the docs say not to delete source file, I tried it and strage errors happen (as expected)

This is the relevant part of my setup:

      {
        module: 'rotating-file-stream',
        args: [
          'nodejs-web.log',
          {
            path: './logs',
            size: '10K',
            interval: '1m',
            compress: function( source, dest) {
              return 'cat ' + source + ' | gzip -c9 > ' + dest + '.gz';
            }
          }
        ]
      }

This is what I get:

ls -l
-rw-rw-r-- 1 user user 1347 Jul 22 20:32 20160722-2032-01-nodejs-web.log.gz
-rw-rw-r-- 1 user user    0 Jul 22 20:44 20160722-2043-01-nodejs-web.log
-rw-rw-r-- 1 user user 1207 Jul 22 20:44 20160722-2043-01-nodejs-web.log.gz
-rw-rw-r-- 1 user user    0 Jul 22 20:44 20160722-2044-01-nodejs-web.log
-rw-rw-r-- 1 user user 1263 Jul 22 20:44 20160722-2044-01-nodejs-web.log.gz
-rw-rw-r-- 1 user user 7107 Jul 22 20:44 nodejs-web.log

Hi,

like there is another issue now, I only added the compression option, and there resulting rotated files are some kind of binary files that I could not figure out how to see their content. they cannot be decompressed with gzip, and mac finder cannot open them, they don't have .gz suffix either.

Can i change the request ip due to the ngnix proxy.
Thank you in advance

I realized that end method is not correctly implemented.

Reguardless what stream.Writable.end documentation says, due to this bug end method works properly only if called in following forms:

rsf.end();
rfs.end(chunk);
rfs.end(callback);

In all other cases, the parameters encoding and callback are ignored.

For debugging I'd like to be able to tee output to stdout and to file. Is that possible?

Hi,
I have this error every time is creating a new compressed file

      throw er; // Unhandled 'error' event
      ^

Error: EPERM: operation not permitted, stat 'C:\Users\cj68qe\Desktop\backend NSIA\logs\data.log'
Emitted 'error' event on RotatingFileStream instance at:
    at errorOrDestroy (internal/streams/destroy.js:108:12)
    at onwriteError (_stream_writable.js:424:5)
    at onwrite (_stream_writable.js:445:5)
    at destroy (C:\Users\cj68qe\Desktop\backend NSIA\node_modules\rotating-file-stream\index.js:75:20)
    at C:\Users\cj68qe\Desktop\backend NSIA\node_modules\rotating-file-stream\index.js:109:28
    at FSReqCallback.oncomplete (fs.js:168:21) {
  errno: -4048,
  code: 'EPERM',
  syscall: 'stat',
  path: 'C:\\Users\\cj68qe\\Desktop\\backend NSIA\\logs\\data.log'
}
[nodemon] app crashed - waiting for file changes before starting...```

my code:
const express = require('express')
const helmet = require("helmet");
const app = express();
const fs = require('fs');
const morganBody = require('morgan-body')
const morgan = require('morgan');
const path = require('path');
var mustacheExpress = require('mustache-express');
const bodyParser = require('body-parser');
const { logger, timeStamp } = require('./logger')
const api = require('./api')
const form = require('./form')
const cors = require('cors')
const compression = require('compression');
const port = (process.env.PORT || 8080);
var rfs = require('rotating-file-stream');
const log = fs.createWriteStream(
  path.join(__dirname, "logs", "data.log"), { flags: "a" }
);

// var accessLogStream = rfs.createStream('access.log', {
//   size: '10M', // rotate 10 Megs
//   compress: "gzip",
//   path: path.join(__dirname, '/logs')
// })
var dataLogStream = rfs.createStream('data.log', {
  size: '2K', // rotate 10 Megs,
  interval: '5s',
  compress: "gzip",
  path: path.join(__dirname, '/logs')
})

// deactivate CORS
app.use(cors())

app.use(helmet())
app.use(compression())
// Mustache middleware
app.engine('mustache', mustacheExpress());
app.set('view engine', 'mustache');
app.set('views', `${__dirname}/views`);


app.use(bodyParser.urlencoded({ extended: true }))
app.use(bodyParser.json())
// middleware logger
app.use(logger)
// middleware to log and add time stamp
app.use(timeStamp)
// setup the logger
// app.use(morgan('combined', { stream: accessLogStream }))

morganBody(app, {  
  noColors: true,
  stream: dataLogStream,
});
// APIS for new UAT & new ST
app.use('/api',api) 

// API for sending SSR form serialized
app.use('/form', form)





app.listen(port, () => {
  console.log(`Example app listening at http://localhost:${port}`)
})

Every several hours app process (debugged in WebStorm) causes Windows dialogue:

Windows cannot open this file:
File: tmp-<...>.tmp

When Cancel is pressed on the dialogue, an error is thrown with stack:

<...>\node_modules\rotating-file-stream\node_modules\tmp\lib\tmp.js:355
      if (e.errno != -_c.EBADF && e.errno != -c.ENOENT) {
                                              ^
ReferenceError: c is not defined
    at _removeCallback (<...>\node_modules\rotating-file-stream\node_modules\tmp\lib\tmp.js:355:47)
    at _cleanupCallback (<...>\node_modules\rotating-file-stream\node_modules\tmp\lib\tmp.js:409:5)
    at cb (<...>\node_modules\rotating-file-stream\compress.js:55:4)
    at ChildProcess.exithandler (child_process.js:197:7)
    at emitTwo (events.js:106:13)
    at ChildProcess.emit (events.js:191:7)
    at maybeClose (internal/child_process.js:877:16)
    at Process.ChildProcess._handle.onexit (internal/child_process.js:226:5)

The issue is likely related to raszi/node-tmp#102.