I attempted to use gulp-csvtojson with a CSV file that was approximately 300MB in size. This causes gulp-csvtojson to crash. The problem seems to be that gulp-csvtojson attempts to read the entire file into a string which fails if the file size exceeds the maximum string length.

As this plugin is specific to csvtojson. I thought it might be useful to give the csvtojson contributors rights to this project.

In addition, they can add it to the readme of csvtojson to indicate a plugin exists.

Is this something you would want to do?

Further reviewing the code...it seems like there are too many paths for the plugin. Ideally there should be a single purpose; i.e. stream csvtojson module using gulp. The current plugin is streaming, but also will create a js file depending on options. The creation of a JS file could(should) be done in a different plugin. Here is the gulp plugin guidelines which provides a little more detail: Gulp Plugin Guidelines

I will open up another issue to discuss further.

Here is what currently exists:

var gulp = require('gulp');
var csvtojson = require('gulp-csvtojson');

gulp.task('default', function () {
    return gulp.src('src/file.csv')
        .pipe(csvtojson())
        .pipe(gulp.dest('dist'));
});

I think this should be:

var gulp = require('gulp');
var csvtojson = require('gulp-csvtojson');
var insert = require('gulp-insert'); //module to insert into streamed data


gulp.task('default', function () {
    return gulp.src('src/file.csv')
        .pipe(csvtojson())
        .pipe(insert.prepend('var anyVariable = '))
        .pipe(insert.append(';'))
        .pipe(gulp.dest('dist'));
});

Gulp Insert plugin

The current plugin doesn't handle streams. Ideally Gulp plugins should handle the three types of vinyl:

• Streams
• Buffers
• Empty (null) - Useful for things like rimraf, clean, where contents is not needed.

https://github.com/gulpjs/gulp/blob/master/docs/writing-a-plugin/README.md