I'm importing up to 100 MB of data using this library and I had the following use-case:

Since I didn't know which escape- or quote character the file had which I'm getting from a customer on a frequent basis, I set this to null in the configuration - which I thought - would never appear. I thought this would affectively remove this feature of quoting and escaping.

As Fortuna decided to hit me hard today, I got hands on a file where the first character is a null-byte.

Long story short: This library tried to put all the 100 MB file into the first column and the rest of my script pushed for inserting it into the database. The BSON parser for my MongoDB fell face down by this amount of data stating Attempt to write outside buffer bounds at Buffer.write (buffer.js:881:13) at serializeObject [...].

As MongoDB anyways has a limit of 16MB per document, I would like this library to be able to catch a ridiculous data-set like this even before it attempts to load all the data into memory, parse it through my validators and finally let the database (or it's parser) crash.

Would it take long time to implement a double-check mechanism which throws an error if the data, read for a single row or column, exceeds a configured amount of bytes or characters?

I have a CSV file with very descriptive column headers, e.g. "First Name", "Country of Residence", etc, so I use the "headers" option to the csv() constructor to specify shorter names, e.g. "first" and "country". However, doing so causes the first line to be emitted as actual data. Can we add a "skipFirstLine" option to prevent this?

Right now there is no way to support non standard multi-byte line endings because newline is treated as only a single character.

Hi there,

I totally hate encodings. I had a csv in utf16-le, that produced weird output when piping it into csv-parser (like \u0000o\u0000r\u0000 \u0000t\u0000h\u0000e\u0000). I could solve it by encoding the file to utf-8. However maybe there is a way that it will also work for this encoding. I will look into this.

Best,
Finn

How do I reread the csv file on the 'end' function?

fs.createReadStream('data.csv')
    .pipe(stream)
    .on('data', function (data) {
    //do something
    }).on('end', function () {
    //How do I reread the csv again
    });

Hey folks,

Noticed I couldn't use csv-parser when parsing a file that looks like this:

"col1","col2"
"hello","Hey, this is a longe string that has\na few new lines in it\n."

Not sure if it's truly valid CSV, but in my experience – it's common practice.

Is there a way to go around this somehow? If not – I may have to modify csv-parser to support this.

$ echo "a
1
2" | csv-parser | head -n 1
{"a":"1"}
events.js:160
      throw er; // Unhandled 'error' event
      ^

Error: write EPIPE
    at exports._errnoException (util.js:1026:11)
    at WriteWrap.afterWrite (net.js:799:14)

Not consuming the entire input seems to trigger an error.

In README.md you are missing a comma after the newline: '\n' in the block:

var stream = csv({
raw: false, // do not decode to utf-8 strings
separator: ',', // specify optional cell separator
newline: '\n' // specify a newline character
headers: ['index', 'message'] // Specifing the headers
})

I know this is kinda silly, but we basically got DOS'd by csv-parser last night.

We use csv-parser for reading whatever CSV files get uploaded to us. One such file was malformed so that it only had two lines. They forgot to include newline separators between rows of data. Line 1 was a header with 3 columns. Line 2 was 800mb of text.

Making things worse, we pipe the contents into csv-parser from the network. They seem to come in 1000 byte packets. It looks like _transform is adding the packet to buf. To make the problem clear, once it gets to 400mb in the buf, it is going to allocate and deallocate 400+mb about 400,000 times.

I will try to create a PR if I have time, but wanted to get this ticket out there since it's a DOS vulnerability and somewhat hard to protect against without modifications to the library.

Using 1.12.0 here, seeing a weird issue. If to project this problem on the sample:

var csv = require('csv-parser')
var fs = require('fs')

fs.createReadStream('some-csv-file.csv')
  .pipe(csv())
  .on('data', function (data) {
    console.log('Name: %s Age: %s', data.NAME, data.AGE)
  })

if AGE is the last header field, it's not possible to access it via data.AGE, but it is possible to access it via data['"AGE"']. What's more interesting is that it happens only with the last header field. So, as a workaround it's possible to add a comma at the end of the header field (which adds a new, zero-length header field). This gets data.AGE (or data['AGE']) to return the expected value.

Anyone else seen this issue?

Reproduced easily with a CSV from government:
http://earthquake.usgs.gov/earthquakes/feed/v1.0/summary/all_month.csv

var csv   = require('csv-parser')
var fs    = require('fs')
var file  = 'all_month.csv'
var count = 0;

fs.createReadStream(file).pipe(csv())
    .on('data', () => count++)
    .on('finish', () => console.log(count));

With 1.8.0: 7303 line read (all of them)
With 1.9.0: 5190 line read

This is a huge issue guys. I cannot investigate right now but you should really add big CSV files to your test suite.

I am using command line csv-parser and the csv file that I am processing is not converted in full, it is stopping in the middle, by looking at the fields, one has normal double quote in (") embedded in the text area (" exactly), I have removed it for test purpose, and now csv-parser goes through all records. This cannot be the final solution for me, as I have other records like this.

I created the below test file which is showing the same issue.

test.csv:

"Field1","Field2","Field3","Field4"
"UN","DEUX","TROIS","QUATRE"
"ONE","T\"WO","THREE","FOUR"
"ABC","DEF","GHI","JKL"

c:>csv-parser test.csv
{"Field1":"UN","Field2":"DEUX","Field3":"TROIS","Field4":"QUATRE"}
{"Field1":"ONE","Field2":"T\\\"WO","Field3":"THREE","Field4":"FOUR\"\r\n\"ABC"}

We are missing the 3rd record, something happened with record 2 and 3 been merged, see ABC is inside record 2.

thanks.

if specified, dont parse columns that arent in the list

One should likely be updated

Hi ,
I have a code which make use of csv-parser and validates the csv file. I'm not using the strict:true option. I'm checking some checks with the headers(like valid headers which my application expects) and parse data in data event etc. However, Can you please let me know how to skip processing data event when I see some error in "header" event?

Ex: fileStream.pipe(csv()) .on('headers', (headers) => { }). on('data',(data) => { }). on('end', () => { });

I've tried out this module with some sample data and I get no output: http://www.briandunning.com/sample-data/

I see this:

Row {
  ' Time': '0.0500',
  ' Lateral Position': '8.25',
  ' Sagittal Position': '-14.91',
  ' Twisting Position': '-7',
  ' Lateral Velocity': '7.66',
  ' Sagittal Velocity': '1.96',
  ' Twisting Velocity': '0.33',
  ' Lateral Acceleration': '82.13',
  ' Sagittal Acceleration': '24.14',
  ' Twisting Acceleration': '22.79' }

why not this:

Row {
  'Time': '0.0500',
  'Lateral Position': '8.25',
  'Sagittal Position': '-14.91',
  'Twisting Position': '-7',
  'Lateral Velocity': '7.66',
  'Sagittal Velocity': '1.96',
  'Twisting Velocity': '0.33',
  'Lateral Acceleration': '82.13',
  'Sagittal Acceleration': '24.14',
  'Twisting Acceleration': '22.79' }

it must because the data I am feeding you is bad right? duh

Sometimes the headers include whitespace like

a, b , c,  d
1, 2 ,3 , 4

In these cases they shouldn't be parsed as ' b' but be ' b'.trim() instead.

is there an event to denote end of file, like '.on()'

Here is example of data in my CSV file (I got this file from odt, xls).
"FirstName","LastName","countMin", "countMax"
"FirstNameTest","LastNameTest",="01",="01"

So Library Office is nice to open this file, but csv-parser has respond me error.

I was just doing some experiments with using this module in conjunction with @maxogden's websocket-stream module and hit a couple of roadblocks with regards to the way the buffers were handled. Here's an example of the client code I'm writing:

var websocket = require('websocket-stream');
var wss = websocket('ws://localhost:3000', { type: Uint8Array });
var csv = require('csv-parser');

wss.pipe(csv())
  .on('data', function(item) {
    console.log(item.Year);
  })
  .on('end', function() {
    console.log('done');
  });

It's more an experiment than anything else, and while csv-parser failed to work as expected in the first instance I found that using @feross's typedarray-to-buffer module to do a data = toBuffer(data) call early in the _transform method made everything play nicely...

Anyway, I've got the code in a branch here:

https://github.com/DamonOehlman/csv-parser/tree/browser-streaming-fix

LMK if you'd like a PR or if you want to take a different approach :)

See here: max-mapper/binary-csv#7.

Basic idea: allow unescaped quotes to compensate for poorly formatted data.

It seems that this was a planned feature of ndjson that never got implemented: https://github.com/maxogden/ndjson/blob/f6184f77d08640f0a67f9b0f7ddb0d1db89e5168/index.js#L26-L30

csv:

"col1","col2"
"aaa","test 32" 123"
"bbb","sss"

expected output:

{ col1: 'aaa', col2: 'test 32" 123' }
{ col1: 'bbb', col2: 'sss' }

current output:

{ col1: 'aaa', col2: 'test 32" 123"\n"bbb' }

code:

var csv = require('csv-parser');

fs.createReadStream('test3.csv')
        .pipe(csv())
        .on('data', function (data) {
            console.log('done', data);
        });

I get a Error: Row length does not match headers if a row's length doesn't match its headers, which causes parsing of the whole file to fail. Is there not a way to log the invalid row with something like an error or invalid event, allowing the rest of the file's parsing to continue?

For example, if I have a column called "Date" with values like:
4/23/12 18:25
I would like to provide a simple converter to have value in the json file look like this:
2012-04-23T18:25:43.511Z

Wouldn't it be more correct to call it dsv-parser since any character can be used? -http://en.wikipedia.org/wiki/Delimiter-separated_values

Currently only the "data" event is documented. Please document all the events that are also emitted. Such as "end", "headers" etc.

I am trying to send a large CSV file to the client using Hapi.js. I get this error when I try to hit the route:

FATAL ERROR: JS Allocation failed - process out of memory

server.route({
    method: 'GET',
    path: '/scheduleA',
    handler: function(req, res){
            var dfd = q.defer();
            var contributions = [];
            fs.createReadStream('./data/Form_460_-_Schedule_A_-_Monetary_Contributions.csv')
              .pipe(csv())
              .on('data', function(cont){
                contributions.push(cont)
              })
              .on('error', function(err){
                dfd.reject(err);
              })
              .on('end', function(){
                dfd.resolve(contributions);
              })

            dfd.promise.then(function(conts){
                res(conts)
            })
        }
    });

Can I send the full file to the client or does this error mean the json is too big?

bcsv is kinda confusing to explain, but i think csv-parser on the cli would make more sense. wanna write a simple cli API to convert csv to newline delimited json?

csv-parser does not keep headers order.

for example:

sign;2015;2016
aaa;12;42

produces:

[
    {
        "2015": "12",
        "2016": "42",
        "sign": "aaa"
    }
]

Right now it only allows specifying a separator. Would be nice if it would support a custom newline option as well.

I'm parsing a csv file that has an account number column with accounts from lengths ranging from 10 - 12 characters. It appears the parser is padding 0's in from of the records that have 10 or 11 characters. For example 55555555555 is being written as 005555555555. Is there any way to have this written as is? I'm writing the results to mongodb and my find queries aren't returning results due to the mismatch.

My work around currently is to check if the number starts with a 0 then remove it. This doesn't add too much overhead but I figured I could be missing something. Thanks!

I'm on a mobile, otherwise I'd do a PR, but please move the testing deps to devDependencies.

I try to process big file (23K lines). Only around 13K lines are processed (i.e. "data" even is raised). And no error events are raised, no exceptions throw.
File is available by link (zipped): http://download.geonames.org/export/dump/cities15000.zip
My code:

function csvToMap(filename, columns, rowCallback) {
    return new Promise(function(resolve, reject) {
        try {
console.log(filename);
            var linesRead = 0;
            fs.createReadStream(filename)
                .pipe(csv({
                    separator: '\t',
                    headers: columns
                }))
                .on('data', function(data) {
                    try {
                        rowCallback(data);
                        linesRead++;
                    } catch (e) {
                        die(e, reject);
                    }
                })
                .on('end', () => {
                    console.log(filename + ": " + linesRead + " lines read");
                    resolve();
                })
                .on('error', (e) => {
                    die(e, reject);
                });

        } catch (e) {
            die(e, reject);
        }
    });
}

function die(msg, callback) {
    console.log("error: " + msg);
    if (callback) {
        callback(msg);
    }
    throw msg;
}

Hey,

I ran into some issues with csv-join, because the CSV I was piping was a String Buffer. I am not sure weather you want to allow usage of Strings here, but if not, maybe there could be a hint about it in the docs.

Best,
Finn

Using the following code, will return the csv object and not the readStream object.
What is the best to close the stream at any time?

var rs = fs.createReadStream(fileToProcess)
                .pipe(csv())
                .on('headers', function (header)
                {
                    ....
                    rs.destory();
                })
....

Assuming you have an "headers" csv field, the following code overwrites its values with the file's headers.

I strongly suggest leaving the _Row.prototype safe from any additional property.

certain terrible programs indicate omited fields via shouting null at you instead of just having an empty field, e.g.

a,b,c,d
foo,"bar,baz",NULL,thing

instead of

a,b,c,d
foo,"bar,baz",,thing

it does understand quoting as fields with a , in them are quoted leading me to suspect a field containing the text null in all caps would be rendered as "NULL"

Double quotes inside double-quoted values cause the parser to stop parsing at the row containing the double quote. When I run this:

printf 'name,age\njoe,40\n"will"iam",30\n"sam",35\n"jan",25' | csv-parser
-------------------------------^ there's a double quote in here

I get this:

{"name":"joe","age":"40"}
{"name":"will\"iam","age":"30\n\"sam\""}

It escaped the double quote nicely, but the "age" field is wrong, and the third and fourth rows were never parsed at all.

You should use semicolons on javascript code:

http://stackoverflow.com/questions/2399935/why-use-semicolon

when running csv-parser, it misses a module

module.js:338
    throw err;
          ^
Error: Cannot find module 'is-property'
    at Function.Module._resolveFilename (module.js:336:15)
    at Function.Module._load (module.js:278:25)
    at Module.require (module.js:365:17)
    at require (module.js:384:17)
    at Object.<anonymous> (/.../node_modules/csv-parser/node_modules/generate-object-property/index.js:1:80)
    at Module._compile (module.js:460:26)
    at Object.Module._extensions..js (module.js:478:10)
    at Module.load (module.js:355:32)
    at Function.Module._load (module.js:310:12)
    at Module.require (module.js:365:17)

I have an exported CSV file here with 2.5 bln rows in it, each row at the moment contains only one column of length about 10 symbols.

When I stream it through fs.createReadStream, the entire stream finishes processing in 2.4 seconds.

When I redirect the stream into your library, using .pipe(csv()), well, I hit Ctrl+Break after 10 mins of it still trying to finish. It is just that slow.

UPDATE: I am processing the stream in paused mode, and I noticed that once I'm getting readable event, the following read always returns just one row. This is what seems to create the major bottleneck.

Is it possible to make this library buffer more than one row at a time, so we can make multiple read-s on each readable to get many rows at once?

Inherently there's no data types in a csv, so when each csv row in parsed into a csv object numeric values end up as string values. It would be nice to have an option to convert to numeric values when possible. See: https://www.npmjs.com/package/csvtojson#field-type

Hello,

I would like to report that not every entries are processed in big CSV files (when using with pipe at least).

I tried https://github.com/Keyang/node-csvtojson and it now works fine. I recommend you to use it instead.

Simple example

id
""""""
"test"

Expected result

{ id: '""' }
{ id: 'test' }

Actual result

{ id: '"""\n"test' }

var stream = csv(['index', 'message'])
// Source from somewere with format 12312,Hello World
origin.pipe(stream)
  .on('data', function(data) {
    console.log(data) // Should output { "index": 12312, "message": "Hello World" }
  })

I tried this but not output as example comment

var stream = csv(['index', 'message'])
// Source from somewere with format 12312,Hello World
var origin = fs.createReadStream('test_csv_parse.csv');
origin.pipe(stream)
  .on('data', function(data) {
   // Should output { "index": 12312, "message": "Hello World" } 
   // but I got [ '12312', 'hello world' ]
    console.log(data)   })

What I'm wrong?

e.g.

require('csv-parser')(csvBuffer) // returns array of parsed objects

This isn't usable. 2800 translates to 28, 10 translates to 1, etc. Also as someone else noted I encountered some sort of padding bug where 5 became 05.