Git Product home page Git Product logo

Comments (10)

mrodrig avatar mrodrig commented on May 18, 2024 1

Since I haven't heard back in a while, I'm going to close this out. I've refactored json2csv and csv2json, which will address the issue you initially reported. I'm working on updating refactoring the tests and should hopefully have 3.0.0 published to NPM before January 1st.

from json-2-csv.

mrodrig avatar mrodrig commented on May 18, 2024

Hi @NilapuAnusha. Thanks for reporting this. I took a look and this is the output that I'm seeing with the data that you provided:

name,title,description.type,description.count,description.message
ABC,title 1,abc,3,message 1
PQR,title 1,pqr,3,message 1
UVW,title 1,uvw,3,message 1

Here's the code that I'm using to generate that:

let converter = require('json-2-csv');

const data = [
     {name: 'ABC', title: 'title 1', description: {type: 'abc', count: 3, message: 'message 1'}},
     {name: 'PQR', title: 'title 1', description: {type: 'pqr', count: 3, message: 'message 1'}},
     {name: 'UVW', title: 'title 1', description: {type: 'uvw', count: 3, message: 'message 1'}}
];

let json2csvCallback = function (err, csv) {
    if (err) throw err;
    console.log(csv);
};

converter.json2csv(data, json2csvCallback);

Are you seeing different CSV output? If so, could you please provide me with some code to replicate the issue you're seeing?

Thanks!

from json-2-csv.

NilapuAnusha avatar NilapuAnusha commented on May 18, 2024

Thanks @mrodrig
Description key is object type with varying keys and keys may contain object type values

const data = [
{name: 'ABC', title: 'title 1', description: {type: 'abc', count: 3, message: 'message 1'}},
{name: 'PQR', title: 'title 1', description: {type: 'pqr', count: 3, message: {type: def, count: 5}},
{name: 'UVW', title: 'title 1', description: {type: 'uvw', count: 3, title: "Test"}}
];

Can you let me know the expected output of above one?

from json-2-csv.

mrodrig avatar mrodrig commented on May 18, 2024

@NilapuAnusha Good test case! In that particular scenario, the module will, by default, throw an error because the schemas are different for the different objects: Unhandled rejection Error: Not all documents have the same schema.

However, there is a way around that error. If you specify checkSchemaDifferences: false in the options object, then the module will convert everything that's available into a CSV and you'll end up with the following result:

name,title,description.type,description.count,description.message,description.message.type,description.message.count,description.title
ABC,title 1,abc,3,message 1,null,null,null
PQR,title 1,pqr,3,def,5,def,5,null
UVW,title 1,uvw,3,null,,,Test

Here's the updated example code:

let converter = require('json-2-csv');

const data = [
        {name: 'ABC', title: 'title 1', description: {type: 'abc', count: 3, message: 'message 1'}},
        {name: 'PQR', title: 'title 1', description: {type: 'pqr', count: 3, message: {type: 'def', count: 5}}},
        {name: 'UVW', title: 'title 1', description: {type: 'uvw', count: 3, title: "Test"}}
];

let json2csvCallback = function (err, csv) {
    if (err) throw err;
    console.log(csv);
};

converter.json2csv(data, json2csvCallback, {
        checkSchemaDifferences: false
});

Hope this helps!

from json-2-csv.

NilapuAnusha avatar NilapuAnusha commented on May 18, 2024

Thanks @mrodrig
But I can see more data columns compared header column and data is messed up(ex: same cell contains two inputs :( )
Could you please check for below example?

 [
     {"eventId": 185919,
      "productId": 1,
      "productName": "ABC",
      "cId": "10696e00-dcd5-11e8-9075-0e75adsh",
      "apiName": "Search",
      "userId": 10000001,
      "logSource": "get:/search/data",
      "logLevel": "Activity",
      "logType": "Success",
      "dateCreated": "2018-10-31T07:40:01",
      "processingTimeMs": 309,
      "clientAddress": "111.00.00.000",
      "message": {type: "Search", countries: "nl", query:{}}
    },
    {
      "eventId": 185920,
      "productId": 1,
      "productName": "Search",
      "cId": "10232e00-dcd5-11e8-9075-0p75adsh",
      "apiName": "Search",
      "userId": 100000002,
      "logSource": "/get:\\/companies:data.xml:113 (HttpException)",
      "logLevel": "Activity",
      "logType": "Exception",
      "dateCreated": "2018-10-31T05:19:46",
      "processingTimeMs": 385,
      "clientAddress": "111.00.00.000",
      "message": {orderType: "Search", countries: "nl", query:{pageSize: "10]", tradeName: "apple", page: "1"}}
    },
    {
      "eventId": 185921,
      "productId": 1,
      "productName": "Search",
      "cId": "10985e00-dcd5-11e8-9075-0p750dsh",
      "apiName": "search",
      "userId": 100000005,
      "logSource": "/get:\\/companies:data.xml:113 (HttpException)",
      "logLevel": "Activity",
      "logType": "Exception",
      "dateCreated": "2018-10-31T07:19:55",
      "processingTimeMs": 237,
      "clientAddress": "111.00.00.000",
      "message": {orderType: "Search", countries: "nl", query:{pageSize: "10", tradeName: "apple", page: "1"}}
    },
    {
      "eventId": 185922,
      "productId": 1,
      "productName": "Search",
      "cId": null,
      "apiName": "search",
      "userId": 1000000008,
      "logSource": "get:/companies",
      "logLevel": "Activity",
      "logType": "Success",
      "dateCreated": "2018-10-31T08:20:02",
      "processingTimeMs": 585,
      "clientAddress": "111.00.00.000",
      "message": {orderType: "Search", countries: "nl", query:{tradeName :"apple", name: "apple", pageSize: "10", page: "1", id: null, type: null}, totalResults: 372}
    },
    {
      "eventId": 185923,
      "productId": 1,
      "productName": "Search",
      "cId": "015l5170-eryt-11e8-9075-06yusdb427c2",
      "apiName": "search",
      "userId": 1000000002,
      "logSource": "/get:\\/companies:data.xml:113 (HttpException)",
      "logLevel": "Activity",
      "logType": "Exception",
      "dateCreated": "2018-10-31T08:20:17",
      "processingTimeMs": 201,
      "clientAddress": "111.00.00.000",
      "message": {orderType: "Search", countries: "fr", query: {pageSize: "10", tradeName: "apple", page: "1"}}
    },
    {
      "eventId": 185924,
      "productId": 1,
      "productName": "Search",
      "cId": null,
      "apiName": "search",
      "userId": 1000000008,
      "logSource": "get:/companies",
      "logLevel": "Activity",
      "logType": "Success",
      "dateCreated": "2018-10-31T06:20:31",
      "processingTimeMs": 339,
      "clientAddress": "111.00.00.000",
      "message": {orderType: "Search", countries: "fr", query: {tradeName: "apple", name: "apple", pageSize:"10", page:"1", id:null, type:null}, totalResults:599}
    },
    {
      "eventId": 185925,
      "productId": 1,
      "productName": "Search",
      "cId": "10677e00-dcd5-11e8-9075-0p75adsh",
      "apiName": "cs-ggs-p-search",
      "userId": 100000008,
      "logSource": "/get:\\/companies:data.xml:113 (HttpException)",
      "logLevel": "Activity",
      "logType": "Exception",
      "dateCreated": "2018-10-31T06:20:38",
      "processingTimeMs": 120,
      "clientAddress": "111.93.14.242",
      "message": {orderType: "Search", countries: "fr", query:{pageSize: 10,tradeName:"apple",page:"1"}}
    }
]

from json-2-csv.

mrodrig avatar mrodrig commented on May 18, 2024

@NilapuAnusha good point about the last example you provided having more data columns than the header. That is unfortunately what the module will currently provide by default since I haven't gotten around to defaulting to use quotes to wrap values. Here's some updated output and example code for that last example:

"name","title","description.type","description.count","description.message","description.message.type","description.message.count","description.title"
"ABC","title 1","abc","3","message 1","null","null","null"
"PQR","title 1","pqr","3",""def","5"","def","5","null"
"UVW","title 1","uvw","3","null","","","Test"
converter.json2csv(data, json2csvCallback, {
        checkSchemaDifferences: false,
        delimiter: { wrap : '"' }
});

Using the quote wrap option, it'll output:

"eventId","productId","productName","cId","apiName","userId","logSource","logLevel","logType","dateCreated","processingTimeMs","clientAddress","message.type","message.countries","message.query","message.orderType","message.query.pageSize","message.query.tradeName","message.query.page","message.query.name","message.query.id","message.query.type","message.totalResults"
"185919","1","ABC","10696e00-dcd5-11e8-9075-0e75adsh","Search","10000001","get:/search/data","Activity","Success","2018-10-31T07:40:01","309","111.00.00.000","Search","nl","","null","null","null","null","null","null","null","null"
"185920","1","Search","10232e00-dcd5-11e8-9075-0p75adsh","Search","100000002","/get:\/companies:data.xml:113 (HttpException)","Activity","Exception","2018-10-31T05:19:46","385","111.00.00.000","null","nl",""10]","apple","1"","Search","10]","apple","1","null","null","null","null"
"185921","1","Search","10985e00-dcd5-11e8-9075-0p750dsh","search","100000005","/get:\/companies:data.xml:113 (HttpException)","Activity","Exception","2018-10-31T07:19:55","237","111.00.00.000","null","nl",""10","apple","1"","Search","10","apple","1","null","null","null","null"
"185922","1","Search","","search","1000000008","get:/companies","Activity","Success","2018-10-31T08:20:02","585","111.00.00.000","null","nl",""apple","apple","10","1","",""","Search","10","apple","1","apple","","","372"
"185923","1","Search","015l5170-eryt-11e8-9075-06yusdb427c2","search","1000000002","/get:\/companies:data.xml:113 (HttpException)","Activity","Exception","2018-10-31T08:20:17","201","111.00.00.000","null","fr",""10","apple","1"","Search","10","apple","1","null","null","null","null"
"185924","1","Search","","search","1000000008","get:/companies","Activity","Success","2018-10-31T06:20:31","339","111.00.00.000","null","fr",""apple","apple","10","1","",""","Search","10","apple","1","apple","","","599"
"185925","1","Search","10677e00-dcd5-11e8-9075-0p75adsh","cs-ggs-p-search","100000008","/get:\/companies:data.xml:113 (HttpException)","Activity","Exception","2018-10-31T06:20:38","120","111.93.14.242","null","fr",""10","apple","1"","Search","10","apple","1","null","null","null","null"

from json-2-csv.

NilapuAnusha avatar NilapuAnusha commented on May 18, 2024

Thanks @mrodrig
When I download same data in file, I got below one
screenshot from 2018-11-22 11-10-40

Column data breaks from message.query

from json-2-csv.

mrodrig avatar mrodrig commented on May 18, 2024

Sorry for the delayed response. I'm working on updating the module to v3.0.0 and will take a look at fixing this in that release.

from json-2-csv.

mrodrig avatar mrodrig commented on May 18, 2024

Hi @NilapuAnusha. Since I'm in the process of working on v3.0.0 of the module and just finished up getting the majority of the functionality working with RFC 4180 support, I wanted to send you the CSV output of the last data set for you to check. If you wouldn't mind looking at the generated CSV and verifying that it matches your expected output, I would greatly appreciate it.

Here's the output from my local v3.0.0 json2csv code:

eventId,productId,productName,cId,apiName,userId,logSource,logLevel,logType,dateCreated,processingTimeMs,clientAddress,message.type,message.countries,message.query,message.orderType,message.query.pageSize,message.query.tradeName,message.query.page,message.query.name,message.query.id,message.query.type,message.totalResults
185919,1,ABC,10696e00-dcd5-11e8-9075-0e75adsh,Search,10000001,get:/search/data,Activity,Success,2018-10-31T07:40:01,309,111.00.00.000,Search,nl,{},undefined,undefined,undefined,undefined,undefined,undefined,undefined,undefined
185920,1,Search,10232e00-dcd5-11e8-9075-0p75adsh,Search,100000002,/get:\/companies:data.xml:113 (HttpException),Activity,Exception,2018-10-31T05:19:46,385,111.00.00.000,undefined,nl,"{""pageSize"":""10]"",""tradeName"":""apple"",""page"":""1""}",Search,10],apple,1,undefined,undefined,undefined,undefined
185921,1,Search,10985e00-dcd5-11e8-9075-0p750dsh,search,100000005,/get:\/companies:data.xml:113 (HttpException),Activity,Exception,2018-10-31T07:19:55,237,111.00.00.000,undefined,nl,"{""pageSize"":""10"",""tradeName"":""apple"",""page"":""1""}",Search,10,apple,1,undefined,undefined,undefined,undefined
185922,1,Search,null,search,1000000008,get:/companies,Activity,Success,2018-10-31T08:20:02,585,111.00.00.000,undefined,nl,"{""tradeName"":""apple"",""name"":""apple"",""pageSize"":""10"",""page"":""1"",""id"":null,""type"":null}",Search,10,apple,1,apple,null,null,372
185923,1,Search,015l5170-eryt-11e8-9075-06yusdb427c2,search,1000000002,/get:\/companies:data.xml:113 (HttpException),Activity,Exception,2018-10-31T08:20:17,201,111.00.00.000,undefined,fr,"{""pageSize"":""10"",""tradeName"":""apple"",""page"":""1""}",Search,10,apple,1,undefined,undefined,undefined,undefined
185924,1,Search,null,search,1000000008,get:/companies,Activity,Success,2018-10-31T06:20:31,339,111.00.00.000,undefined,fr,"{""tradeName"":""apple"",""name"":""apple"",""pageSize"":""10"",""page"":""1"",""id"":null,""type"":null}",Search,10,apple,1,apple,null,null,599
185925,1,Search,10677e00-dcd5-11e8-9075-0p75adsh,cs-ggs-p-search,100000008,/get:\/companies:data.xml:113 (HttpException),Activity,Exception,2018-10-31T06:20:38,120,111.93.14.242,undefined,fr,"{""pageSize"":10,""tradeName"":""apple"",""page"":""1""}",Search,10,apple,1,undefined,undefined,undefined,undefined

Hopefully that's much closer to the output that you are expecting than the current v2.4.0 json2csv code. I still need to revamp the csv2json functionality in order to get it up to RFC 4180 compliance, which might take a bit (since I'm only able to work on this after work or on the weekends). If you need the new code sooner though, please let me know and I can work to finalize the last few items I need to address in json2csv and then I can look into releasing a "beta" version of the module.

Thanks again for your help to improve the module's functionality!

from json-2-csv.

mrodrig avatar mrodrig commented on May 18, 2024

Version 3.0.0 is available in beta form on NPM as 3.0.0-beta.0 and should be merged at some point soon in #96.

from json-2-csv.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.