I would like to know how can I convert the json string to a csv string without saving the files to the computer.
The code I'm using reads and writes to PC files, but I'd like to use it without dealing with the files.

The code that I was using is this:

using (var r = new ChoJSONReader("sample.json"))
{
 using (var w = new ChoCSVWriter("sample.csv").WithFirstLineHeader())
 {
  w.Write(r);
 }
}

I would like a code like this:

public string json2CSV(string jsonData)
{
 string csvData;
 //Convert jsonData and save in csvData...
 //...
 return csvData;
}

Hey love your framework but I need some help to achieve something.

I have a large dataset and need to use AsDataReader() and send the reader to SQL Bulk Copy.

Each data file is different and so we will be using a varchar max field to hold the columns read in as JSON.
I need to transform each record to a custom object. I have been through all the samples, even trying the before record read event/interface to change the source, but haven't had any luck.

The POCO object will look like this: (it an alternately be a DataRow object with same properties)

class ImportRow
{
int ImportId {get;set;}
int RowIndex {get;set;}
string fields {get;set;}
}

So if the data comes in like this:
"acme" "1" "1/1/2015"
"contoso" "34" "1/2/2018"

We should get the following objects when calling read()

{
ImportId:42
RowIndex:0
Fields "{[{value:"acme"}, {value:"1"}, {value:"1/1/2015"}]}"
},
{
ImportId:42
RowIndex:1
Fields "{[{value:"contoso"}, {value:"34"}, {value:"1/2/2018"}]}"
}

Please help!

When a string has a Double Quote Char it needs to be doubled up...

class EmployeeRecSimple
{
    public int Id { get; set; }
    public string Name { get; set; }
}

static void Main(string[] args)
{
    List<EmployeeRecSimple> objs = new List<EmployeeRecSimple>()
    {
        new EmployeeRecSimple() { Id = 20, Name = "John Smith" },
        new EmployeeRecSimple() { Id = 21, Name = "Jack in \"Da Box" }
    };
    Console.WriteLine(ChoCSVWriter.ToText(objs));
    Console.ReadLine();
}

the output is:

20,John Smith
21,Jack in "Da Box

the second line needs to be change to...

21,"Jack in ""Da Box"

can i load the response of json from an api ???
@Cinchoo

Some configurations are not considered during the CSV file reading:

• IgnoreDuplicateFields

• IgnoreEmptyLine

    	using (var reader = new ChoCSVReader<Class1>(filePath)
    		.Configure(c => c.IgnoreDuplicateFields = true)
    		.Configure(c => c.IgnoreEmptyLine = true)
    		.WithFirstLineHeader())
    	{
    		csvFileAsDataTable = reader.AsDataTable();
    	}
  

Data example could be found in attachment:
CsvFile.txt

Cannot find a solution for my scenario. I've googled and read several sites explaining the XmlReader. Its unclear how to setup the attributes correctly to handle this situation. Thanks in advance.... This is a great library!!!!!

<SalesLead>
    <Customer>
         <Name part="first">Foo</Name>
         <Name part="last">Bar</Name>
    </Customer>
</SalesLead>

public class SalesLead
{
   public CustomerModel Customer {get;set;}
}

public class CustomerModel
{
    public IEnumerable<Name> Name {get;set;}
}

public class Name
{
    public string Part {get; set;}
    public string Value {get;set;
}

Hello, I gave choEtl a CSV file with 20 cols/headers in first row.

1. I tried both new ChoCSVReader(fileName) and new ChoCSVReader(fileName).WithFirstLineHeader() both dont have headers in them.
2. Can you also tell me if there is no Datatable, will ChoETL with the help of SqlBulkCopy create the table.?The default MSDN seems to indicate so, but when I see my Db there is no table created.

            using (SqlBulkCopy bcp = new SqlBulkCopy(dbConnection))
            {
                ...
                using (var p = new ChoCSVReader(fileName).WithFirstLineHeader())
                {
                    var dataTable = p.AsDataTable();
                    using (var cmd = dbConnection.CreateCommand())
                    {
                        dbConnection.Open();                        
                        try
                        {
                            var columnNames = new List<string>();
                            foreach(DataColumn c in dataTable.Columns)  // **** See picture, fails does not have col names or types
                            {
                                console.writeline (c.ColumnName);
                            }                          
                        }
                        catch (Exception ex)
                        {
                            var temp = ex.Message;
                        }
                    }

                    bcp.DestinationTableName = tableNameWithoutSpaces;
                    bcp.EnableStreaming = EnableStreaming;
                    bcp.BatchSize = BatchSize;
                    bcp.BulkCopyTimeout = BulkCopyTimeout;
                    bcp.NotifyAfter = NotifyAfter;
                    bcp.SqlRowsCopied += delegate (object sender, SqlRowsCopiedEventArgs e)
                    {
                        Console.WriteLine(e.RowsCopied.ToString("#,##0") + " rows copied to Sql Table ");
                    };
                    bcp.WriteToServer(dataTable); // ****fails, does not create table
                }
            }

Hi
I modified a bit the example in the tutorial https://www.codeproject.com/Articles/1171289/Cinchoo-ETL-Xml-Reader, and I'm trying to load the following XML

<Employees>
    <Employee Id='1'>
        <Name>Tom</Name>
        <Address>
            <AddressLine Id='1'>XYZ road</AddressLine>
            <AddressLine Id='2'>MiceTown</AddressLine>
        </Address>
    </Employee>
    <Employee Id='2'>
        <Name>Mark</Name>
        <Address>
            <AddressLine Id='1'>123 street</AddressLine>
            <AddressLine Id='2'>TigerCity</AddressLine>
        </Address>
    </Employee>
</Employees>

I need to have 1+ AddressLine for each employee

I modified POCO as follow

    public partial class EmployeeRec
    {
        public int Id { get; set; }
        public string Name { get; set; }
        public AddressRec Address { get; set; }
    }

    public partial class AddressRec
    {
        public AddressLineRec[] AddressLines { get; set; }
    }
    
    public partial class AddressLineRec
    {
        public int Id { get; set; }
        public string AddressLine { get; set; }
    }

But when I read the XML, AddressLines Array is always empty
I Tried to specify [ChoXmlNodeRecordField(XPath=...] on my POCO fields, but with no success

I think I miss something important here :(

I saw the answer to Issue#14, but I hope I can have something less "dynamic"

any suggestion will be appreciated!

TIA

stefano

p.s. I attach full example
Program.cs.txt

Got anything for this? I know there is the the skip xx rows at top, but the headers span two rows. Its tab delimited, but fixed width from what I can tell. use the * as col widths

Headers should be this:
CUSTOMER ID, CUSTOMER NAME, INVOICE ID, PURCHASE date, PRODUCT ID, PURCHASED amount, PURCHASED QUANTITY, LOCATION ID

* Select	d  : 02:02:12 20 MAR 2017						
* Shippi	g Date >= 01/20/2017 ; Shipping Dat	<= 03/20/2017	; Shipping	Branch = 2	9,15,19,21,22,	5,26,27,2	,29,30,31,
********	***********************************	**************	**********	**********	**************	*********	**********
							
CUSTOMER	CUSTOMER NAME	INVOICE ID	PURCHASE	PRODUCT ID	PURCHASED	PURCHASED	LOCATION
ID			DATE		AMOUNT	QUANTITY	ID
22160	MANSFIELD BROTHERS HEATING & AIR	sss.001	02/08/2017	193792	69.374	2	30
27849	OWSLEY SUPPLY LLC  - EQUIPMENT	sss.001	03/14/2017	123906	70.409	1	2
27849	OWSLEY SUPPLY LLC  - EQUIPMENT	sss.001	03/14/2017	40961	10.000	1	2
16794	ALEXANDER GILMORE dba AL'S HEATING	sss.001	01/25/2017	116511	63.016	1	15
16794	ALEXANDER GILMORE dba AL'S HEATING	sss.001	01/25/2017	116511	-63.016	-1	15
16794	ALEXANDER GILMORE dba AL'S HEATING	sss.001	01/25/2017	122636	30.748	1	15
16794	ALEXANDER GILMORE dba AL'S HEATING	sss.001	01/25/2017	137661	432.976	1	15
16794	ALEXANDER GILMORE dba AL'S HEATING	sss.001	01/25/2017	137661	-432.976	-1	15

How to open a file and write data without overwriting the already saved data using ChoJSONWriter

Would be great to have a way to get some basic info:
FileName
FileSize
Row Count
Header Row Start Line Number
Data Row Start Line Number
Headers (List of names, and type)
Errors (List of errors by type with row numbers and message)

In the JsonWrite have option to put this object before the data

Is there a way to add a column that isnt in a CSV file, but is needed when writing to the table in SQL. I dont want to put it to a datatable as these files are >5gb and millions of rows, When we get a file in we give it a unique ID, then when we load to the table, we reference the ID the data came in on. I thought I could just give it a default value but it tells me its missing, is there an override or something I might of missed?

 var config = new ChoCSVRecordConfiguration
                                                   {
                                                       FileHeaderConfiguration =
                                                       {
                                                           HasHeaderRecord = true
                                                       }, Delimiter = "|", AutoDiscoverColumns = false, AutoDiscoverFieldTypes =  false, IsDynamicObject = true, ThrowAndStopOnMissingField = false
                                                   };

                
                config.CSVRecordFieldConfigurations.Add(new ChoCSVRecordFieldConfiguration("FileLogId", 0){DefaultValue = fileLogId, FallbackValue = fileLogId});
                
                
                foreach (var field in fileDefinitionFields.Where(x => x.FieldOrder > 0))
                {
                    config.CSVRecordFieldConfigurations.Add(new ChoCSVRecordFieldConfiguration(field.ColumnName, field.FieldOrder));
                }
                

                dynamic row;
                
                            using (var parser = new ChoCSVReader(blobFile.OpenRead(), config).AsDataReader())
                            {
                                bulkCopy.WriteToServer(parser);
                            }

I am trying to write a JSON string into CSV file. I have defined the following POCO class

public class Facility {
    [ChoJSONRecordField]
    public int? Id {get; set;}
    [ChoJSONRecordField]
    public string Name {get; set;}
    [ChoIgnoreMember] //Ignore Uuid
    public string Uuid {get; set;}
    [ChoJSONRecordField]
    public string CreatedAt {get; set;}
    [ChoJSONRecordField]
    public string UpdatedAt {get; set;}
    [ChoJSONRecordField]
    public bool Active {get; set;}
}

Here is the sample JSON string:

{"facilities":[{"id":39205, "name":"Sample1" ,"uuid":"ac2f3464-c425-4063-86ad-163521b1d610","createdAt":"2019-03-06T14:25:32Z","updatedAt":"2019-03-06T14:29:31Z","active":true},
{"id":35907,"name":"Sample2","uuid":"d371debb-f030-4c1e-b198-5eb562ceac0f","createdAt":"2019-02-21T09:33:25Z","updatedAt":"2019-02-21T09:33:25Z","active":true} ]}

I want to ignore Uuid field from being written in the CSV output. Here is how I am doing the conversion:

var memStream = new MemoryStream();
using (var r = ChoJSONReader<Facility>.LoadText(json).WithJSONPath("$..facilities")
.Configure(c => c.NullValue = "").Configure(c => c.IgnoredFields.Add("Uuid")))
{
    foreach (var facility in r)
    {
        var w = new ChoCSVWriter<Facility>(memStream).WithFirstLineHeader().Configure(c => 
               c.IgnoredFields.Add("Uuid"));
        w.Write(facility);
    }

    return memStream;
 }

However, the above code returns memStream without writting anything on it (memStream.Length equals 0)

What's wrong with the above code? Or is there a better way to do this?

I have a simple CSV file made of 3 field (pipe separated), the second of which is empty, like the following

some||1234

I'm trying to read it using

var reader = new ChoCSVReader(csvFilePath).WithDelimiter("|");

but getting a "At least one of the field header is empty." exception message in ChoETL.ChoCSVRecordReader.GetHeaders(String line)

I tried to use .WithFirstLineHeader(false) or a POCO object without any success.

any clue?
thanks

stefano

Currently, backslash escapes the following double quote, but it should not. For example, if I have the file like this (tab delimited):
"Field1" "Field2"
"Line 1 Field 1" "Line 1 Field 2"
"Line 2 Field 1" "Line 2 Field 2"

the reader for the second line should return:
reader[0] = "Line 1 Field 1\"
reader[1] = "Line 1 Field 2\"
but it returns:
reader[0] = "Line 1 Field 1\"
reader[1] = "Line 1 Field 2"\r\n"Line 2 Field 1".

As I recall, in order to escape the double quote character the second double quote should be used and the backslash.

Hi,

I noticed that there are two license files in the repo with different license terms:

1. https://github.com/Cinchoo/ChoETL/blob/master/License.txt (seems like a more recent one)
2. https://github.com/Cinchoo/ChoETL/blob/master/LICENSE (seems old and outdated)

NuGet Org points to the 2nd. Would be great if you can remove the old one and update the NuGet org link.

Cheers,
Guy

The latest version appends a double quote into the last field's name. Version 1.0.6.8 does not have this issue. I attached the sample project with the sample file. With v.1.0.6.8 the output will be [Field1] [Field2], with the latest (1.0.9.6) the output will be [Field1] [Field2"].

CsvReaderTest.zip

What are your thoughts on allowing multiple fixed (or delimited) fields specs and telling the parser which field spec to use in the RecordSelector callback? To deal with flat files that have multiple record layouts in a single file but without having to create a class for a layout.

I like the field specs because the potentially could be loaded at runtime from a file.

It's not showing up in the CSVWriter

Hi
I'm trying to install nuget package 1.0.8.8, but it seems that it cannot find System.ComponentModel.DataAnnotations.4.0.0

ChoETL 1.0.8.7 was not asking for a specific version

    <frameworkAssemblies>
      ..
      <frameworkAssembly assemblyName="System.ComponentModel.DataAnnotations" targetFramework="" />
      ..
    </frameworkAssemblies>

whereas 1.0.8.8 does require 4.0.0 (at least for .Net Framework 4.5

  <dependencies>
     <group targetFramework=".NETFramework4.5">
       ...
       <dependency id="System.ComponentModel.DataAnnotations" version="4.0.0" />
       ....
     </group>

I'm facing this error in my application.

I have simply started an empty asp.net 4.6.1 web application project and have installed nuget package.

Following is my code.

public class HomeController : Controller
    {
        // GET: Home
        public ActionResult Index()
        {
            StringBuilder bldr = new StringBuilder();
            foreach (var e in new ChoCSVReader("Emp.csv"))
                //Console.WriteLine(e.ToStringEx());
                bldr.AppendLine(e.ToStringEx());

            return Content(bldr.ToString());
        }
    }

But I'm facing error below

@Cinchoo can you pl guide?

https://github.com/Cinchoo/ChoETL/search?utf8=%E2%9C%93&q=WithCustomRecordTyoeCodeExtractor&type=

Currently using
https://github.com/ExcelDataReader/ExcelDataReader

I'm using a ChoCSVReader over a ZipInputStream and it fails because ZipInputStream isn't seekable.

Any harm in doing something like:

if( sr is StreamReader sr2 && sr2.BaseStream.CanSeek )
                sr2.Seek( 0, SeekOrigin.Begin );

I need to save a JSON to CSV and I'm trying to use ChoETL, but when I use ChoJSONReader, it takes just the first level of my json.
How do I read this JSON for multi-level, if it's dynamic? I don't know how many levels it will have.
I tried using .WithField and it worked, but it's dynamic.

I read all the objects from the file but the is issue is after reading ChoJSONReader is not releasing the file
foreach (dynamic objectData in new ChoJSONReader(fileName))
{
objectsFromFile.Add(objectData);
}
this is how I am using
The error I am getting: The process cannot access the file 'file.json' because it is being used by another process."}

I am trying to read a .csv file and passing it AsDataReader to a SqlBulkCopy command, but I get the error that a certain column doesn't allow DBNull values. But I checked the .csv file and the values for that column are ' ' (a white space).
Why does the reader convert it to null? How can I fix that?

Hi, I spent a lot of time looking at your help in codeproject, but I could only get this far.. please provide some guidance

I don't have a POCO, how can I get the column names and their types from the stream

[HttpPost]
public ActionResult UploadCsv(int Id, HttpPostedFileBase upload, int headerRowNumber)
{
            using (var choStream = upload.InputStream)
            using (var choReader = new StreamReader(choStream))

            //I cant get the parser to work, without a POCO / class, how can I just parse it.
                using (var parser = new ChoCSVReader(choReader).WithFirstLineHeader())
                    
            // How to get list of Cols? & Types
                List<String, Type>  listOfColumns = parser.???
     
           //or just list of Column Names
              List<String>  listOfColNames = ??

Before I begin, I should say I am very very very new to C#.

I am writing a program that will spit out a .csv file with a lot of information, then directly after I call the following convertFile method using ChoETL:

    public void convertFile(string inputFile, string outputFile)
    {
        using (var writer = new StreamWriter(outputFile))
        {
            int row = 0;
            writer.Write("[\r\n");

            foreach (var e in new ChoCSVReader(inputFile).WithHeaderLineAt())
            {
                writer.Write((row > 0 ? ",\r\n" : "") + e.DumpAsJson());
                writer.Flush();
                row++;
            }
            writer.Write("]");
            writer.Flush();
            writer.Close();
  
        }
        
    }
}

My program creates a csv file and json file, then appends each every 5 minutes. However I am unable to do this because I get "The process cannot access the file "csv file" because it is being used by another process." I have narrowed this down to ChoETL because if I remove the convertFile method at the end of my main, it will work just fine and append the csv file every 5 minutes without creating the json. Any help?

Hi, awesome lib. 👍 in my scenario many csv files have rows where the header could run upto 2nd or 3rd rows.

1. How can I set a start reader from a specific row like row X, (and let Chincoo know, to read after that, and also start maxscan rows after the header row, so it can discover the type of the col.)

public void ReadFileWithHeaderAt(int HeaderRowStart, string fileName)
        {
            using (var parser = new ChoCSVReader(fileName.ToString() + ".csv").WithHeaderLineAt(HeaderRowStart))
            {
                foreach (var t in parser)
                {
                    Console.WriteLine(t.Id);
                    Console.WriteLine(t.Date);
                }
            // Since, I dont know the types, how to auto discover while loading so I can create the typed columns
            //  and header starts in row 4 or 5 or 12 so.... but my configuration below doesnt help
            // ChoCSVReader(fileName.ToString() + ".csv").WithHeaderLineAt(HeaderRowStart).Configure(c => c.MaxScanRows = 10) 
            // at row does not work, *** how can I tell it scan after the header row till next 10 rows to discover the types?
             ... i.e. ultimately, how to create Auto discovered Typed SQL table(filename.ToString()); 
            }
        }
 }

2. Also, I saw a comment on discovering data-types here , how can I turn it on, so that I can use them to create the col, headers in the SQL table. Or, does this happen by default.
3. It says the library can transform data, can I query or Sum a col in this lib?

Just wanted to let you know, while reading your help, I found your codeproject articles/links was broken on your wiki page

thanks

I use the ChoFixedLengthReader<T>.LoadText(text), if my class Inherit IChoNotifyRecordRead, then will not be able to read text.

public class FixedLength : IChoNotifyRecordRead
{
    public bool SkipUntil(long index, object source)
    {
        // This is not work.
        return false;
        // Does that mean I can only use this?
        throw new NotImplementedException();
    }
}

I looked at the 'ChoFixedLengthRecordReader.cs' 72 lines, the 'skipUntil' will only ture?

bool? skipUntil = true;
bool? doWhile = true;

using (ChoPeekEnumerator<Tuple<long, string>> e = new ChoPeekEnumerator<Tuple<long, string>>(
    new ChoIndexedEnumerator<string>(source is IEnumerable<string> ? (IEnumerable<string>)source : 
    sr.ReadLines(Configuration.EOLDelimiter, Configuration.QuoteChar, false /*Configuration.MayContainEOLInData*/)).ToEnumerable(),
    (pair) =>
{
    //bool isStateAvail = IsStateAvail();
    skip = false;

    if (skipUntil != null)
    {
        if (skipUntil.Value)
        {
            skipUntil = RaiseSkipUntil(pair);
            if (skipUntil == null)
            {

            }
            else
            {
                if (skipUntil.Value)
                    skip = skipUntil;
                else
                    skip = true;
            }
        }
    }

I've found an issue that the library doesn't works on MacOS. It fails during initialization with following exception:

Exception Type: System.DllNotFoundException
Message: Unable to load shared library 'kernel32.dll' or one of its dependencies. In order to help diagnose loading problems, consider setting t
he DYLD_PRINT_LIBRARIES environment variable: dlopen(libkernel32.dll, 1): image not found
TypeName:
Data: System.Collections.ListDictionaryInternal
TargetSite: Boolean SetConsoleCtrlHandler(ChoETL.ConsoleCtrlMessageHandler, Boolean)
HelpLink: NULL
Source: ChoETL.Core
HResult: -2146233052

StackTrace Information
*********************************************
   at ChoETL.ChoETLFramework.SetConsoleCtrlHandler(ConsoleCtrlMessageHandler consoleCtrlRoutine, Boolean Add)
   at ChoETL.ChoETLFramework.RegisterConsoleControlHandler()
   at ChoETL.ChoETLFramework._Initialize()

It seems that the problem is in the following lines that tries to import function from kernel.dll without check if the platform NETSTANDARD2_0 or no:

Looking forward to trying ChoETL out to handle some JSON/CSV massaging, but am having trouble installing the package in my project. It looks like maybe the package is not compatible with the newest .NET Core or .NET framework I am using. Any chance that this is an error on my part? Or is this just not supported yet and ought to be a feature request?

I am targeting the .NETCoreApp 1.1 framework with my project. The out put of the installation command is below:

PM> Install-Package ChoETL
Restoring packages for C:\Users\jnelso07\Documents\Visual Studio 2017\Projects\JupyterWebService\JupyterWebService\JupyterWebService.csproj...
Install-Package : Package ChoETL 1.0.3.7 is not compatible with netcoreapp1.1 (.NETCoreApp,Version=v1.1). Package ChoETL 1.0.3.7 supports: net (.NETF
ramework,Version=v0.0)
At line:1 char:16
+ Install-Package <<<<  ChoETL
    + CategoryInfo          : NotSpecified: (:) [Install-Package], Exception
    + FullyQualifiedErrorId : NuGetCmdletUnhandledException,NuGet.PackageManagement.PowerShellCmdlets.InstallPackageCommand
 
Install-Package : One or more packages are incompatible with .NETCoreApp,Version=v1.1.
At line:1 char:16
+ Install-Package <<<<  ChoETL
    + CategoryInfo          : NotSpecified: (:) [Install-Package], Exception
    + FullyQualifiedErrorId : NuGetCmdletUnhandledException,NuGet.PackageManagement.PowerShellCmdlets.InstallPackageCommand
 
Install-Package : Package restore failed. Rolling back package changes for 'JupyterWebService'.
At line:1 char:16
+ Install-Package <<<<  ChoETL
    + CategoryInfo          : NotSpecified: (:) [Install-Package], Exception
    + FullyQualifiedErrorId : NuGetCmdletUnhandledException,NuGet.PackageManagement.PowerShellCmdlets.InstallPackageCommand
 
Time Elapsed: 00:00:01.7027199

Thanks for any assistance!

Hi can the parser read/parse CSV and create/copy to a new SQL table.

The csv header is not know before hand.
Can I simple insert based on the header from the CSV table
Can I create C# Ienumerable entity at runtime without knowing whats inthe csv file.

I have xml element with attributes

I am converting that to dynamic using the following code:

using(ChoXmlReader reader = new ChoXmlReader("some.xml").WithXPath("Diagnostics"))
{
foreach(dynamic record in reader)
{
//trying to access record.
}
}
I get an error when accessing "record.EpisodeID" but it fails. Any idea how I can make this work?

ChoETL.ChoParserException: 'Atleast one of the field header is empty. Please check the field headers at [1,4,5,6,7,8].'

Hi, i got a problem! Help me plz.

this is my code.

        //...
        var csvReader = new ChoCSVReader(CSVfilepath).WithFirstLineHeader(true);
        foreach (dynamic e in csvReader)
        {
            foreach (var value in e.Values)
            {
                   //...
            }
        }

1. 100 fields CSV file : read Success (in 10 seconds).
2. 8000 fields CSV file : Error Occured.
   Error Line : var csvReader = new ChoCSVReader(CSVfilepath).WithFirstLineHeader(true);

i want read CSV file (maybe 8000 fields, 1200 records) by high performancely.
How many field can read using CSVReader?

However
Encoding.GetEncoding("UTF8") is failing.
and should be
Encoding.GetEncoding("UTF-8")

I encountering this error while trying to use your NACHA Library

Setting IsNullable property of ChoCSVRecordFieldConfiguration object doesn't fill AllowDBNull column values from table schema (dataReader.GetSchemaTable()) of dataReader obtained from AsDataReader() method.

Is it possible to pass information about nullable columns to datareader?

Hi

I'm trying to read an xml file whose root element is

<SyncInvoice xmlns="http://schema.infor.com/InforOAGIS/2" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="SyncInvoice.xsd" languageCode="IT">

using the following

using (var parser = new ChoXmlReader<SyncInvoice>(@"invoice.xml")
               .WithXPath("SyncInvoice")
          )

where SyncInvoice is my topmost POCO

I cannot read the XML file unless I modify the root element to

<SyncInvoice languageCode="IT">

I tried using WithXmlNamespace(), but with no success.

Could you help me?

thanks
stefano

Is there a way to pass an ILogger to the logging method, or where can I override it? This will be running as an Azure Function so I need to pass it up using an ILogger.

// Type: Microsoft.Extensions.Logging.ILogger
// Assembly: Microsoft.Extensions.Logging.Abstractions, Version=2.1.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60

Ive tried multiple variations to not have the json output the null properties. Am I not understanding this correctly? If the JSON value is null, it should remove the property form the json?

https://stackoverflow.com/questions/6507889/how-to-ignore-a-property-in-class-if-null-using-json-net

var output = new StringBuilder();

	        using (var csv = new ChoCSVReader(new StringReader(reqParams.Data), cconfig)
		        .WithFirstLineHeader()
	        )
	        {
		        var test = csv;
		        using (var json = new ChoJSONWriter(new StringWriter(output), jconfig))
		        {
			        json.Write(csv);
		        }
	        }


	        return output.ToString();

[12/20/2018 8:46:17 PM]  {
[12/20/2018 8:46:17 PM]   "_Column1": "456546456",
[12/20/2018 8:46:17 PM]   "Rep_Employee_Name": "FOWLES, DANIEL (DANIEL)",
[12/20/2018 8:46:17 PM]   "Ship_To_Customer_Number": "645456456456411",
[12/20/2018 8:46:17 PM]   "_Column2": "79606",
[12/20/2018 8:46:17 PM]   "Ship_To_Customer_Name": "456456",
[12/20/2018 8:46:17 PM]   "Patient_Last_Name": [],
[12/20/2018 8:46:17 PM]   "Patient_First_Name": [],
[12/20/2018 8:46:17 PM]   "Patient_Location": [],
[12/20/2018 8:46:17 PM]   "_Column3": "456456",
[12/20/2018 8:46:17 PM]   "_Column4": "7/1/2004 11:44:16 AM",
[12/20/2018 8:46:17 PM]   "_Column5": "7/1/2004 3:25:00 PM",
[12/20/2018 8:46:17 PM]   "_Column6": "9/30/2004 5:08:26 PM",
[12/20/2018 8:46:17 PM]   "Serial_Number": [],
[12/20/2018 8:46:17 PM]   "Product_Description___Used": "VAC"
[12/20/2018 8:46:17 PM]  }
[12/20/2018 8:46:17 PM] ]

var rp = new RequestParams(data)
	        {
		        FileName = name,
				HasHeader = true,
				ChoCsvRecordConfiguration = new ChoCSVRecordConfiguration
				                            {
					                            Delimiter = "|",
					                            AutoDiscoverColumns = true,
					                            AutoDiscoverFieldTypes = true,
					                            IgnoreFieldValueMode = ChoIgnoreFieldValueMode.Any,
					                            //IgnoreFieldValueMode = ChoIgnoreFieldValueMode.Any, 
					                            FileHeaderConfiguration = new ChoCSVFileHeaderConfiguration
					                                                      {
						                                                      IgnoreColumnsWithEmptyHeader = true,
						                                                      HasHeaderRecord = true
					                                                      }	   
					                        },
				ChoJsonRecordConfiguration = new ChoJSONRecordConfiguration
				                             {
					                             //NullValueHandling = ChoNullValueHandling.Ignore,
					                             IgnoreFieldValueMode = ChoIgnoreFieldValueMode.Any,
					                             //Formatting = Formatting.Indented,
					                            /* UseJSONSerialization = true,
					                             JsonSerializerSettings	= new JsonSerializerSettings
					                                                      {
						                                                      NullValueHandling = NullValueHandling.Ignore,
						                                                      DefaultValueHandling = DefaultValueHandling.Ignore
					                                                      }*/
				                             }
	        };

	        var json = Converter.CsvToJson(log, rp);

CSV

""|"Rep Employee Name"|"Ship To Customer Number"|""|"Ship To Customer Name"|"Patient Last Name"|"Patient First Name"|"Patient Location"|""|""|""|""|"Serial Number"|"Product Description - Used"
"34534543"|"CHRISTMAN, AMY"|"580788"|"4543"|"dfgfdgfdgdfgdfsgfdgdfg"|""|""|""|"6025"|"5/13/2002 12:45:00 PM"|"5/13/2002 2:59:00 PM"|"7/2/2002 10:15:44 AM"|""|"VAC"
"34534534634"|"NAGORNY, WILLIAM"|"3453"|"363463"|"345435435"|""|""|""|"6079"|"5/15/2002 7:39:51 AM"|"3/20/2002 11:00:00 AM"|"9/25/2002 8:18:32 AM"|""|"VAC"
"34634643634"|"MOORE, NICHOLAS (NICHO"|"654287"|"98188"|"asdfdsfdfasasdf"|""|""|""|"6007"|"5/31/2002 2:45:16 PM"|"5/31/2002 3:51:00 PM"|"9/10/2002 10:51:55 AM"|""|"VAC"

I'm having difficulty configuring the ChoCSVReader to honor nullable types of a POCO. Anytime the reader finds an empty cell, it puts a default values in. Any idea on how I can configure the reader to parse a null value for the code below?

Reader

using (var reader = new ChoCSVReader<GiveOptionalValuesSomeLove>("Path goes here!").WithDelimiter("\t").WithFirstLineHeader())
{
    return reader.AsEnumerable();
}

POCO

public class GiveOptionalValuesSomeLove
{
    public string SomeValue { get; set; } //<- will be an empty string instead of null
    public int? SomeOptionalValue { get; set; } //<- will be zero if blank on the csv
}

Currently using version: 1.0.8.7

When I use the following code, the CSV doesn't load into the POCO. After removing the the configuration parameter on the ChoCSVReader call on the Reader it loads just fine into the POCO.

            ChoCSVRecordConfiguration etlConfig = new ChoCSVRecordConfiguration();
            etlConfig.MayContainEOLInData = true; //Handling for multi-line values in CSV
            etlConfig.IsDynamicObject= false;
            var reader = new ChoCSVReader<SF1150>(csvData,etlConfig).WithFirstLineHeader();
            foreach (var e in reader)
            {            
                Console.WriteLine(String.Format("Id: {0}", e.Name));
                Console.WriteLine(String.Format("Name: {0}", e.Type));
                Console.WriteLine(String.Format("Name: {0}", e.BLOCK_CURR));
            }

If I remove etlconfig from the reader it loads into the object just fine. Is this a bug or am I doing something wrong with the preconfiguration. The intent is to use field attributes against my POCO properties to match the csv headers that are being loaded.

Hi,

I am getting this error in my script.

1. Exception Information

Exception Type: System.EntryPointNotFoundException
Message: SetConsoleCtrlHandler
TypeName:
Data: System.Collections.ListDictionaryInternal
TargetSite: NULL
HelpLink: NULL
Source: NULL
HResult: -2146233053

StackTrace Information

at (wrapper managed-to-native) ChoETL.ChoETLFramework.SetConsoleCtrlHandler(ChoETL.ConsoleCtrlMessageHandler,bool)
at ChoETL.ChoETLFramework.RegisterConsoleControlHandler () [0x0001b] in <859501fe10ae4398bca6219fbdd4f60a>:0
at ChoETL.ChoETLFramework._Initialize () [0x00098] in <859501fe10ae4398bca6219fbdd4f60a>:0

I have an import file with 16632 rows with the first line header. When I set use .WithFirstLineHeader(), it only reads about 5000 rows. Am I missing something?

I am using ChoNACHAWriter to generate ACH files, however they are including a Byte Order Mark that the bank is having trouble with. How can I generate the file without a Byte Order Mark?

Also, it seems this question has been asked on SO but without a concrete answer:
https://stackoverflow.com/questions/53606446/ach-processing-adding-special-characters-in-file

What is the expected behavior for an empty file? The reader throws an exception when you call Read() or GetSchemaTable() Is there another way to handle this gracefully, i.e. return null?

Hi,
I'm using ChoCSVReader (with POCO) to load a file
My POCO contains a bunch of string properties.

When I hit the following code

var reader = new ChoCSVReader<csvRow>(csvFilePath, config).WithDelimiter("|");

I get a couple of

System.Reflection.ReflectionTypeLoadException: Unable to load one or more of the requested types. Retrieve the LoaderExceptions property for more information.
in System.Reflection.RuntimeModule.GetTypes(RuntimeModule module)
in System.Reflection.RuntimeModule.GetTypes()
in System.Reflection.Assembly.GetTypes()
in ChoETL.ChoType.ExtractTypes(Type attributeType, ArrayList types, Assembly assembly)

I tried to delete my obj and bin and recompile, but it does not solve

I'm using Nuget version 1.0.8.7 (I think it is the latest)

any suggestion?

TIA

stefano

When I parse the file with

Hello, I am parsing this sample CSV file with this code.

The problem:
The Header "ProblemHeader"/ 3rd col, is parsed as wrong data type with the code below, true or false, while it should be parsed as int!

        using (var r = new ChoCSVReader(csvIn)
            .WithFirstLineHeader()
            .WithMaxScanRows(10)
            )
        {
            foreach (IDictionary<string, object> rec in r.Take(1))
            {
                foreach (var kvp in rec)
                    Console.WriteLine($"{kvp.Key} - {r.Configuration[kvp.Key].FieldType}");
            }
        }