geofflangdale / simdcsv Goto Github PK

Terminology: Selecting a subset of columns is called a projection.

Since extracting the indexes is expensive, you may want to only pick some of the indexes, never committing to memory the unneeded indexes. So maybe you want the 4th index and the 10th one. Then you want the 14th index and the 20th index and so forth. Skipping an index is cheap, it is a single instruction (blsr).

So, conceptually, getting just the indexes you need is easy. Practically, there is a software engineering issue in getting the whole thing to work... but it can be done.

https://github.com/uniVocity/csv-parsers-comparison

So I got 2 different instances, both CPU supports AVX2, I was expecting the shared library built from one instance can be used in another, however that's not the case. I was able to narrow the issue down to the -march=native flag: When replace it with -mavx2 -mpclmul it works perfectly in both instances. Just a note for anyone who may got same issue.

It happens in every csv file parsing, especially there is only one line in the file

Any chance you could switch to a more permissive license like MIT?

Thanks for your hard work on this!

I've proposed a PR (#10) that allows to compile on MSVC.

However the performance is awful (0.65GB/sec instead of 11GB/sec on the same machine, with MSVC Clang)

If anybody is able to look into that.