Data parsing bug when the first column has nulls about openrefine HOT 7 CLOSED

We actually have already use two different separator guessers, but if they disagreed we were just giving up which caused it to default to TSV, which also disabled the quote handling which defaults to ON for CSV and OFF for TSV. The Univocity parser guessed " " (the space character), which isn't completely unreasonable, and we guessed "," (comma), which is correct. I've created a patch which will use our internal guess in cases of disagreement, which preserves backward compatibility.

from openrefine.

Ok, let's do that

from openrefine.

Yes, you're not the first one to be surprised by that! We are aware that the records mode is pretty unintuitive for newcomers, especially because it gets activated by default in a case like yours.
You can read more about it here:
https://openrefine.org/docs/manual/exploring#rows-vs-records

We have some ideas of how to make that more transparent #5174, #5175. Do you think they would have helped?

from openrefine.

Wow I did not consider the record mode because I've never used it, despite years of using OpenRefine! Is it for dealing with json and other hierarchical data?

My bad then! I really like the suggestion in #5175 as it would have made clear to me that OR is warning me about something, even if I don't know yet what it is. It would then be easy to have a Q&A page with "why is my first column red?" for people looking for answers.

A tutorial would have been useless because probably forgotten by the time you end up activating record mode by chance. As I mentionned, I've been using OpenRefine for years, and I mentioned the issue to 2 other colleagues who use it regularly, and everybody thought that was a bug.

What about the double quotes though?

from openrefine.

What about the double quotes though?

Ah yes I had missed that part, well, that's something that we should definitely investigate, I don't see any reason why this would be by design.

from openrefine.

I think perhaps just changing the defaults to never start in record mode might be the best solution.

There's no "hallucination" here because a) we don't use LLMs and b) the quotes are in the original data. Selecting the option Use character " to enclose cells containing column separators will make them go away.

I see three things that could be improved with this dataset:

1. Guess CSV instead of TSV, as it currently does, for the initial format
2. Enable quote stripping by default when it's indicated
3. Start in row mode instead of record mode

The CSV package that we use has a format "sniffer" which may be able to help with 1 & 2.

from openrefine.

@wetneb I recommend that the fix for this be included in 3.8 since the regression was first introduced in that release with #6098 and this is, in my opinion, a low risk fix which restores the prior behavior.

from openrefine.