mechatroner / rainbow_csv Goto Github PK

When executing RBQL using the node backend, node complains aboout broken utf-8. The error ist:

IO handling
Unable to decode input table as UTF-8. Use binary (latin-1) encoding instead

When I set let g:rbql_encoding = 'latin-1 I get the expected result.

Is this a buggy CSV-file I have (claims to be utf-8 at least) or is there a bug in the Neovim - Windows - node - Pipeline?

I deal with a balanced mix of files with , and ; delimiters and for some reason i've never been able to autodetect semicolons. If i forcibly set every csv file to csv_semicolon it works but it is as if i can't get the autodetect to run at all.

First i played around with various configs to see if they would change anything but e.g these did nothing:
vim.g.disable_rainbow_csv_autodetect = 0
vim.g.rbql_with_headers = 1
vim.g.rcsv_delimiters = {";"}
vim.g.rbql_use_system_python = 1

Despite zero vimscript knowledge, i even tried debugging the code with error messages. From what i can tell it seems that no autodetect code is ever run on file open. It always stops doing anything when it finds that "rainbow_features_enabled" is on and just always set the file type to "csv" with this:
execute "set ft=" . ft_power_cycle

Am i just dumb or why is it not able to detect that e.g this is a semicolon file?:

date;amount;other_party;category;account;recipient;bank;description
2018-05-31;148.35;p;data;a;b;;data
2018-05-31;415;p;data;a;b;;data
2018-05-31;-2.6;p;data;a;b;;data
2018-06-01;-2.6;p;data;a;b;;data

Manually setting it works fine but not auto. Is there a config i'm missing?

Are there any keyboard shortcuts for the commands in the package?

it is showing these message at the bottom and cursor moving speed is extremely slow. like 3 seconds a row

I like the autodetection function, since I have multiple .csv files, some of them have different separator other than comma.
However the autodetection also apply to the txt files.
I want to use the autodetection function, and not apply to files like .txt
what can I do?
thanks.

I tried to define a WHERE statement for the python worker with more than 10 items in the IN list and it failed:

If one looks at the resulting worker python file one sees the following problem:

a select statement like

Select a1 where a1 in ["1","2","3","4","5","6","7","8","9","10","11","12"]

is being translated into the following statement of the process_update function in the resulting worker file:

if len(rhs_records) == 1 and (a1 in ["1","2","3","4","5","6","7","8","9","10","2"0,"2"1]):

I think it is quite obvious what goes wrong with the list items "11" and "12" and so on.

I installed vim-rainbow_csv using vim-plug. However, when I press F5 I get the following error:

function provider#python#Call[9]..remote#host#Require[10]..provider#pythonx#Require[13]..provider#Poll, line 4 Vim(if):Error invoking 'poll' on channel 8:^@ch 8 was closed by the client Failed to load python host. You can try to see what happened by starting nvim with $NVIM_PYTHON_LOG_FILE set and opening the generated log file. Also, the host stderr is av ailable in messages. Press ENTER or type command to continue

I did pip install rbql, but it did not fix anything.
What could be the cause of this error?

This would be especially useful for multiline CSV fields.
This also could be useful for very long (oversized) textual fields even if they don't have newlines in them - in a separate buffer they can be line-wrapped by Vim - in this case, one record would correspond to multiple screen lines.
The following commands would be also required:

• Extract current column into a separate vertical split buffer - will create at most two new buffers: to the left and to the right from the current column.
• Show current logical row (record number)
• Sync rows (in all vertical csv buffers jump to the physical lines that correspond to the logical row in the current split buffer)

Hi, first off, I love the project!
It's Exactly what I was looking for to get a visual representation of columns without trying turn vim into excel.

I did notice, though that delimiters within quotes are still interpreted as delimiters. I am aware of a lack of standardization with *sv's in general, but it seems pretty common to put cells that contain the delimiter within quotes, ex:

Jack,20  
Maria,18 
"John, the cool one",40  
Dmitry,27
Maria,30 
John,17

which ends up highlighting like this:

Because of the lack of standardization, people also may use \ escaping, but I'm not sure how common that is - I know excel uses double quotes and have a feeling google sheets uses the same.

Is there any way to add support for delimiters that should be ignored? At least the common double-quoted case?

I'd be happy to help -- haven't looked at your code to see how much work it would be, but should theoretically be possible, though it may add a lot of complication and may open some doors to problems like "\"" -- would have to do some experimenting with excel, libreoffice and google sheets to see what's common

I may just really not understand but no matter what I do col 6 and 7 have the same color is there a way to fix this? See image.

Adding a modeline for vts values turns off the colors unless I also add "ft=tsv".
Occurs on Ubuntu Fossa and cygwin64.

Good day to you!

Is it possible to use hex-type of color in rcsv_colorpairs?

I'v tried something like this - but no luck

Thankyou!

Hello,

for some reason, the plugin sets laststatus to 2 every time a csv buffer is focused. Why is it like that? Could we have an option to disable that?

CSV files sometimes contain cells with newlines. This is valid when the cell is properly delimited with double quotes, for instance. Many tools like LibreOffice, the Frictionless Framework and even Github CSV visualization do work well with such files.

However, Rainbow CSV treats such files as invalid, and does not recognize the cell (e.g. for coloring, or the hovering tooltip with the column name).

Here is an example of one such file that contains newlines inside cells.

Only available is rfc_csv and rfc_semicolon.

This issue is just for hosting project images, see https://stackoverflow.com/questions/18360714/official-image-host-for-github-projects

Tooltips are really helpful, however, if the first row (if there is a header) would move along with scrolling would be really great. Any idea if this is feasible?

After setting g:rainbow_comment_prefix to #. when I put the cursor on a comment line, rainbow_csv reports "The number of fields in header and this (comment) line are different."
I'd expect it to ignore the comment lines.

This plugin seems to hijack the space key to create a line fold or something far less useful than my leaderkey, which is space. Using Neovim .7.

I have let g:disable_rainbow_key_mappings = 1 but this is still happening.

Assuming it is this plugin as if I :set ft=md my space leaderkey works as normal.

Using Rainbow_CSV on Neovim nightly on Windows 10.

Scripting engine is node, installed with npm install -g rbql and set let g:rbql_backend_language = 'js' in init.vim

The following statement

select *
where a19.length > 0
and a19 % 1 != 1
limit 100

should filter rows in the CSV-file, where column 19 contains non-integer numbers.

It results in the following error message:

unexpected
Unexpected identifier

For reasons outside of my control I often deal with partially populated CSVs - meaning that the header is fully populated but many lines are missing some trailing data without even commas to denote no data.

I have already patched out the warning that appears when moving the cursor to the lines in question, but Select operations still fail.

For my purposes, it would be completely appropriate to assume missing columns are just all empty string values. Does anybody have tips on how to do this easily? If not, I may look into modifying rainbow_csv further.

Thanks!

RBQL doesn't work if CSV filename contains spaces

I see this when doing a CSV Preview.

Wijmo License

The Wijmo license in this application has expired. The license expiration date is 2/1/2021.

Please create a license or contact GrapeCity: [email protected].

CLOSE
Wijmo Evaluation Version (5.20201.664)

" $ uname -r
" 5.13.10-arch1-1
" $ has nvim
" ✓ nvim 0.5.0
" $ has python
" ✓ python 3.9.6
" $ cat test.vim
set runtimepath=$VIMRUNTIME
set runtimepath+=~/.local/share/nvim/repos/github.com/mechatroner/rainbow_csv
autocmd VimEnter * colorscheme default
" $ vi -u test.vim a.csv

The color will disappear.

I guess it maybe should add some code like

autocmd! colorscheme * call some_function_render_csv()

Thanks!

Hi,
When a quoted field contains newline characters, the plugin interprets them as a row delimiter:

AFAICT, newlines inside quoted fields are allowed by the spec:

escaped = DQUOTE *(TEXTDATA / COMMA / CR / LF / 2DQUOTE) DQUOTE

my rainbow is not colorful enough.
is there a way to customize your rainbow?

Using RBQL on Windows 10 and node as the RBQL backend and with shell set to Powershell as described here:

https://neovim.io/doc/user/options.html , search for powershell:

let &shell = has('win32') ? 'powershell' : 'pwsh'
set shellquote= shellpipe=\| shellxquote=
set shellcmdflag=-NoLogo\ -NoProfile\ -ExecutionPolicy\ RemoteSigned\ -Command
set shellredir=\|\ Out-File\ -Encoding\ UTF8

When trying to run an RBQL command with F5 the interaction between Powershell and node dies with this error message:

IO handling
Unknown output csv policy

Why not change

    if !exists("g:disable_rainbow_key_mappings")
        nnoremap <buffer> <F7> :call rainbow_csv#copy_file_content_to_buf(b:self_path, b:root_table_buf_number)<cr>
    endif

to

execute 'nnoremap <buffer> ' . get(g:, 'rainbow_key_mapping', '<F7>') . ':call rainbow_csv#copy_file_content_to_buf(b:self_path, b:root_table_buf_number)<cr>'

?

Thanks!

Hi,
I am facing a slight issue, would appreciate your thoughts on this. if say I've a CSV data which contains a index field which would mean header would have trailing comma in it. Take a look in the following example:

,first,second,third,fourth
1,the,why,you,align
2,the,why,you,align
3,the,why,you,align

1,2&3 is the index number of the data.

Plugin is able to detect the correct order of colours when I'm not trying to align it. As we can see in the picture, it is colouring the index in the separate colour. But what happens when I try to align the data is:

first ,second ,third ,fourth 
    1 ,the    ,why   ,you    ,align 
    2 ,the    ,why   ,you    ,align 
    3 ,the    ,why   ,you    ,align 

It somehow align the index into first column and last column is orphan without any header for it. It kind of ignores the first comma of the header. VSCode rainbow plugin handles this properly.
I hope I explained the issue rightfully. Do let me know if I missed anything. Will put in more detail if required.
Thanks

When searching with /, usually you get an indicator of how many results appeared in the status line:

[1/5]

This plugin writes the column name the cursor is on in the status line. I think that it is overwriting the line, though, as I can't see the results info anymore :( any ideas how to fix?

Such as user can assign a variable to change ~/.rbql_table_index,
~/.rainbow_csv_storage to ~/.cache/nvim/.rbql_table_index,
~/.cache/nvim/.rainbow_csv_storage? or more compatible with XDG standard,
$XDG_CACHE_HOME/nvim/.rbql_table_index,
$XDG_CACHE_HOME/nvim/.rainbow_csv_storage?

Thanks!

is it possible to retain the headers/column names in query results? It would be helpful to have column names in the results buffer.

Either way, this plugin is incredibly helpful. Thanks for releasing it.

I don't want to change the mapping by default. So please add an option to disable the mapping.

eg. g:disable_default_mapping = 1

csv for comma, tsv for tab and some technical ids for the rest of them.

#!/usr/bin/env -S vi -u
" $ uname -r
" 5.19.7-arch1-1
" $ has
" ✓ vi 0.7.2
" $ cat test.vim
set runtimepath=$VIMRUNTIME
set runtimepath+=~/.local/share/nvim/repos/github.com/mechatroner/rainbow_csv
" $ chmod +x test.vim
" $ ./test.vim

ex: filetype=none

Expected

Nothing happened.

Actual

Error detected while processing /home/wzy/.local/share/nvim/repos/github.com/mechatroner/rainbow_csv/autoload/rainbow_csv.vim:
line  132:
E12: Command not allowed from exrc/vimrc in current dir or tag search: augroup RainbowCsvPluginInitAuGrp
line  134:
E12: Command not allowed from exrc/vimrc in current dir or tag search:     autocmd!
line  136:
E12: Command not allowed from exrc/vimrc in current dir or tag search:     autocmd VimEnter * call rainbow_csv#init_rb_color_groups()
line  137:
E12: Command not allowed from exrc/vimrc in current dir or tag search:     autocmd ColorScheme * call rainbow_csv#init_rb_color_groups()
line  138:
E12: Command not allowed from exrc/vimrc in current dir or tag search: augroup END

I can set the comment prefix in runtime by running the RainbowComment command over a pound sign (#), but if I try to set it in my vim init via vim.g.rainbow_comment_prefix = '#' the comments are not highlighted and warnings are not ignored on that line.

I can check that when during runtime with:
:let g:rainbow_comment_prefix
which does return:
g:rainbow_comment_prefix #

With g:rbql_use_system_python enabled, RBQL query does not work well.

" ${HOME}/.vimrc
Plugin 'mechatroner/rainbow_csv'

let g:rbql_backend_language  = 'python'
let g:rbql_use_system_python = v:true

:Select a1 group by a1

" Vim message
executing...
OK^M

" The target window does not change.

It works with g:rbql_backend_language = 'js'.

In my env, Windows MSYS, Vim native python looks not working well (see msys2/MSYS2-packages#1930, The python lib subprocess raises an error).

So, I'd like to use external python binary, but it just returns the message OK as above.

It looks writing into /tmp/TARGET_FILE_NAME.txt on every query,
but it does not switching the window to the output file.

Env

% cmd /v | head -n1
Microsoft Windows [Version 10.0.22631.3155]

% pacman -Q msys2-runtime
msys2-runtime 3.4.10-5

% pacman -Q vim
vim 9.1.0000-1

% vim --version
VIM - Vi IMproved 9.1 (2024 Jan 02, compiled Jan 04 2024 08:28:51)
Compiled by <https://www.msys2.org/>
Huge version without GUI.  Features included (+) or not (-):
+acl               +file_in_path      +mouse_urxvt       -tag_any_white
+arabic            +find_in_path      +mouse_xterm       -tcl
+autocmd           +float             +multi_byte        +termguicolors
+autochdir         +folding           +multi_lang        +terminal
-autoservername    -footer            -mzscheme          +terminfo
-balloon_eval      +fork()            +netbeans_intg     +termresponse
+balloon_eval_term +gettext           +num64             +textobjects
-browse            -hangul_input      +packages          +textprop
++builtin_terms    +iconv             +path_extra        +timers
+byte_offset       +insert_expand     +perl/dyn          +title
+channel           +ipv6              +persistent_undo   -toolbar
+cindent           +job               +popupwin          +user_commands
-clientserver      +jumplist          +postscript        +vartabs
+clipboard         +keymap            +printer           +vertsplit
+cmdline_compl     +lambda            +profile           +vim9script
+cmdline_hist      +langmap           -python            +viminfo
+cmdline_info      +libcall           +python3/dyn       +virtualedit
+comments          +linebreak         +quickfix          +visual
+conceal           +lispindent        +reltime           +visualextra
+cryptv            +listcmds          +rightleft         +vreplace
+cscope            +localmap          +ruby/dyn          +wildignore
+cursorbind        -lua               +scrollbind        +wildmenu
+cursorshape       +menu              +signs             +windows
+dialog_con        +mksession         +smartindent       +writebackup
+diff              +modify_fname      -sodium            -X11
+digraphs          +mouse             -sound             +xattr
-dnd               -mouseshape        +spell             -xfontset
-ebcdic            +mouse_dec         +startuptime       -xim
+emacs_tags        -mouse_gpm         +statusline        -xpm
+eval              -mouse_jsbterm     -sun_workshop      -xsmp
+ex_extra          +mouse_netterm     +syntax            -xterm_clipboard
+extra_search      +mouse_sgr         +tag_binary        -xterm_save
-farsi             -mouse_sysmouse    -tag_old_static    
   system vimrc file: "/etc/vimrc"
     user vimrc file: "$HOME/.vimrc"
 2nd user vimrc file: "~/.vim/vimrc"
      user exrc file: "$HOME/.exrc"
       defaults file: "$VIMRUNTIME/defaults.vim"
  fall-back for $VIM: "/etc"
 f-b for $VIMRUNTIME: "/usr/share/vim/vim91"
Compilation: gcc -c -I. -Iproto -DHAVE_CONFIG_H -I/usr/include/ncursesw -march=nocona -msahf -mtune=generic -O2 -pipe -D_REENTRANT -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=1 
Linking: 
gcc -pipe -Wl,--as-needed -o vim.exe -lm -lncursesw -liconv -lacl -lintl 

% pacman -Q {mingw-w64-x86_64-,}python
mingw-w64-x86_64-python 3.11.8-1
python 3.11.8-1

% type -ap python
/mingw64/bin/python             # for MINGW64 binary
/usr/bin/python                 # for MSYS binary
/bin/python                     # for MSYS binary (the same of the above)
/c/Users/xxxxxxxx/AppData/Local/Microsoft/WindowsApps/python
                                # for Windows binary installer (not installed now)

Using neovim and lightline on a German Windows 10. Neovim is set to english (FWIW)

When opening the RBQL - query window with F5 I get the following error message:

Error detected while processing function rainbow_csv#restore_statusline:
line    8:
E540: Unclosed expression sequence: statusline=%{lightline#link()}%#LightlineLeft_active_0#%(\ %{lightline#mode()}\ %)%{(&paste)?"
E10: \ should be followed by /, ? or &

such as type Tab after :Select ma it will automatically complete
max(?

Thanks!

I observed that due to the setting of nocompatible in the function rainbow_csv#buffer_enable_rainbow_features
my viminfo setting was reset to its default values.

Therefore I think the setting must not be unconditional but only if not yet set.
Thus my suggestion for a patch of the autoload file as follows:
diff_rainbow_csv_vim.txt