yugabyte / cqlsh Goto Github PK

Upstream cqlsh is able to round-trip csv files containing the \ character, ycqlsh with --debug led to corrupted data when the contents of the string contains \n. Every \n was transformed into \\n.

I'm using COPY $table TO/FROM STDOUT WITH HEADER = FALSE AND NULL = '<NULL>' AND ESCAPE = '\"';

Originally posted by @hstenzel in #3 (comment)

It seems that Python 3.10 is unsupported.

This section of the wrapper script considers 3.10 to be unsupported because it is numerically less than 3.6:

if [ "$version" = "3.6" ] || [ "$version" \> "3.6" ] || [ "$version" = "2.7" ]; then
    echo "supported"
else
    echo "unsupported"
fi

If I patch the script so that it considers 3.10 to be supported, then it instead fails with this error:

Traceback (most recent call last):
  File "C:\Work\Software\cqlsh\bin\ycqlsh.py", line 160, in <module>
    from cassandra.cluster import Cluster
  File "C:\Work\Software\cqlsh\bin\..\lib\cassandra-driver-internal-only-3.13.0.3-8d292edb.zip/cassandra-driver\3.13.0.3-8d292edb\cassandra\cluster.py", line 22, in <module>
ImportError: cannot import name 'Mapping' from 'collections' (C:\Program Files\Python310\lib\collections\__init__.py)

I have two High Sierra macs that I think have the same versions of everything from the OS through to Yugabyte. However something must be different. On one of them the CLI does not handle arrow keys correctly;

The sequence of keys escape, left arrow, up arrow, down arrow, right arrow puts this text into the prompt; ^[^[[D^[[A^[[B^[[C.

This is terribly frustrating because I can't navigate or edit input or recall previous inputs from history using the up key.

Thanks to https://stackoverflow.com/questions/893053/seeing-escape-characters-when-pressing-the-arrow-keys-in-python-shell, I finally found that adding import gnureadline solves the problem.

Let me know what else I can provide to help work out why it works on one of my macs and not the other.

ycqlsh fails with <stdin>:1:<lambda>() got an unexpected keyword argument 'encoding' in scenarios where "standard" cqlsh succeeds.

It's the same failure if I run it using the docker image or the local install. Being explicit about encoding has no bearing.

# ycqlsh fails
12:15 $ ~/cqlsh-3.10-yb-6/bin/ycqlsh $ADDR $PORT -k $KEYSPACE -e 'COPY table TO STDOUT WITH HEADER = TRUE;'
. . .
<stdin>:1:<lambda>() got an unexpected keyword argument 'encoding'

# but regular cqlsh succeeds
12:15 $ cqlsh $ADDR $PORT -k $KEYSPACE -e 'COPY table TO STDOUT WITH HEADER = TRUE;'
. . .

Description

Consider the following schema on YCQLSH

CREATE KEYSPACE sample_key;
USE sample_key;
CREATE table samplezeros(h int primary key, j jsonb);
CREATE table sampletwos(h int primary key, j jsonb);

COPY samplezeros(h, j) from '/tmp/yb/testzeros.csv';
COPY sampletwos(h, j) from '/tmp/yb/testtwos.csv';

testzeros.csv is a CSV with the first column being whole numbers and the second column is a json document with one key value pair. The value of this is very large (up to 10KB). Both the key and values only contain zeros.
testtwos.csv is the exact same except it contains twos instead of zeros.

Now note that selecting from samplezeros takes a much longer time than from sampletwos through ycqlsh.

time ./bin/ycqlsh -e "paging off; use sample_key; select * from sampletwos"
<<data>>
real	0m1.146s
user	0m0.469s
sys	0m0.260s

time ./bin/ycqlsh -e "paging off; use sample_key; select * from samplezeros"
<<data>>
real	0m7.737s
user	0m6.956s
sys	0m1.442s

This difference does not appear when we use this python script to access the data instead:
cqltest.zip

time python2.7 ./cqltester.py --twos
<<data>>
real	0m0.630s
user	0m0.157s
sys	0m0.141s

time python2.7 ./cqltester.py --zeros
<<data>>
real	0m0.619s
user	0m0.153s
sys	0m0.137s

This suggests that there is some slowdown when certain characters are involved in a column value at the YCQLSH level.

I noticed in yb 2.1.8.1 the release notes make mention of moving to python 3 for build and packaging scripts, though I'm under the impression ycqlsh requires python 2.x. Any thoughts on whether ycqlsh will be able to leverage python 3? thanks all 👍

vs

Hi,

ycqlsh fails to import blob columns using the copy from command (yugabyte version 2.14.3.1-b1).
I get the following error message:
Failed to import 1 rows: ParseError - Failed to parse 0xff : global name 'BlobType' is not defined, given up without retries Failed to process 1 rows; failed rows written to import_ks_test.err

Steps to reproduce the issue:

docker run --rm -d --name yugabyte yugabytedb/yugabyte:2.14.3.1-b1 bin/yugabyted start --daemon=false

docker exec -it yugabyte /bin/bash

ycqlsh

create keyspace ks;
-- create minimal table with a blob column
create table ks.test(key text primary key, value blob);
insert into ks.test(key, value) values('k1', 0xff);
copy ks.test to 'test.dump';

-- copying the table from the dump fails
copy ks.test from 'test.dump';

Failed to import 1 rows: ParseError - Failed to parse 0xff : global name 'BlobType' is not defined,  given up without retries
Failed to process 1 rows; failed rows written to import_ks_test.err

Adding the import "from cqlshlib.formatting import BlobType" to cqlsh/copyutil.py fixed the issue for me.