Connector reads 0 rows although Cluster returned results about databricks-sql-python HOT 10 OPEN

Any update on this?

from databricks-sql-python.

• I think it does depend on the row count. I did a test where I executed the problematic query with a limit of 100/1.000/10.000 and it worked;
• cursor.fetchall() is empty itself;
• cursor.description contains info about the columns
• using the flag use_cloud_fetch = false seems to work. From what I know, it should work better with the cloud fetch on, given the amount of rows, correct?

from databricks-sql-python.

Thank you @pimonteiro! Yes, CloudFetch should improve handling of very large results. I asked you to disable it to narrow down the scope of the issue. If you're able to get your data with use_cloud_fetch = false (+ considering that with smaller results everything is fine) - then probably there's some bug in CloudFetch-related code. I need to poke around, and when I'll have other questions - I'll get back to you.

P.S. I see you provided library version and warehouse details - which is very nice, thank you! Can you please also tell us if you're using AWS or Azure workspace. Thanks!

Sounds good, we will in the meantime try and reduce the query results size, optimize it in order to continue development. I assume there's nothing wrong in, for now, disabling use_cloud_fetch?

And lastly, I'm using Azure Workspace. That should have been on the opening line of the ticket, I do apologize :)

from databricks-sql-python.

Yes, you can just disable CloudFetch and go on. You'll still be able to get large results, just maybe less efficient than with CloudFetch enabled, that's it

from databricks-sql-python.

@andrefurlan-db thoughts?

from databricks-sql-python.

@pimonteiro

• if you run other queries - do you see the same behavior? Do you think it may depend on rows count?
• if you run the same query but limit rows count explicitly (say, to 10 rows or so) - does the behavior change?
• did you check if the result of cursor.fetchall() itself is empty, or you checked pandas dataframe and it's empty?
• if cursor.fetchall() actually returns data but dataframe if empty - have you checked what's in cursor.description?
• if you pass use_cloud_fetch = false to adb_sql.connect( - does it change anything?

from databricks-sql-python.

Thank you @pimonteiro! Yes, CloudFetch should improve handling of very large results. I asked you to disable it to narrow down the scope of the issue. If you're able to get your data with use_cloud_fetch = false (+ considering that with smaller results everything is fine) - then probably there's some bug in CloudFetch-related code. I need to poke around, and when I'll have other questions - I'll get back to you.

P.S. I see you provided library version and warehouse details - which is very nice, thank you! Can you please also tell us if you're using AWS or Azure workspace. Thanks!

from databricks-sql-python.

Hi, we're facing the same issue. Any update on this?

from databricks-sql-python.

@pimonteiro @akshay-s-ciq sorry, no much updates for now. I'm still trying to figure this out. However, recently we've got a very similar bug report but for Go driver. But we're still not sure what's going on.

Also, considering all the mentioned above, may I ask you to run same query but using a Nodejs connector? If any of you volunter to help - I can prepare a test project for you

from databricks-sql-python.