Comments (1)
Following up on this now that we've implemented SQLAlchemy 2 on main
. We have a much better understanding of how SQLAlchemy handles types now and there are some limitations of Databricks itself that make this unfeasible for the moment. Namely, databricks-sql-connector
doesn't have a way to bind parameter values of BINARY
type. sqla needs every data type to bindable for parameterisation. Since DBR doesn't support binding BINARY values, our dialect doesn't either. This same limitation applies to ARRAY
, MAP
, and STRUCT
types.
That said, where there is a will there is a way. While DBR doesn't support parameterisation of these types, you can write a string SQL query that includes them. DBR understands hexadecimal strings and can translate them to BINARY
types during ingestion. And pysql will deliver them as io.Buffer
's during egress.
So if this is necessary for your use-case, you can implement a workaround by using your own custom SQLAlchemy type class (which extends TypeDecorator
). You'll need to implement the process_bind_param
, process_literal_param
, and process_result_value
methods for this custom type in such a way that the ...param
methods accept an io.Buffer
and convert it to a hexadecimal string and render it inline with the query text.
We've marked this as a FutureFeature
to be added to this dialect here. If DBR adds support for binding these parameter types then it would be very easy to implement as part of our repository. If not, then inlining these values could work. Although would not be strictly safe from SQL injection, since an arbitrary value would be rendered in the resulting SQL string.
Although this may not be as risky, since hexadecimal strings cannot include any of the dangerous SQL commands: the hexadecimal charset doesn't include ;
or enough consonants to spell DROP
, UPDATE
, CREATE
etc.
from databricks-sql-python.
Related Issues (20)
- Provide actionable feedback when internal operations are failing rather than silently retrying for over 10mins HOT 6
- Self-signed certificate in certificate chain error on Debian-based system with databricks-sql-connector >= 2.6.0 HOT 4
- Obtain query execution time or query ID HOT 2
- Proxy authentication not working
- Is it possible to insert an arrow table? HOT 2
- Insert performance is woeful! 😢 HOT 11
- cursor.execute returing error 'NoneType' object is not iterable HOT 15
- [Document] Support databricks-cli authentication HOT 1
- Dash/Minus in column name causes UNBOUND_SQL_PARAMETER in bind values HOT 2
- Extremely slow import times on Python 3.12 HOT 9
- [sqlalchemy] execute("select 1") gives TypeError: int() argument must be a string, a bytes-like object or a real number, not 'NoneType' HOT 8
- `databricks.sql.connect` hangs in a long retrying loop when an invalid access token is used HOT 3
- Idea: arrow_record_batches cursor method
- SQLAlchemy engine from workspace level service principle?
- Unable to write list/array type data HOT 1
- Issue with version 3.1.1
- Failure on cursor.fetchall() HOT 2
- Fixing a couple type problems. (adding py.typed, typing connect, returning Any from fetchall (which I failed to fix!)) HOT 2
- Connector reads 0 rows although Cluster returned results HOT 8
- support new Cursor attribute that provides information on completed commands HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from databricks-sql-python.