Git Product home page Git Product logo

pydrda's Introduction

pydrda

This is a DRDA protocol (https://en.wikipedia.org/wiki/DRDA) database driver.

Requirements

  • Python 3.8+

Installation

$ pip install pydrda

If you want to connect to Db2, you may need to install pyDes.

$ pip install pyDes

Supported Databases

Db2

https://www.ibm.com/analytics/db2

Example

No SSL

import drda

conn = drda.connect(host='serverhost', database='dbname', user='user', password='password', port=xxxxx)
cur = conn.cursor()
cur.execute('select * from foo where name=?', ['alice'])
for r in cur.fetchall():
    print(r[0], r[1])

With SSL connection

import drda

conn = drda.connect(host='serverhost', database='dbname', use_ssl=True, user='user', password='password', port=xxxxx)
cur = conn.cursor()
cur.execute('select * from foo where name=?', ['alice'])
for r in cur.fetchall():
    print(r[0], r[1])

With SSL and client certificate

import drda

conn = drda.connect(host='serverhost', database='dbname', use_ssl=True, ssl_client_cert_path='/some/what/path/cert.crt', user='user', password='password', port=xxxxx)

Apache Derby

https://db.apache.org/derby/

You need to start derby as a network server. http://db.apache.org/derby/papers/DerbyTut/ns_intro.html#start_ns

Example

import drda

conn = drda.connect(host='serverhost', database='dbname', port=1527)
cur = conn.cursor()
cur.execute('select * from foo')
for r in cur.fetchall():
    print(r[0], r[1])

This driver can't execute with parameters against Apache Derby.

Unit Tests

I have tested the following steps.

Db2

Start Db2 server

$ docker run -itd --name db2 --privileged=true -p 50000:50000 -e LICENSE=accept -e DB2INST1_PASSWORD=password -e DBNAME=testdb --platform=linux/amd64 icr.io/db2_community/db2

Execute test

$ python test_db2.py

Apache Derby

Install Apatch Derby https://db.apache.org/derby/ and start as a server

$ curl -O https://downloads.apache.org//db/derby/db-derby-10.15.2.0/db-derby-10.15.2.0-bin.tar.gz
$ tar zxf db-derby-10.15.2.0-bin.tar.gz
$ echo 'grant {permission java.lang.RuntimePermission "getenv.SOURCE_DATE_EPOCH", "read";};' > ${HOME}/.java.policy
$ db-derby-10.15.2.0-bin/bin/startNetworkServer &

Execute test

$ python test_derby.py

pydrda's People

Contributors

climbgunks avatar data-henrik avatar nakagami avatar raphgauthier avatar walz avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar

pydrda's Issues

invalid DDS packet from socket

Hi,

I was trying to test the connection

import drda

conn = drda.connect(host='localhost', port=1527, database='jdbc:derby:<path to the database directory>')
cur = conn.cursor()
cur.execute('select * from <table>;')

but I got the following error:

ConnectionError: invalid DDS packet from socket

The Derby server outputted the following error:

Mon Oct 31 16:25:55 CET 2022 : Execution failed because of Permanent Agent Error: SVRCOD = 40; RDBNAM = jdbc:derby:/home/p; diagnostic msg = Cannot invoke "org.apache.derby.iapi.jdbc.EngineStatement.clearWarnings()" because the return value of "org.apache.derby.impl.drda.DRDAStatement.getStatement()" is null
org.apache.derby.impl.drda.DRDAProtocolException: Execution failed because of Permanent Agent Error: SVRCOD = 40; RDBNAM = jdbc:derby:/home/p; diagnostic msg = Cannot invoke "org.apache.derby.iapi.jdbc.EngineStatement.clearWarnings()" because the return value of "org.apache.derby.impl.drda.DRDAStatement.getStatement()" is null
	at org.apache.derby.impl.drda.DRDAProtocolException.newAgentError(DRDAProtocolException.java:340)
	at org.apache.derby.impl.drda.DRDAConnThread.sendUnexpectedException(DRDAConnThread.java:8805)
	at org.apache.derby.impl.drda.DRDAConnThread.handleException(DRDAConnThread.java:8762)
	at org.apache.derby.impl.drda.DRDAConnThread.run(DRDAConnThread.java:325)
Mon Oct 31 16:25:55 CET 2022 : Cannot invoke "org.apache.derby.iapi.jdbc.EngineStatement.clearWarnings()" because the return value of "org.apache.derby.impl.drda.DRDAStatement.getStatement()" is null
java.lang.NullPointerException: Cannot invoke "org.apache.derby.iapi.jdbc.EngineStatement.clearWarnings()" because the return value of "org.apache.derby.impl.drda.DRDAStatement.getStatement()" is null
	at org.apache.derby.impl.drda.DRDAConnThread.parseEXCSQLSETobjects(DRDAConnThread.java:5604)
	at org.apache.derby.impl.drda.DRDAConnThread.parseEXCSQLSET(DRDAConnThread.java:5466)
	at org.apache.derby.impl.drda.DRDAConnThread.processCommands(DRDAConnThread.java:801)
	at org.apache.derby.impl.drda.DRDAConnThread.run(DRDAConnThread.java:300

I've managed to successfully connect with the database and execute the same query through derby console (ij). Do you have any suggestions what I might did wrong?

Versions:
python: 3.9.13
pydrda: 0.4.5
db-derby: 10.15.2.0 and 10.16.1.1 (tested both)
java:
openjdk version "17.0.4" 2022-07-19
OpenJDK Runtime Environment (build 17.0.4+8-Ubuntu-122.04)
OpenJDK 64-Bit Server VM (build 17.0.4+8-Ubuntu-122.04, mixed mode, sharing)

Derby database location different from when using java org.apache.derby.tools.ij

Hi, I was not being able to connect to a Database I've set and I realized it was happening because my Derby database default location was on "C:\Users[my user]", while pydrda was looking for it in the bin folder of my Derby installation folder (C:\Apache\Derby\bin). Do you know if it is a malfunction of the package or if it is some misconfiguration of mine?

close a connect

Hi,

i have encountered an error while querying in a test database. When I run

cur.execute('select * from syscat.tables where tabschema = current schema')
get the following error :
Traceback (most recent call last):
File "D:\admin\PyCharm Community Edition 2019.1.1\helpers\pydev\pydevd.py", line 1741, in
main()
File "D:\admin\PyCharm Community Edition 2019.1.1\helpers\pydev\pydevd.py", line 1735, in main
globals = debugger.run(setup['file'], None, None, is_module)
File "D:\admin\PyCharm Community Edition 2019.1.1\helpers\pydev\pydevd.py", line 1135, in run
pydev_imports.execfile(file, globals, locals) # execute the script
File "D:\admin\PyCharm Community Edition 2019.1.1\helpers\pydev_pydev_imps_pydev_execfile.py", line 18, in execfile
exec(compile(contents+"\n", file, 'exec'), glob, loc)
File "D:/exe/DB2Test/pydrda-master/db2Example.py", line 5, in
cur.execute('select * from syscat.tables where tabschema = current schema')
File "D:\exe\DB2Test\pydrda-master\drda\cursor.py", line 62, in execute
self._rows, self.description = self.connection._query(self.query, args)
File "D:\exe\DB2Test\pydrda-master\drda\connection.py", line 389, in _query
rows, description, params_description = self._parse_response()
File "D:\exe\DB2Test\pydrda-master\drda\connection.py", line 50, in _parse_response
dds_type, chained, number, code_point, obj = ddm.read_dds(self.sock)
File "D:\exe\DB2Test\pydrda-master\drda\ddm.py", line 229, in read_dds
b = _recv_from_sock(sock, 6)
File "D:\exe\DB2Test\pydrda-master\drda\ddm.py", line 38, in _recv_from_sock
bs = sock.recv(n)
ConnectionResetError: [WinError 10054]

Max DSS Size

Hi,

i have encountered an error while querying in a large database. When I run

cur.execute(""" SELECT <col> FROM <table> """)

I get the following error :

AssertionError                            Traceback (most recent call last)
<ipython-input-3-592b61ee4a58> in <module>
   1 cur = connection.cursor()
   2 try:
----> 3     cur.execute(""" SELECT <col> FROM <table> """)
   4 except drda.OperationalError:
   5     pass

/opt/conda/lib/python3.7/site-packages/drda/cursor.py in execute(self, query, args)
  60         self.query = query32776
  61         if query.strip().split()[0].upper() == 'SELECT':
---> 62             self._rows, self.description = self.connection._query(self.query, args)
  63         else:
  64             self.connection._execute(self.query, args)

/opt/conda/lib/python3.7/site-packages/drda/connection.py in _query(self, query, args)
 364                 cur_id, False, True
 365             )
--> 366             rows, description, params_description = self._parse_response()
 367             return rows, description
 368 

/opt/conda/lib/python3.7/site-packages/drda/connection.py in _parse_response(self)
  45         err_msg = None
  46         while chained:
---> 47             dds_type, chained, number, code_point, obj = ddm.read_dds(self.sock)
  48             print("dds_type:", dds_type, "chained:", chained, "number:", number, "code_point:", code_point)
  49             if code_point == cp.SQLERRRM:
sql
/opt/conda/lib/python3.7/site-packages/drda/ddm.py in read_dds(sock)
 239     print("b[3]", b[3])
 240 
--> 241     assert int.from_bytes(obj[:2], byteorder='big') == ln - 6
 242     code_point = int.from_bytes(obj[2:4], byteorder='big')
 243 

AssertionError:

At the time it fails into AssertionError, obj[:2] is equal to 32776, and ln to 65535.

Here is the Server Trace file. I am not expert in DRDA protocol at all, but I guess that the server has to send more data than a single DSS can contain, so it cuts it in multiple parts as described in DssProtocolErrors. Is it correct?

Receive hangs when reading large response from select (>32k?)

I am connecting to a DB2 database and selecting N rows from a CUSTOMER table. For large N, this hangs in ddm.py:_recv_from_sock().

In my case the hang occurs when LIMIT N > 122. When I use LIMIT 122, the header is bytes 7f7cd0530002, which is the correct length of 32636 and _recv_from_sock() reads the additional 32630 bytes as it should. All rows are approximately the same size.

For X = 123:
The DDS header (6 bytes) looks like bytes ffffd0530002 Treating the first 2 bytes as the length gives 65535, but this is incorrect. _recv_from_sock() hangs after 33025 bytes

Note that there is nothing in the data for row 123 that is causing the issue, because I can pull it by itself or in a group of rows with an appropriate WHERE clause.

DRDA protocol error ?

Hello, i have a problem when i tried to connect to Derby network server, it raise an exception (see picture below)
image
so i wonder is there something to configure?
image

also this how i start the derby network server

public void cmdStartServer(String[] args) {
		try {
			getDBConnection();
			NetworkServerControl server = new NetworkServerControl(InetAddress.getByName("0.0.0.0"),1527);
			server.start(null);
			System.out.println("Server started: " + server);
		}catch(UnknownHostException e){
			e.printStackTrace();
		}catch(Exception e){
			e.printStackTrace();
		}
	}

Thank you in advance, i'm planning to port this to Golang after this protocol error fix

Q: How to specify SSL certificate or other options?

I followed the README for testing the connection to a Db2 database with SSL. I am using Db2 on Cloud and its credentials. I received this error:

File "/home/xxxxx/.local/lib/python3.6/site-packages/drda/ddm.py", line 236, in read_dds
    raise ConnectionError("invalid DDS packet from socket")
ConnectionError: invalid DDS packet from socket

To me it looks like I need to specify more SSL options, e.g., a certificate. Any advice?

Error connecting to DB2 (ModuleNotFoundError: No module named 'pyDes')

Hi Hajime,

I hope you are doing well, thanks for your beautiful work. we use your library for a while to connect to db2 for development, unfortunately when we tried to connect to db2 with des encryption we get this error
I figured out pyDes library is not included I appreciate your help.

Here is sample of connection string
drda.connect(host=db2_host, database=db2_database, user=db_username, password=db_password, port=db2_port)

File "/tmp/drda.zip/drda/init.py", line 122, in connect
return Connection(host, database, port, user, password, use_ssl, db_type)
File "/tmp/drda.zip/drda/connection.py", line 194, in init
self.encoding
File "/tmp/drda.zip/drda/ddm.py", line 290, in packSECCHK
des = secmec9.des(sectkn, private_key)
File "/tmp/drda.zip/drda/secmec9.py", line 51, in des
import pyDes
ModuleNotFoundError: No module named 'pyDes'

Thanks,

Errors on insert with parameter markers

I am trying to run an INSERT statement that has string, integer, and date values. The statement works when passing the values as literals, but fails when trying to use parameter markers (?).

Could such a scenario be added to the Db2 unit tests, both for QA and serving as example?

Can't SELECT CLOB column (derby)

Hi, I think I found a bug with executing SELECT query for a specific columns. I don't understand what is specific about the column I try to select but I know that SELECT returns empty listy only when I include it. There is no error neither in python code nor in Derby logs.

I have a database which contains table TBL_UNITS:

Column Name		#	Data Type	Length	Scale	Not Null
ID			1	BIGINT		19	[NULL]	true
REF_ID			2	VARCHAR		36	[NULL]	false
NAME			3	VARCHAR		2048	[NULL]	false
DESCRIPTION		4	CLOB		65536	[NULL]	false
VERSION			5	BIGINT		19	[NULL]	false	
LAST_CHANGE		6	BIGINT		19	[NULL]	false		
CONVERSION_FACTOR	7	DOUBLE		52	[NULL]	false
SYNONYMS		8	VARCHAR		255	[NULL]	false	
F_UNIT_GROUP		9	BIGINT		19	[NULL]	false	
  • Query SELECT * FROM TBL_UNITS returns []
  • Query SELECT DESCRIPTION FROM TBL_UNITS returns []
  • Any query without column DESCRITPTION return expected results.

Sample working query:

SELECT ID, NAME, CONVERSION_FACTOR, F_UNIT_GROUP FROM  TBL_UNITS fetch first 5 rows only

Results of the query:

[
    (505, 'LVL 2000', 1.78, 506),
    (507, 'm2*d', 0.00274, 508),
    (509, 'kcal', 0.00419, 510),
    (511, 'GJ', 1000.0, 510),
    (512, 'lb av', 0.45, 513)
]

It looks that DESCRIPTION is the only column with data type CLOB. Could it be a reason of this strange behavior? Please let know if you need any additional information.

Versions:

pydrda==0.4.7
derby==10.16.1.1
python==3.9.13
java==openjdk 17.0.4

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.