Git Product home page Git Product logo

rethinkdb-python's Introduction

RethinkDB Python driver

PyPI version Build Status Codacy Badge Codacy Badge

Overview

What is RethinkDB?

RethinkDB is the first open-source scalable database built for realtime applications. It exposes a new database access model -- instead of polling for changes, the developer can tell the database to continuously push updated query results to applications in realtime. RethinkDB allows developers to build scalable realtime apps in a fraction of the time with less effort.

Installation

$ pip install rethinkdb

Note: this package is the extracted driver of RethinkDB's original python driver.

Quickstart

The main difference with the previous driver (except the name of the package) is we are not importing RethinkDB as r. If you would like to use RethinkDB's python driver as a drop in replacement, you should do the following:

from rethinkdb import r

connection = r.connect(db='test')

Blocking and Non-blocking I/O

This driver supports blocking I/O (i.e. standard Python sockets) as well as non-blocking I/O through multiple async frameworks:

The following examples demonstrate how to use the driver in each mode.

Default mode (blocking I/O)

The driver's default mode of operation is to use blocking I/O, i.e. standard Python sockets. This example shows how to create a table, populate with data, and get every document.

from rethinkdb import r

connection = r.connect(db='test')

r.table_create('marvel').run(connection)

marvel_heroes = r.table('marvel')
marvel_heroes.insert({
    'id': 1,
    'name': 'Iron Man',
    'first_appearance': 'Tales of Suspense #39'
}).run(connection)

for hero in marvel_heroes.run(connection):
    print(hero['name'])

Asyncio mode

Asyncio mode is compatible with Python ≥ 3.5.

import asyncio
from rethinkdb import r

async def main():
    async with await r.connect(db='test') as connection:
        await r.table_create('marvel').run(connection)

        marvel_heroes = r.table('marvel')
        await marvel_heroes.insert({
            'id': 1,
            'name': 'Iron Man',
            'first_appearance': 'Tales of Suspense #39'
        }).run(connection)

        # "async for" is supported in Python ≥ 3.6. In earlier versions, you should
        # call "await cursor.next()" in a loop.
        cursor = await marvel_heroes.run(connection)
        async for hero in cursor:
            print(hero['name'])
    # The `with` block performs `await connection.close(noreply_wait=False)`.

r.set_loop_type('asyncio')

# "asyncio.run" was added in Python 3.7.  In earlier versions, you
# might try asyncio.get_event_loop().run_until_complete(main()).
asyncio.run(main())

Gevent mode

import gevent
from rethinkdb import r

def main():
    r.set_loop_type('gevent')
    connection = r.connect(db='test')

    r.table_create('marvel').run(connection)

    marvel_heroes = r.table('marvel')
    marvel_heroes.insert({
        'id': 1,
        'name': 'Iron Man',
        'first_appearance': 'Tales of Suspense #39'
    }).run(connection)

    for hero in marvel_heroes.run(connection):
        print(hero['name'])

gevent.joinall([gevent.spawn(main)])

Tornado mode

Tornado mode is compatible with Tornado < 5.0.0. Tornado 5 is not supported.

from rethinkdb import r
from tornado import gen
from tornado.ioloop import IOLoop

@gen.coroutine
def main():
    r.set_loop_type('tornado')
    connection = yield r.connect(db='test')

    yield r.table_create('marvel').run(connection)

    marvel_heroes = r.table('marvel')
    yield marvel_heroes.insert({
        'id': 1,
        'name': 'Iron Man',
        'first_appearance': 'Tales of Suspense #39'
    }).run(connection)

    cursor = yield marvel_heroes.run(connection)
    while (yield cursor.fetch_next()):
        hero = yield cursor.next()
        print(hero['name'])

IOLoop.current().run_sync(main)

Trio mode

from rethinkdb import r
import trio

async def main():
    r.set_loop_type('trio')
    async with trio.open_nursery() as nursery:
        async with r.open(db='test', nursery=nursery) as conn:
            await r.table_create('marvel').run(conn)
            marvel_heroes = r.table('marvel')
            await marvel_heroes.insert({
                'id': 1,
                'name': 'Iron Man',
                'first_appearance': 'Tales of Suspense #39'
            }).run(conn)

            # "async for" is supported in Python ≥ 3.6. In earlier versions, you should
            # call "await cursor.next()" in a loop.
            cursor = await marvel_heroes.run(conn)
            async with cursor:
                async for hero in cursor:
                    print(hero['name'])

trio.run(main)

The Trio mode also supports a database connection pool. You can modify the example above as follows:

db_pool = r.ConnectionPool(db='test', nursery=nursery)
async with db_pool.connection() as conn:
    ...
await db_pool.close()

Twisted mode

from rethinkdb import r
from twisted.internet import reactor, defer

@defer.inlineCallbacks
def main():
    r.set_loop_type('twisted')
    connection = yield r.connect(db='test')

    yield r.table_create('marvel').run(connection)

    marvel_heroes = r.table('marvel')
    yield marvel_heroes.insert({
        'id': 1,
        'name': 'Iron Man',
        'first_appearance': 'Tales of Suspense #39'
    }).run(connection)

    cursor = yield marvel_heroes.run(connection)
    while (yield cursor.fetch_next()):
        hero = yield cursor.next()
        print(hero['name'])

main().addCallback(lambda d: print("stopping") or reactor.stop())
reactor.run()

Misc

To help the migration from rethinkdb<2.4 we introduced a shortcut which can easily replace the old import rethinkdb as r import with from rethinkdb import r.

Run tests

In the Makefile you can find three different test commands: test-unit, test-integration and test-remote. As RethinkDB has dropped the support of Windows, we would like to ensure that those of us who are using Windows for development can still contribute. Because of this, we support running integration tests against Digital Ocean Droplets as well.

Before you run any test, make sure that you install the requirements.

$ pip install -r requirements.txt
$ make prepare

Running unit tests

$ make test-unit

Running integration tests

To run integration tests locally, make sure you intstalled RethinkDB

$ make test-integration

Running remote integration tests

To run the remote tests, you need to have a Digital Ocean account and an API key.

Remote test will create a new temporary SSH key and a Droplet for you until the tests are finished.

Available environment variables

Variable name Default value
DO_TOKEN N/A
DO_SIZE 512MB
DO_REGION sfo2
$ pip install paramiko python-digitalocean
$ export DO_TOKEN=<YOUR_TOKEN>
$ make test-remote

Contributing

Hurray! You reached this section which means, that you would like to contribute. Please read our contributing guide lines and feel free to open a pull request.

rethinkdb-python's People

Contributors

alexa-infra avatar asakatida avatar camrudnick avatar codeskyblue avatar cxmcc avatar danielmewes avatar dependabot-preview[bot] avatar dependabot-support avatar dependabot[bot] avatar encryptio avatar frank-trampe avatar gabor-boros avatar jdoliner avatar jipperinbham avatar larkost avatar lsabi avatar marshall007 avatar mehaase avatar mlucy avatar neumino avatar nh2 avatar raitobezarius avatar rntz avatar srh avatar takluyver avatar timmaxw avatar tryneus avatar vexocide avatar wmrowan avatar yograterol avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

rethinkdb-python's Issues

Refactor handshake code.

Description

Refactor handshake code. Removal of older handshake code is pending spectrum discussion.

Restore not working on Windows due to : (colon) in file and folder names

rethinkdb dump by default creates a file that contains colons, such as rethinkdb_dump_2017-01-24T21:34:23.tar.gz. This can be overriden with -f, BUT, since all the actual data files are stored inside a folder named rethinkdb_dump_2017-01-24T21:34:23 inside the archive no matter what -f is set to, it's basically impossible to restore (and probably dump) on Windows.

Windows does not allow colon in paths, and also fails horribly if you try (colon is usually the drive separation sign, such as C:\stuff ).

Restoring doesn't pick up primary_key from info files

Describe the bug
We upgraded RethinkDB from 2.3.6 to 2.4.0, and the driver accordingly, rebuilt indexes. Now, dumps generated by the most recent python driver (2.4.4.post1) fail when restoring, with every table with a primary_key that isn't id failing as follows:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/rethinkdb/_import.py", line 400, in read_to_queue
    for batch in self.batches(warning_queue=warning_queue):
  File "/usr/local/lib/python3.6/dist-packages/rethinkdb/_import.py", line 326, in batches
    self.setup_table()
  File "/usr/local/lib/python3.6/dist-packages/rethinkdb/_import.py", line 271, in setup_table
    (self.db, self.table, primary_key, self.primary_key))
RuntimeError: Error: table mydb.settings primary key was `id` rather than the expected: name

Here is a simple info + table file from the dump:

$ cat settings.info
{"db": {"id": "adb308f5-6d96-4e56-8002-911fb0676582", "name": "mydb", "type": "DB"}, "doc_count_estimates": [2], "id": "ccaf09fa-fb79-4c95-9db1-2f1f591cd8d6", "indexes": [], "name": "settings", "primary_key": "name", "type": "TABLE", "write_hook": null}
$ cat settings.json
[
{"name": "counter", "value": 30750}
]

To Reproduce
Steps to reproduce the behavior:

  1. Upgrade a 2.3.6 DB to 2.4.0
  2. Spin up our DB: docker run --rm -v /var/lib/rethinkdb:/data --name db -it rethinkdb
  3. Spin up a fresh DB: docker run --rm --name newdb -it rethinkdb
  4. Get the latest rethinkdb python client:
    docker run --rm --link db -v /var/lib/rethinkdb:/data -it rethinkdb bash
    apt update && apt install -y python3-pip && pip3 install             # inside bash
    
  5. Dump the DB (inside bash):
    # rethinkdb dump -c db
    
  6. Try to restore the DB:
    # rethinkdb restore -c newdb $(ls rethinkdb_dump_*.tar.gz | tail -n1)
    

Expected behavior
The restore succeeds.

System info

  • OS: Debian 9
  • RethinkDB Version: 2.4.0 (as provided by docker)
  • Python client version: 2.4.4.post1

Additional context
Using the 2.3.0.post6 client works fine, just like in #157

`str(r.now())` or `repr(r.now())` raise RuntimeError in python 3.7+

Describe the bug

This happens in python 3.7 but seemingly not earlier versions.

>>> from rethinkdb import RethinkDB
>>> r = RethinkDB()
>>> r.now()
Traceback (most recent call last):
  File "/Users/nlevitt/workspace/doublethink/doublethink-ve37/lib/python3.7/site-packages/rethinkdb/errors.py", line 246, in __iter__
    for sub in next(itr):
StopIteration

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/nlevitt/workspace/doublethink/doublethink-ve37/lib/python3.7/site-packages/rethinkdb/ast.py", line 147, in __repr__
    return "<RqlQuery instance: %s >" % str(self)
  File "/Users/nlevitt/workspace/doublethink/doublethink-ve37/lib/python3.7/site-packages/rethinkdb/ast.py", line 144, in __str__
    return printer.print_query()
  File "/Users/nlevitt/workspace/doublethink/doublethink-ve37/lib/python3.7/site-packages/rethinkdb/errors.py", line 202, in print_query
    return ''.join(self.compose_term(self.root))
  File "/Users/nlevitt/workspace/doublethink/doublethink-ve37/lib/python3.7/site-packages/rethinkdb/errors.py", line 251, in __iter__
    for sub in token:
RuntimeError: generator raised StopIteration

Can be a major problem if code tries to log a query before running it, for example.

To Reproduce
Steps to reproduce the behavior:

  1. see above

Expected behavior
Return a reasonable stringification.

Screenshots
If applicable, add screenshots to help explain your problem.

System info

  • OS: tested on mac, linux
  • RethinkDB Version: any
  • Python client version: 2.4.1, 2.3.0

Additional context
Add any other context about the problem here.

Python driver. Python3.7. How to get_all() from DB if RuntimeError following StopIteration.

>>> import rethinkdb
>>> all = rethinkdb.db('content').table('brand_filters').get_all()
Traceback (most recent call last):
  File ".venv/lib/python3.7/site-packages/rethinkdb/errors.py", line 219, in iter
    for sub in next(itr):
StopIteration
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
  File ".venv/lib/python3.7/site-packages/IPython/core/formatters.py", line 224, in catch_format_error
    r = method(self, *args, **kwargs)
  File ".venv/lib/python3.7/site-packages/IPython/core/formatters.py", line 702, in call
    printer.pretty(obj)
  File ".venv/lib/python3.7/site-packages/IPython/lib/pretty.py", line 400, in pretty
    return _repr_pprint(obj, self, cycle)
  File ".venv/lib/python3.7/site-packages/IPython/lib/pretty.py", line 695, in _repr_pprint
    output = repr(obj)
  File ".venv/lib/python3.7/site-packages/rethinkdb/ast.py", line 130, in repr
    return "<RqlQuery instance: %s >" % str(self)
  File ".venv/lib/python3.7/site-packages/rethinkdb/ast.py", line 127, in str
    return qp.print_query()
  File ".venv/lib/python3.7/site-packages/rethinkdb/errors.py", line 175, in print_query
    return ''.join(self.compose_term(self.root))
  File ".venv/lib/python3.7/site-packages/rethinkdb/errors.py", line 224, in iter
    for sub in token:
RuntimeError: generator raised StopIteration

Refactoring caused connection error

Description

The refactoring of net.py made in commit 89db318 closes the connection anyway not only when exception occurred.

Traceback

traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "rebirthdb/net.py", line 722, in connect
    return conn.reconnect(timeout=timeout)
  File "rebirthdb/net.py", line 617, in reconnect
    return self._instance.connect(timeout)
  File "rebirthdb/net.py", line 473, in connect
    self._socket = SocketWrapper(self, timeout)
  File "rebirthdb/net.py", line 306, in __init__
    self._socket = socket.create_connection((self.host, self.port), timeout)
rebirthdb.errors.ReqlDriverError: Could not connect to localhost:28015. Error: 'NoneType' object has no attribute 'settimeout'

Suggestion

Put back the socket close to the exception handling.

Later improvements

When refactoring the package, we should figure out a nicer way to handle these kind of socket closes.

Update documentation for the new driver

Is your feature request related to a problem? Please describe.
I have been checking the documentation on the python driver and frankly it's old. I've also tried using the ConnectionPool feature but i'm having a hard time doing so. The example is not really concise and the documentation does not have this.
Also I use PyCharm as an IDE and most of the time it tells me certain things like r.table_create() and such are not available

Describe the solution you'd like
Updating the documentation to reflect the changes would be nice with some actual practical examples.

Additional context
I'm also migrating from the 2.3.x driver and the transition in a big application is painful without proper docs

add rate limiting to export and dump commands

When export or dump run they can overwhelm some clusters, slowing down normal traffic. An ultimate solution would be to de-prioritize traffic going to the backup tools, but that would be a lot of work.

A much more approachable solution would be to put rate limiting into the export command (which is used by dump). This would be per-concurrent table, and would have to be something simplistic like number-of-records-per-second, but would probably be the biggest bang-for-the-buck.

improve dump/restore testing

With rethinkdb/rethinkdb#2598 there is now basic testing for dump/restore and friends. However the testing completed in that push is only the most basic positive testing, there is no testing at all for bad cases, or for any of the options beyond --force. Those test are going to look a little different, and so will be done in a separate push.

A few things that I think should be included:

  • rebuilding outdated indexes
  • handling bad data
  • as many command line options as are practical, at a minimum fields and custom-header

This is going to require cleaning up this portion of the driver, as despite some work I did in the last push, the interface code is still wound tight into the process, so that will contribute to the effort required.

Rethinkdb with python driver installed

It would be great to have an image with python and the python driver already installed via pip.
I needed to use the rethinkdb command line utility that needs the python driver to dump, restore, etc.
Nowadays you have to sh in the container and install python-pip and the driver.
If this could be useful to someone other, I can make a pull request.

AttributeError when serializing RqlConstant

When using a constant in queries, rethinkdb will raise an AttributeError when serialising it to JSON.

self = <RqlQuery instance: r.minval >

    def build(self):
>       res = [self.term_type, self._args]
E       AttributeError: 'RqlConstant' object has no attribute 'term_type'

/home/tom/env/lib/python3.7/site-packages/rethinkdb-1.0.0-py3.7.egg/rethinkdb/ast.py:151: AttributeError

RqlConstant is a subclass of RqlQuery, but does not have the expected attribute term_type, instead it has tt.

RqlConstant:

class RqlConstant(ast.RqlQuery):

RqlQuery requires term_type:
res = [self.term_type, self._args]

add `with` support to Python cursor

The Connection class in the Python driver currently supports the Python with structure, but our Cursor class does not. I just ran into a case where it would be nice to have this. An example for clarity:

with r.db('foo').table('bar').run(conn) as full_table:
     for row in full_table:
          pass # do something in reality

While our Cursors are not that expensive, it does make sure that things get cleaned up on-time.

Deliver the missing unit tests to cover all query cases.

Description

Deliver the missing unit tests to cover all query cases. This will become an ongoing task based on the permutations that can be expressed. For this first pass getting most combinations of two terms will be enough to release.

rethinkdb-dump is broken

Describe the bug
v2.4.3 tries to connect to localhost and ignores commandline arguments pointing to another host
v2.4.0 - v2.4.2.post1 error at check_tls_option()
v2.3.0.post6 last working version

To Reproduce
Steps to reproduce the behavior:

  1. run rethinkdb-dump -c rethinkdb:28015 -f dump.tar.gz --tls-cert ./tls/rethinkdb-cert.pem --password-file pw

Expected behavior
A dump archive is successfully created without error

Errors
v2.4.3

$ rethinkdb-dump -c rethinkdb:28015 -f dump_2019-08-24.tar.gz --tls-cert ./tls/rethinkdb-cert.pem --password-file pw
Usage: rethinkdb dump [-c HOST:PORT] [-p] [--password-file FILENAME] [--tls-cert FILENAME] [-f FILE] [--clients NUM] [-e (DB | DB.TABLE)]...

rethinkdb-dump: error: Unable to connect to server: Could not connect to localhost:28015. Error: [Errno 99] Cannot assign requested address

v2.4.0 - v2.4.2.post1

$ rethinkdb-dump -c rethinkdb -f dump_2019-08-24.tar.gz --tls-cert ./tls/rethinkdb-cert.pem --password-file pw
Traceback (most recent call last):
  File "/usr/local/bin/rethinkdb-dump", line 10, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.7/site-packages/rethinkdb/_dump.py", line 143, in main
    options = parse_options(argv or sys.argv[2:], prog=prog)
  File "/usr/local/lib/python3.7/site-packages/rethinkdb/_dump.py", line 94, in parse_options
    options, args = parser.parse_args(argv)
  File "/usr/local/lib/python3.7/site-packages/rethinkdb/utils_common.py", line 356, in parse_args
    options, args = super(CommonOptionsParser, self).parse_args(*args, **kwargs)
  File "/usr/local/lib/python3.7/optparse.py", line 1387, in parse_args
    stop = self._process_args(largs, rargs, values)
  File "/usr/local/lib/python3.7/optparse.py", line 1427, in _process_args
    self._process_long_opt(rargs, values)
  File "/usr/local/lib/python3.7/optparse.py", line 1501, in _process_long_opt
    option.process(opt, value, values, self)
  File "/usr/local/lib/python3.7/optparse.py", line 779, in process
    value = self.convert_value(opt, value)
  File "/usr/local/lib/python3.7/optparse.py", line 771, in convert_value
    return self.check_value(opt, value)
  File "/usr/local/lib/python3.7/optparse.py", line 766, in check_value
    return checker(self, opt, value)
TypeError: check_tls_option() takes 2 positional arguments but 3 were given

System info
docker container built using FROM python:3

  • OS:
$ cat /etc/os-release 
PRETTY_NAME="Debian GNU/Linux 10 (buster)"
NAME="Debian GNU/Linux"
VERSION_ID="10"
VERSION="10 (buster)"
VERSION_CODENAME=buster
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"
  • Python version:
$ python -V
Python 3.7.4

ssl wildcat is not support

Describe the bug
I can't connect to RethinkDB using wildcat ssl

To Reproduce

  1. TODO
    Change hostname at match_hostname to wilcat and have a arg to check wildcat

Expected behavior

match_hostname(self._socket.getpeercert(), hostname="*."+(self.host.split('.', 1)[-1]))

Screenshots
image

System info

  • OS: macOS Mojave 10.14.3
  • RethinkDB Version: 2.3.6
  • Python client version: 2.4.2.post1

Package it as python3-rethinkdb for Debian/Ubuntu and other supported platforms.

rethinkdb-python driver is an "official driver" and in fact it is required by rethinkdb for such crucial subcommand as import, so it would be great to have it packaged and make it available as python3-rethinkdb(*) next to rethinkdb "official packages"

(*) Debian/Ubuntu naming convention, it might be different for other distributions.

Index-rebuild failure after upgrading from 2.0.4 to 2.2.0

Every time while I am upgrading the web console always remind to run rethinkdb index-rebuild

But the release note just states that it was not necessary to do this, why ?

Anyway I perform the index rebuild but the python driver complains following

ReQL error during 'rename `aaa.bbb` index `idx_ccc`': The server(s) hosting table `weixin.$reql_temp_index$_idx_ccc` are currently unreachable. The secondary index was not renamed.

Python Error: %d format: a number is required, not str

rethinkdb export does not work!

It causes the python driver to crash: ReQL error during 'version check': Could not connect to localhost:28015. Error: %d format: a number is required, not str

the driver seems to be up-to-date:

sudo pip show rethinkdb

---
Name: rethinkdb
Version: 2.3.0
Location: /usr/lib/python2.7/site-packages
Requires:

Running command:

/usr/bin/rethinkdb dump --password '*******' --temp-dir $TEMPDIR --file "$BACKUPDIR/rethinkdb_dump_$(date +'%Y%m%d_%H%M%S').tar.gz"

Python driver fails with unexpected bytes

I'm running tasks using celery in Python with python rethinkdb driver. I followed Ten-minute guide with Rethinkdb and Python and it seems to be running fine for a while, but it fails with UnicodeDecodeError after some time.

[2017-06-18 16:24:00,102: WARNING/PoolWorker-6] b'{"deleted":0,"errors":0,"generated_keys":["cef00e30-fdb1-443e-8313-54ab6da8812d"],"inserted":1,"replaced":0,"skipped":0,"unchanged":0}]}\x02\x00\x00\x00\x00\x00\x00\x00\x94\x00\x00\x00'
[2017-06-18 16:24:00,105: ERROR/PoolWorker-6] Task tasks.crawl[ddaffbc4-e56d-4da8-8304-c5abd243be79] raised unexpected: UnicodeDecodeError('utf-8', b'{"deleted":0,"errors":0,"generated_keys":["cef00e30-fdb1-443e-8313-54ab6da8812d"],"inserted":1,"replaced":0,"skipped":0,"unchanged":0}]}\x02\x00\x00\x00\x00\x00\x00\x00\x94\x00\x00\x00', 144, 145, 'invalid start byte')
Traceback (most recent call last):
  File "/home/ssut/.local/share/virtualenvs/server-NO78Iaa1/lib/python3.6/site-packages/celery/app/trace.py", line 367, in trace_task
  R = retval = fun(*args, **kwargs)
  File "/home/ssut/.local/share/virtualenvs/server-NO78Iaa1/lib/python3.6/site-packages/celery/app/trace.py", line 622, in __protected_call__
  return self.run(*args, **kwargs)
  File "/home/ssut/dev/.../tasks.py", line 215, in crawl
  r.table('logs').insert(log).run(conn)
  File "/home/ssut/.local/share/virtualenvs/server-NO78Iaa1/lib/python3.6/site-packages/rethinkdb/ast.py", line 123, in run
  return c._start(self, **global_optargs)
  File "/home/ssut/.local/share/virtualenvs/server-NO78Iaa1/lib/python3.6/site-packages/rethinkdb/net.py", line 626, in _start
  return self._instance.run_query(q, global_optargs.get('noreply', False))
  File "/home/ssut/.local/share/virtualenvs/server-NO78Iaa1/lib/python3.6/site-packages/rethinkdb/net.py", line 459, in run_query
  res = self._read_response(query)
  File "/home/ssut/.local/share/virtualenvs/server-NO78Iaa1/lib/python3.6/site-packages/rethinkdb/net.py", line 507, in _read_response
  self._parent._get_json_decoder(query))
  File "/home/ssut/.local/share/virtualenvs/server-NO78Iaa1/lib/python3.6/site-packages/rethinkdb/net.py", line 74, in __init__
  json_str = json_str.decode('utf-8')
  UnicodeDecodeError: 'utf-8' codec can't decode byte 0x94 in position 144: invalid start byte

The first line is a hint I modified to check where the error comes from:
image
I think the root cause of this issue is that the part of the socket receives response from the rethinkdb server does the wrong behavior.

Error restoring a database

This is the output that I receive when trying to restore (on root):

Extracting archive file...
  Done (0 seconds)
Traceback (most recent call last):
  File "/usr/local/bin/rethinkdb-restore", line 11, in <module>
    sys.exit(main())
  File "/usr/local/lib/python2.7/dist-packages/rethinkdb/_restore.py", line 284, in main
    do_restore(options)
  File "/usr/local/lib/python2.7/dist-packages/rethinkdb/_restore.py", line 255, in do_restore
    sources = _import.parse_sources(options)
  File "/usr/local/lib/python2.7/dist-packages/rethinkdb/_import.py", line 1444, in parse_sources
    primary_key, indexes, write_hook = parse_info_file(info_path)
  File "/usr/local/lib/python2.7/dist-packages/rethinkdb/_import.py", line 1346, in parse_info_file
    return primary_key, indexes, write_hook
UnboundLocalError: local variable 'write_hook' referenced before assignment

RethinkDB version:

rethinkdb 2.3.6.srh.1~0bionic (CLANG 6.0.0 (tags/RELEASE_600/final))

The python driver is at version 2.4.1.

rethinkdb dump -- fails

Describe the bug
rethinkdb dump is broken

To Reproduce
Steps to reproduce the behavior:
rethinkdb dump -c db.site.com:28015 --password-file pass.txt -f backup.tar.gz

Expected behavior
rethinkdb dump completes successfully

Screenshots
If applicable, add screenshots to help explain your problem.

System info

  • OS: macOS Catalina
  • rethinkdb: 2.4.0
  • Python client version: 2.4.4

Additional context

            cluster metadata.  You will need to recreate your cluster setup yourself after
            you run 'rethinkdb-restore'.
  Exporting to temporary directory...
    [                                        ]   0%
    0 rows exported from 14 tables, with 18 secondary indexes, and 0 hook functions
Unrecognized TermType: 190.
Unrecognized TermType: 190.
Unrecognized TermType: 190.
Unrecognized TermType: 190.
Unrecognized TermType: 190.
Unrecognized TermType: 190.
Unrecognized TermType: 190.
Unrecognized TermType: 190.
Unrecognized TermType: 190.
Unrecognized TermType: 190.
Unrecognized TermType: 190.
Unrecognized TermType: 190.
Unrecognized TermType: 190.
Unrecognized TermType: 190.
Errors occurred during export
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/rethinkdb/_dump.py", line 169, in main
    _export.run(options)
  File "/usr/local/lib/python3.7/site-packages/rethinkdb/_export.py", line 536, in run
    run_clients(options, working_dir, db_table_set)
  File "/usr/local/lib/python3.7/site-packages/rethinkdb/_export.py", line 479, in run_clients
    raise RuntimeError("Errors occurred during export")
RuntimeError: Errors occurred during export
Error: export failed, Errors occurred during export```

[proposal] Drop Python 2 support

As of January 2020, the support Python 2 officially over. Since the code is lot more complex because of the Python 2 and 3 support, I suggest to drop Python 2 in favour of 3.

I would propose the following:
On a separate branch refactor the code to support only Python 3. Since RethinkDB 2.5.0 will introduce bigger changes (storage engine change), that would be a great opportunity to release the Python 2 version removal at that time when we release the DB itself.

Refactor async connections into an internal api.

Description

Refactor async connections into an internal api. This is probably going to look like JSONEncode/Decode injection with defaults and sensible overrides such as per connection opened.

net_asyncio Connection,....

Hi!

I'm using with asyncio rethinkdb and on the last version (2.4.2) i hav this problem

Describe the bug

    conn = await rc.async_connect()
  File "/home/david/Proyectos/DataDBS/datadbs/rethinkdb.py", line 72, in async_connect
    raise ex
  File "/home/david/Proyectos/DataDBS/datadbs/rethinkdb.py", line 62, in async_connect
    host=dbhost, **kwargs)
  File "/home/david/.virtualenvs/aguita37/lib/python3.7/site-packages/rethinkdb/__init__.py", line 96, in connect
    return self.make_connection(self.connection_type, *args, **kwargs)
  File "/home/david/.virtualenvs/aguita37/lib/python3.7/site-packages/rethinkdb/net.py", line 751, in make_connection
    **kwargs)
  File "/home/david/.virtualenvs/aguita37/lib/python3.7/site-packages/rethinkdb/asyncio_net/net_asyncio.py", line 333, in __init__
    super(Connection, self).__init__(ConnectionInstance, *args, **kwargs)
TypeError: super(type, obj): obj must be an instance or subtype of type

During handling of the above exception, another exception occurred:

I solved, in a while, with these changes:

On net_asyncio.py, line 333:
super().init(ConnectionInstance, *args, **kwargs)

To Reproduce
Steps to reproduce the behavior:

  1. Create a new connection with await....
    Using this module:
    https://gitlab.com/pineiden/datadbs
    Method: async_connect

Expected behavior
Connect like before

System info

  • Debian 9
  • RethinkDB Version: [2.4.0]
  • Python client version: 3.7

Additional context

Adding `async` and `await` support for Python 3.5

Hi guys!

With the advent of async and await keywords in Python 3.5, I think we need to add some convenience methods to the connection object that is returned when we make a r.connect(... function call. This would not be difficult to do, because we can already do a yield r.connect anyways.

So, the proposal is to add an __await__ method to the connection object. This will not be hard to implement though, as you can see with aiohttp's await implementation:

@asyncio.coroutine
def __iter__(self):
    resp = yield from self._coro
    return resp
if PY_35:
    def __await__(self):
        resp = yield from self._coro
        return resp

Write end to end tests on each async connection.

Description

Write end to end tests on each async connection. These tests will need to exercise change feeds and out of order requests for data. First #18 should be merged and Gevent completed for step 3 and 4

Python driver: auto reconnect options?

Is there any better way of getting a connection object that automatically reconnects than below? Closed connections keep throwing exceptions in random places in my code. I really don't care if my connection closed itself for whatever reason, I just want queries to run, so I'm trying to implement something like this 👍

an instance patch

import types

c = r.connect()

def auto_reconnect(self):
    if self._instance is None or not self._instance.is_open():
        self.reconnect()
    
c.check_open = types.MethodType( auto_reconnect, c )

c.close()

# Succeeds
r.db('simulator').table('watch_list').count().run(c)

or a general monkey patch

from rethinkdb import Connection

def auto_reconnect(self):
    if self._instance is None or not self._instance.is_open():
        self.reconnect()

Connection.check_open = auto_reconnect

rethinkdb dump failing

Ran the following

apt-get -y  update && apt-get install -y  python python-pip && pip install rethinkdb
I am getting the following error
root@bd07b97dd7c9:/data# rethinkdb dump -c 127.0.0.1:28015  --password-file /data/pw
Traceback (most recent call last):
  File "/usr/local/bin/rethinkdb-dump", line 11, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.4/dist-packages/rethinkdb/_dump.py", line 143, in main
    options = parse_options(argv or sys.argv[2:], prog=prog)
  File "/usr/local/lib/python3.4/dist-packages/rethinkdb/_dump.py", line 94, in parse_options
    options, args = parser.parse_args(argv)
  File "/usr/local/lib/python3.4/dist-packages/rethinkdb/utils_common.py", line 359, in parse_args
    options, args = super(CommonOptionsParser, self).parse_args(*args, **kwargs)
  File "/usr/lib/python3.4/optparse.py", line 1386, in parse_args
    stop = self._process_args(largs, rargs, values)
  File "/usr/lib/python3.4/optparse.py", line 1426, in _process_args
    self._process_long_opt(rargs, values)
  File "/usr/lib/python3.4/optparse.py", line 1500, in _process_long_opt
    option.process(opt, value, values, self)
  File "/usr/lib/python3.4/optparse.py", line 778, in process
    value = self.convert_value(opt, value)
  File "/usr/lib/python3.4/optparse.py", line 770, in convert_value
    return self.check_value(opt, value)
  File "/usr/lib/python3.4/optparse.py", line 765, in check_value
    return checker(self, opt, value)
TypeError: file_contents() takes 2 positional arguments but 3 were given

Generate proto files on git install

Currently, protocol buffer files are not generated during a git install. They require a separate make prepare step, which means we can't use a git reference with tools like pip or pipenv. This leads to errors like:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/tom/reinfer/rethinkdb-python/rethinkdb/ast.py", line 28, in <module>
    from rethinkdb import ql2_pb2
ImportError: cannot import name 'ql2_pb2' from 'rethinkdb' (/home/tom/reinfer/rethinkdb-python/rethinkdb/__init__.py)

It's possible to run external commands like make during the install process, see this SO question. It should also be possible to only run these steps if the protocol buffer files are not already present (i.e. during a git install, not in a PyPI install).

This isn't something I've done before, but happy to give it a go if others are open to the idea?

Create proper travis ci config

The current Travis CI setup is a mess (sorry for that by the way). We need to figure out how to do it better.

A partially related issue is that we can not run integration tests on forks because of encrypted env variables. Potentially we could turn them off for PRs. What do you think @grandquista?

Bring back signleton

Before it was easy just to write

import rethinkdb as r

conn = r.connect(...)
r.table(...).run(conn)

But documentation (simply README) of rethinkdb python driver >=2.4.0 insist that now the proper way to do it is this:

from rethinkdb import RethinkDB

r = RethinkDB()
conn = r.connect(...)
r.table(...).run(conn)

It's OK to do it in single function. But it is NOT OK to do it in multiple functions and pass around the rethinkdb_instance parameter in each function that makes requests to your rethinkdb.

Python 3.7+ asyncio loop type Syntax Error

I have rethinkdb==2.3.0.post6 and Python 3.7.2.
When I set r.set_loop_type('asyncio'), I get an error:

File ".../rethinkdb/asyncio_net/net_asyncio.py", line 114
    asyncio.async(self.conn._parent._continue(self))
                ^
SyntaxError: invalid syntax

Fix this, please.

Issue building with Docker with 2.4.2

Describe the bug
When building docker container to the 2.4.2 I get the following error. This has been tested on two different docker containers that utilize the RethinkDB Python Driver. I have had to use the 2.4.1 driver in order to have docker run.

from rethinkdb import RethinkDB

File "/usr/local/lib/python3.6/dist-packages/rethinkdb/init.py", line 29, in
version = version.VERSION
AttributeError: module 'rethinkdb.version' has no attribute 'VERSION'

[BUG] .do() methods are not working as expected

Back in 2.3.0 I tested this the .do(), and it still doesn't work i 2.4.

r.db_list().contains(DATABASE).do(r.branch(True,{ 'dbs_created': 0 },r.db_create(DATABASE))).run(connection)

Expected Behavior

It is expected that the .do() would execute the lamda functions and r.branch() functions

Current Behavior

The .do() function does not execute correctly

Steps to Reproduce

  1. Open the Python interpreter
  2. run the following piece of code:
from rethinkdb import RethinkDB
r = RethinkDB()
connection = r.connect(host="10.30.0.94", port=28015)
r.db_list().contains("non_existant_db").do(r.branch(
        True,
        { 'dbs_created': 0 },
        r.db_create("non_existant_db")
    )).run(connection)
  1. It will produce {'dbs_created': 0} for every database name

Context (Environment)

Python version: 3.7.2
Driver Version: 2.4.1
Server Version: rethinkdb 2.3.6~0jessie (GCC 4.9.2)

In a application, I needed to use .do() to watch a .changes() routine. Not being able to use .do() requires me to create a loop ( and this breaks the rest of my code). It also makes some code bigger than needed.

Another Secptum user reported using a similar query, achieving the following results:

r.db_list().do(lambda result: print(result.__dict__)).run(connection)
> {'_args': [<RqlQuery instance: 4 >], 'optargs': {}}

Python driver incorrectly requires datetime.date type to have tzinfo

Python datetime.date types cannot have tzinfo because timezone only has meaning when a time is associated with it. Here's a simple script to demonstrate the issue:

try:
    r.table_create("test_date").run(conn)
    r.table("test_date").wait().run(conn)
except r.errors.ReqlOpFailedError as e:
    pass

try:
    r.table("test_date").insert({"id": 1, "date": date(2016, 1, 1)})
except r.errors.ReqlOpFailedError as e:
    print("Failed:", e)

Asyncio support: 'TypeError: 'async for' received an object from __aiter__ that does not implement __anext__: generator'

Cannot use cursor with async for.

Expected Behavior

cursor = await marvel_heroes.run(connection)
    async for hero in cursor:
        print(hero['name'])

Current Behavior

cursor = await marvel_heroes.run(connection)
    async for hero in cursor:
        print(hero['name'])

TypeError: 'async for' received an object from `__aiter__` that does not implement __anext__: generator

Possible Solution

Remove @asyncio.coroutine decorator from __aiter__ method of AsyncioCursor on line 95 in net_asyncio.py

Steps to Reproduce

Given above.

Context (Environment)

Python 3.7.2, you can't use cursors in async for

Show stopping bugs in `rethinkdb.logger`

There are a few show stopping bugs in rethinkdb.logger.

The first issue is that the call to logging.basicConfig() is incorrect. The format argument should be a str, not a logging.Formatter. This causes all logging calls to produce the following error:

Traceback (most recent call last):
  File "/usr/lib/python3.7/logging/__init__.py", line 1034, in emit
    msg = self.format(record)
  File "/usr/lib/python3.7/logging/__init__.py", line 880, in format
    return fmt.format(record)
  File "/usr/lib/python3.7/logging/__init__.py", line 620, in format
    if self.usesTime():
  File "/usr/lib/python3.7/logging/__init__.py", line 588, in usesTime
    return self._style.usesTime()
  File "/usr/lib/python3.7/logging/__init__.py", line 430, in usesTime
    return self._fmt.find(self.asctime_search) >= 0
AttributeError: 'Formatter' object has no attribute 'find'

The corrected code should be:

        log_format = '%(asctime)s - %(name)s - %(levelname)s - %(message)s'
        logging.basicConfig(format=log_format)

The second issue is that a library module should never call logging.basicConfig() because it reconfigures global logging and affects all other modules. For example, in an application I am developing, I have logging calls throughout the code. As soon as I import rethinkdb my own code is no longer able to do any logging due to the bug above. As an application developer, I don't want a module to change the way my application does its logging.

Finally, I'm not sure what the purpose of this DriverLogger class is. It has some weird features for exception handling and "write to console", but these all seem like operations only used for debugging that could be handled using Python's native logging instead.

I am happy to write up a PR to fix any or all of these issues, but will wait for feedback on this issue before doing so. Please let me know what you think.

Support for binary data in primary key

I'm getting an error trying to use binary data as a primary key, Python 3.5.2 driver version 2.3.0.post6, RethinkDB 2.3.6.

Primary key address cannot be changed (null -> {\n\t"address":{"$reql_type$":"BINARY","data":"usABAAAzM09RHqvfOr+yo6lmvxER/wAAAAAAAAAAAAAAAAA="}

Am I doing something wrong or is there a known bug?

Set loop type back to synchronous in set_loop_type

So, after I do r.set_loop_type('asyncio') the returning value from r.connect changes to asyncio Future. But what if I want to use normal version of r.connect in some threads that are lacking an asyncio loop? Right now it leads to some consequences - 'generator' objects in unexpected places.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.