Git Product home page Git Product logo

tornado-mysql's Introduction

Documentation Status codecov

PyMySQL

This package contains a pure-Python MySQL client library, based on PEP 249.

Requirements

  • Python -- one of the following:
  • MySQL Server -- one of the following:

Installation

Package is uploaded on PyPI.

You can install it with pip:

$ python3 -m pip install PyMySQL

To use "sha256_password" or "caching_sha2_password" for authenticate, you need to install additional dependency:

$ python3 -m pip install PyMySQL[rsa]

To use MariaDB's "ed25519" authentication method, you need to install additional dependency:

$ python3 -m pip install PyMySQL[ed25519]

Documentation

Documentation is available online: https://pymysql.readthedocs.io/

For support, please refer to the StackOverflow.

Example

The following examples make use of a simple table

CREATE TABLE `users` (
    `id` int(11) NOT NULL AUTO_INCREMENT,
    `email` varchar(255) COLLATE utf8_bin NOT NULL,
    `password` varchar(255) COLLATE utf8_bin NOT NULL,
    PRIMARY KEY (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin
AUTO_INCREMENT=1 ;
import pymysql.cursors

# Connect to the database
connection = pymysql.connect(host='localhost',
                             user='user',
                             password='passwd',
                             database='db',
                             cursorclass=pymysql.cursors.DictCursor)

with connection:
    with connection.cursor() as cursor:
        # Create a new record
        sql = "INSERT INTO `users` (`email`, `password`) VALUES (%s, %s)"
        cursor.execute(sql, ('[email protected]', 'very-secret'))

    # connection is not autocommit by default. So you must commit to save
    # your changes.
    connection.commit()

    with connection.cursor() as cursor:
        # Read a single record
        sql = "SELECT `id`, `password` FROM `users` WHERE `email`=%s"
        cursor.execute(sql, ('[email protected]',))
        result = cursor.fetchone()
        print(result)

This example will print:

{'password': 'very-secret', 'id': 1}

Resources

License

PyMySQL is released under the MIT License. See LICENSE for more information.

tornado-mysql's People

Contributors

addbook avatar bbigras avatar clarkevans avatar clelland avatar cleure avatar cyphergrue avatar djmitche avatar emosbaugh avatar exarkun avatar infinitylx avatar j0hnsmith avatar jakesylvestre avatar jamadden avatar jcotton1123 avatar jettify avatar julien-duponchelle avatar jvankoten avatar lecram avatar methane avatar mgrechukh avatar mlongtin0 avatar mr-ping avatar petehunt avatar raggiskula avatar rfk avatar sethmurphy avatar valcapri avatar wraziens avatar yoriksar avatar zzzeek avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

tornado-mysql's Issues

Hi,I try pool demo,it doesn't run.

centos7.1
python2.7.5

it‘s block,and doesn't run print(n, cur.fetchall()).

[root@localhost Ease]# python t1.py
0 sleeping 1 seconds
1 sleeping 1 seconds
2 sleeping 1 seconds
3 sleeping 1 seconds
4 sleeping 1 seconds
5 sleeping 1 seconds
6 sleeping 1 seconds
7 sleeping 1 seconds
8 sleeping 1 seconds
9 sleeping 1 seconds

IOLoop blocked

[W 150626 18:40:53 ioloop:326] IOLoop blocked for 1.000000 seconds in
      File "server.py", line 104, in <module>
        loop.start()
      File "/Users/tk/Projects/dr/lib/python2.7/site-packages/tornado/ioloop.py", line 784, in start
        self._run_callback(callback)
      File "/Users/tk/Projects/dr/lib/python2.7/site-packages/tornado/ioloop.py", line 568, in _run_callback
        ret = callback()
      File "/Users/tk/Projects/dr/lib/python2.7/site-packages/tornado/stack_context.py", line 314, in wrapped
        ret = fn(*args, **kwargs)
      File "/Users/tk/Projects/dr/lib/python2.7/site-packages/tornado/gen.py", line 893, in <lambda>
        self.future, lambda f: self.run())
      File "/Users/tk/Projects/dr/lib/python2.7/site-packages/tornado/gen.py", line 812, in run
        yielded = self.gen.send(value)
      File "/Users/tk/Projects/dr/lib/python2.7/site-packages/tornado_mysql/connections.py", line 1066, in read
        yield self._read_result_packet(first_packet)
      File "/Users/tk/Projects/dr/lib/python2.7/site-packages/tornado/gen.py", line 224, in wrapper
        Runner(result, future, yielded)
      File "/Users/tk/Projects/dr/lib/python2.7/site-packages/tornado/gen.py", line 757, in __init__
        self.run()
      File "/Users/tk/Projects/dr/lib/python2.7/site-packages/tornado/gen.py", line 812, in run
        yielded = self.gen.send(value)
      File "/Users/tk/Projects/dr/lib/python2.7/site-packages/tornado_mysql/connections.py", line 1119, in _read_result_packet
        yield self._read_rowdata_packet()
      File "/Users/tk/Projects/dr/lib/python2.7/site-packages/tornado/gen.py", line 224, in wrapper
        Runner(result, future, yielded)
      File "/Users/tk/Projects/dr/lib/python2.7/site-packages/tornado/gen.py", line 757, in __init__
        self.run()
      File "/Users/tk/Projects/dr/lib/python2.7/site-packages/tornado/gen.py", line 812, in run
        yielded = self.gen.send(value)
      File "/Users/tk/Projects/dr/lib/python2.7/site-packages/tornado_mysql/connections.py", line 1155, in _read_rowdata_packet
        packet = yield self.connection._read_packet()
      File "/Users/tk/Projects/dr/lib/python2.7/site-packages/tornado/gen.py", line 224, in wrapper
        Runner(result, future, yielded)
      File "/Users/tk/Projects/dr/lib/python2.7/site-packages/tornado/gen.py", line 746, in __init__
        self.running = False

Cannot connect to MySQL after upgrade to v8

I have a huge website with Tornado-MySQL which code I haven't modified for almost 2 years because it worked fine all this time. However, a day ago I updated MySQL Server from 5.7 to 8 (because of needs of my another project on that server), and now my Tornado website doesn't work: Tornado-MySQL cannot connect to the new MySQL Server. Here is the error text:

ERROR:tornado.application:Future exception was never retrieved: Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/tornado/gen.py", line 1141, in run
    yielded = self.gen.throw(*exc_info)
  File "/aivanf/Tornado/database.py", line 33, in execute
    cur = yield POOL.execute(query, params)
  File "/usr/local/lib/python2.7/dist-packages/tornado/gen.py", line 1133, in run
    value = future.result()
  File "/usr/local/lib/python2.7/dist-packages/tornado/concurrent.py", line 261, in result
    raise_exc_info(self._exc_info)
  File "/usr/local/lib/python2.7/dist-packages/tornado/gen.py", line 1141, in run
    yielded = self.gen.throw(*exc_info)
  File "/usr/local/lib/python2.7/dist-packages/tornado_mysql/pools.py", line 122, in execute
    conn = yield self._get_conn()
  File "/usr/local/lib/python2.7/dist-packages/tornado/gen.py", line 1133, in run
    value = future.result()
  File "/usr/local/lib/python2.7/dist-packages/tornado/concurrent.py", line 261, in result
    raise_exc_info(self._exc_info)
  File "/usr/local/lib/python2.7/dist-packages/tornado/gen.py", line 1141, in run
    yielded = self.gen.throw(*exc_info)
  File "/usr/local/lib/python2.7/dist-packages/tornado_mysql/__init__.py", line 88, in connect
    yield conn.connect()
  File "/usr/local/lib/python2.7/dist-packages/tornado/gen.py", line 1133, in run
    value = future.result()
  File "/usr/local/lib/python2.7/dist-packages/tornado/concurrent.py", line 261, in result
    raise_exc_info(self._exc_info)
  File "/usr/local/lib/python2.7/dist-packages/tornado/gen.py", line 1141, in run
    yielded = self.gen.throw(*exc_info)
  File "/usr/local/lib/python2.7/dist-packages/tornado_mysql/connections.py", line 823, in connect
    2003, "Can't connect to MySQL server on %r (%s)" % (self.host, e))
OperationalError: (2003, "Can't connect to MySQL server on '127.0.0.1' (255)")

Below are listed the versions of my packages. I have updated them to last available ones.

  • OS: Ubuntu 16.04.5 LTS
  • MySQL: 8.0.13
  • Python: checked both 2.7 and 3.5
  • Tornado: 5.1.1
  • Tornado-MySQL: 0.5.1

In addition, during installation of MySQL Server 8, I specified to use legacy users stuff to maintain compatibility with my Python apps, and MySQLdb package (also called mysqlclient, version 1.3.13) works well.

How to fix the problem??

Random Response corruption

Hi, I have been using tornado with tornado-mysql. And we run this stack in high throughput servers on AWS (EC2 and RDS). Till a week back everything was running smoothly, but recently we have been getting the the following errors from our code. The diagnosis suggests that the Tornado-MYSQL driver has been responding with empty "tuple"'s, that too quite erratically.

Traceback (most recent call last):
  File "/home/ec2-user/server/core-api/app-server.py", line 606, in get
    auth = await check_auth(pool=pool, token=token, session_size=config.session_size)
  File "/home/ec2-user/server/core-api/core/users/auth.py", line 45, in check_auth
    if (timenow - ts) < session_size * 3600):
TypeError: unsupported operand type(s) for -: 'int' and 'str'

and

Traceback (most recent call last):
  File "/home/ec2-user/server/core-api/app-server.py", line 606, in get
    auth = await check_auth(pool=pool, token=token, session_size=config.session_size)
  File "/home/ec2-user/server/core-api/core/users/auth.py", line 39, in check_auth
    ts = check[5]
IndexError: tuple index out of range

The query for both the exception is as follows:

SELECT u_id, chain_id, rest_id, designation, BIN(permission), ts 
FROM user.auth 
WHERE o_tk = %s;

What could be the issue and how do I solve it?
Any help is appreciated as tornado-mysql is a crucial part of our stack.

cannot update

sql2='update watch_monitor_profile set data='test' where id=43127'
yield cur1.execute(sql2)

but it not work,and nothing happend

Stream is closed

Hi, I encountered a problem when using the connection pool.

I see the source code

    def _get_conn(self):
        now = self.io_loop.time()

        # Try to reuse in free pool
        while self._free_conn:
            conn = self._free_conn.popleft()
            if now - conn.connected_time > self.max_recycle_sec:
                self._close_async(conn)
                continue
            log.debug("Reusing connection from pool: %s", self.stat())
            fut = Future()
            fut.set_result(conn)
            return fut
    ......
conn = self._free_conn.popleft()
if now - conn.connected_time > self.max_recycle_sec:
    self._close_async(conn)
    continue

If conn is closed at this time,
but now - conn.connected_time <= self.max_recycle_sec,
Using conn at this point will throw tornado.iostream.StreamClosedError: Stream is closed
But I can't find where this conn was closed.

Get Exception socket.error: error(9, 'Bad file descriptor') in testing.

Hello, INADA:
I write a class inherits pools.Pool names DBServer, and it is a singleton class. I also write a class DBServerTestCase that inherit AsyncTestCase.
This is part of DBServerTestCase:

class DBServerTestCase(AsyncTestCase):
    def setUp(self):
        super(DBServerTestCase, self).setUp()
        self.db = DBServer(dict(host='127.0.0.1', port=3306, user='root', passwd='root', db='mydatabase', cursorclass=DictCursor),
            max_idle_connections=2,
            max_recycle_sec=3,
            max_open_connections=5)

...Many test methods...

When I run this test case, I get many errors like:

Exception socket.error: error(9, 'Bad file descriptor') in <bound method Connection.__del__ of <tornado_mysql.connections.Connection object at 0x106753ed0>> ignored

I don't override tearDown method.
If I just run one method, everything is fine, no error.
Is it appropriate that writing setUp method like this?

Release a new version to fix the pool hungup bug

Hi,
I use PyPi upgrade Tornado-MySQL to latest stable version(0.5) and I find all connection pool don't work.
If a pool don't set max_open_connections option, the pool won't create new connection and cause program hangup.
This bug already fixed in master branch by this commit, but it don't release a new version after that.
I hope you can release a new version to PyPi as soon as possible.

what is the problem?

This library is experimental. Don't use for production unless you can fix problem yourself.

I like Tornado-MySQL very much. it is useful.
I am concerned about the problem.
what is the problem?
which condition can cause the problem?

Async support for PyMySQL

Hi Naoki,

I've decided to open a ticket here, because I did not find any other reliable way to contact you.

Anyways,
I'm co-maintainer of python-momoko - its a psycopg2 wrapper for Tornado. It utilizes psycopg2 excellent async interface and contains very little amount of psycopg2-specific code.

I was thinking about generalizing momoko by adding MySQL support as well, but I did not find any async support in neither PyMySQL nor mysqlclient-python (both of them you are happen to maintain).

So I've thought I'll hack such support myself into PyMySQL and then plug it to momoko (or to any other event-based library for this matter). So far have have a proof of concept code that can only connect, and does not support SSL. (I've roughly merged it against your latest changes, so its most probably broken. I've pushed just to demonstrate).

Now I've run into your new Tornado-MySQL project and looks like you are forking PyMySQL all together for async support Tornado-specific way...

The thing is that Tornado *SQL support is rather sparse; everyone seems to do it their own way. Its not just for Tornado but for async + sql in general (psycopg2 being a notable exception).
I think there is an opportunity here to standardize it a bit.

  • sql libraries can expose a standard async API (can be based on psycopg2 definition) which is independent on particular event loop implementation
  • then Tornado or any other event library can easily utilize them

So, my question is whether you can help with adding psycopg2-like async support to PyMySQL, so I'll can concentrate my efforts on momoko. I think it will provide more mature MySQL support in tornado much sooner.

P.S. My idea regarding async+PyMySQL is as follows:

  • Convert all current functions that access socket to be async, with yields, etc.
  • Then create sync versions of those functions that will call self.poll() by themselves.
  • This way we'll have minimum amount of code duplication

P.P.S. I'm on vacation till October, and will not be able to contribute any code till then.

"yielded unknown object None" Error on connect

I was just trying out the demo's I get the following error in both when the code attempts to open a connection

File "/Library/Python/2.7/site-packages/tornado_mysql/connections.py", line 820, in connect
    2003, "Can't connect to MySQL server on %r (%s)" % (self.host, e))
tornado_mysql.err.OperationalError: (2003, "Can't connect to MySQL server on 'databaseserver' (yielded unknown object None)")

I am running on Yosemite with Python 2.7. MySQL server is running on a remote VM. I can confirm this machine will accept remote connections, I have tested this using the blocking MySQL tornado library.

OperationalError: (2006, 'MySQL server has gone away (Stream is closed)')

My code goes like this:

            print "trying to connect to MySQL"
            conn = yield tornado_mysql.connect(host='127.0.0.1',
                                               user='root',
                                               db='camera')
            cur = conn.cursor()
            print cur
            query = "INSERT INTO ...."
            yield cur.execute(query)                         # this is getting executed!
            cur.close()
            # if conn is not None:
            conn.commit()
            yield conn.close_async()                     # tried conn.close() as well, Both throws an exception:
ERROR:tornado.application:Future exception was never retrieved: Traceback (most recent call last):
  File "/Users/ishansharma/workplace/server-camera/venv/lib/python2.7/site-packages/tornado/gen.py", line 1014, in run
    yielded = self.gen.throw(*exc_info)
  File "/Users/ishansharma/workplace/server-camera/venv/lib/python2.7/site-packages/tornado_mysql/connections.py", line 679, in commit
    yield self._read_ok_packet()
  File "/Users/ishansharma/workplace/server-camera/venv/lib/python2.7/site-packages/tornado/gen.py", line 1008, in run
    value = future.result()
  File "/Users/ishansharma/workplace/server-camera/venv/lib/python2.7/site-packages/tornado/concurrent.py", line 232, in result
    raise_exc_info(self._exc_info)
  File "/Users/ishansharma/workplace/server-camera/venv/lib/python2.7/site-packages/tornado/gen.py", line 1014, in run
    yielded = self.gen.throw(*exc_info)
  File "/Users/ishansharma/workplace/server-camera/venv/lib/python2.7/site-packages/tornado_mysql/connections.py", line 654, in _read_ok_packet
    pkt = yield self._read_packet()
  File "/Users/ishansharma/workplace/server-camera/venv/lib/python2.7/site-packages/tornado/gen.py", line 1008, in run
    value = future.result()
  File "/Users/ishansharma/workplace/server-camera/venv/lib/python2.7/site-packages/tornado/concurrent.py", line 232, in result
    raise_exc_info(self._exc_info)
  File "/Users/ishansharma/workplace/server-camera/venv/lib/python2.7/site-packages/tornado/gen.py", line 1014, in run
    yielded = self.gen.throw(*exc_info)
  File "/Users/ishansharma/workplace/server-camera/venv/lib/python2.7/site-packages/tornado_mysql/connections.py", line 845, in _read_packet
    raise OperationalError(2006, "MySQL server has gone away (%s)" % (e,))
OperationalError: (2006, 'MySQL server has gone away (Stream is closed)')

Why doesn't this repo make mysql faster than torndb?

My test code just like this


#!/usr/bin/env python
# -*-coding:utf-8-*-

import sys
reload(sys)
sys.setdefaultencoding("utf-8")

import torndb
import tornado.web
import tornado.httpserver
import tornado.ioloop
import tornado.options
from tornado.gen import coroutine, sleep
from tornado.options import options, define

from tornado_mysql.pools import Pool

define("port", type=int, default=8888)
options.parse_command_line()

sync_conn = torndb.Connection(host="localhost:3306", user="xxx", password="xxx", database="xxx")
POOL = Pool(
    dict(host='127.0.0.1', port=3306, user='xxx', passwd='xxx', db='xxx'),
    max_idle_connections=100,)

class TorndbHandler(tornado.web.RequestHandler):
    @coroutine
    def get(self):
        sql = "select * from user"
        res = sync_conn.query(sql)
        return


class PoolHandler(tornado.web.RequestHandler):
    @coroutine
    def get(self):
        sql = "select * from user"
        res = yield POOL.execute(sql)
        res.fetchall()
        return

if __name__ == "__main__":
    settings = {
        'debug': True
    }
    app = tornado.web.Application(
        handlers=[
            (r'/torndb', TorndbHandler),
            (r'/asyncpooldb', PoolHandler),
        ],
        **settings
        )
    tornado.httpserver.HTTPServer(app).listen(options.port)
    loop = tornado.ioloop.IOLoop.instance()
    loop.start()

and i use ab test, the result like this :

>>ab -c 10 -n 40 http://127.0.0.1:8888/torndb

Concurrency Level:      10
Time taken for tests:   0.680 seconds
Complete requests:      40
Failed requests:        0
Total transferred:      7760 bytes
HTML transferred:       0 bytes
Requests per second:    58.84 [#/sec] (mean)
Time per request:       169.963 [ms] (mean)
Time per request:       16.996 [ms] (mean, across all concurrent requests)
Transfer rate:          11.15 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.1      0       0
Processing:    16  148  93.1    148     679
Waiting:       16  148  93.1    148     679
Total:         16  148  93.1    149     680
>>ab -c 10 -n 40 http://127.0.0.1:8888/asyncpooldb

Concurrency Level:      10
Time taken for tests:   1.605 seconds
Complete requests:      40
Failed requests:        0
Total transferred:      7760 bytes
HTML transferred:       0 bytes
Requests per second:    24.91 [#/sec] (mean)
Time per request:       401.371 [ms] (mean)
Time per request:       40.137 [ms] (mean, across all concurrent requests)
Transfer rate:          4.72 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.1      0       1
Processing:   389  401   9.2    405     413
Waiting:      389  401   9.2    405     413
Total:        389  401   9.3    406     413

The mysql user table has hundreds of lines.

The result shows that torndb is faster than this repo. Is this correct? Why? Shouldn't async be faster?

Best Regards,

pools how to commit?

I use the POOL to connect to my database,
select is worked fine,but the update and insert doing nothing,how can I use commit in the pool,
please give me a example,thanks.

multiple queries with warnings in one execute()

sql = "DROP PROCEDURE IF EXISTS test; CREATE PROCEDURE test() SELECT 1"
yield db.execute(sql)

The DROP would cause a warning if procedure not existed, but CREATE would succeed. result.warning_count would not be 0, but conn.show_warnings() would be empty.
Resulting uncaught exception.

can't get a connection when using connection pool if max_open_connections=0

File: tornado_mysql/pools.py
`

class Pool(object):

    def _get_conn(self):

`
my code:
mypool = tornado_mysql.pool.Pool(**{host:"localhost"})

then self._free_conn if empty forever, equal to false
self.max_open = 0, equal to false

now, the pool is waiting for a connection, but there is no chance to create a connection.

tornado_mysql_pools

thanks to fix the bug.

LOAD DATA LOCAL Broken?

I'm having issues getting LOAD DATA LOCAL INFILE... commands to run, getting the following error:

  File "<snip>/tornado_mysql/connections.py", line 1188, in _get_descriptions
    for i in range_type(self.field_count):
TypeError: 'NoneType' object cannot be interpreted as an integer

I checked and there is a test file for this functionality (https://github.com/PyMySQL/Tornado-MySQL/blob/master/tornado_mysql/tests/test_load_local.py), but the test doesn't run. Looks like they are old and I had to sprinkle yield statements to even get them to work w/ the tornado async interfaces. I moved those tests into a file that was being picked up (didn't investigate too much why the tests weren't running from that file), and indeed got the same errors as when I ran the queries manually.

Going against MySQL Server 5.5 (specifically Server version: 5.5.47-0ubuntu0.14.04.1).

Happy to provide more details and/or help fix. But this is a major roadblock for me and would appreciate any feedback / collaboration.

Full stacktrace:

2016-05-03 22:08:24,317 (unknown.audience) managepy.manage[40128]: CRITICAL App failed with status
Traceback (most recent call last):
  File "/Users/scott/.virtualenvs/audience/lib/python3.4/site-packages/managepy/manage.py", line 230, in run
    return_code = yield result
  File "/Users/scott/.virtualenvs/audience/lib/python3.4/site-packages/tornado/gen.py", line 807, in run
    value = future.result()
  File "/Users/scott/.virtualenvs/audience/lib/python3.4/site-packages/tornado/concurrent.py", line 209, in result
    raise_exc_info(self._exc_info)
  File "<string>", line 3, in raise_exc_info
  File "/Users/scott/.virtualenvs/audience/lib/python3.4/site-packages/tornado/gen.py", line 810, in run
    yielded = self.gen.throw(*sys.exc_info())
  File "src/audience/shared/commands/test.py", line 51, in run
    yield cxn.execute(query)
  File "/Users/scott/.virtualenvs/audience/lib/python3.4/site-packages/tornado/gen.py", line 807, in run
    value = future.result()
  File "/Users/scott/.virtualenvs/audience/lib/python3.4/site-packages/tornado/concurrent.py", line 209, in result
    raise_exc_info(self._exc_info)
  File "<string>", line 3, in raise_exc_info
  File "/Users/scott/.virtualenvs/audience/lib/python3.4/site-packages/tornado/gen.py", line 810, in run
    yielded = self.gen.throw(*sys.exc_info())
  File "/Users/scott/dev/tbcode/svc-audience/src/audience/shared/db_async.py", line 126, in execute
    yield cursor.execute(query)
  File "/Users/scott/.virtualenvs/audience/lib/python3.4/site-packages/tornado/gen.py", line 807, in run
    value = future.result()
  File "/Users/scott/.virtualenvs/audience/lib/python3.4/site-packages/tornado/concurrent.py", line 209, in result
    raise_exc_info(self._exc_info)
  File "<string>", line 3, in raise_exc_info
  File "/Users/scott/.virtualenvs/audience/lib/python3.4/site-packages/tornado/gen.py", line 810, in run
    yielded = self.gen.throw(*sys.exc_info())
  File "/Users/scott/.virtualenvs/audience/lib/python3.4/site-packages/tornado_mysql/cursors.py", line 132, in execute
    yield self._query(query)
  File "/Users/scott/.virtualenvs/audience/lib/python3.4/site-packages/tornado/gen.py", line 807, in run
    value = future.result()
  File "/Users/scott/.virtualenvs/audience/lib/python3.4/site-packages/tornado/concurrent.py", line 209, in result
    raise_exc_info(self._exc_info)
  File "<string>", line 3, in raise_exc_info
  File "/Users/scott/.virtualenvs/audience/lib/python3.4/site-packages/tornado/gen.py", line 810, in run
    yielded = self.gen.throw(*sys.exc_info())
  File "/Users/scott/.virtualenvs/audience/lib/python3.4/site-packages/tornado_mysql/cursors.py", line 389, in _query
    yield conn.query(q, unbuffered=True)
  File "/Users/scott/.virtualenvs/audience/lib/python3.4/site-packages/tornado/gen.py", line 807, in run
    value = future.result()
  File "/Users/scott/.virtualenvs/audience/lib/python3.4/site-packages/tornado/concurrent.py", line 209, in result
    raise_exc_info(self._exc_info)
  File "<string>", line 3, in raise_exc_info
  File "/Users/scott/.virtualenvs/audience/lib/python3.4/site-packages/tornado/gen.py", line 810, in run
    yielded = self.gen.throw(*sys.exc_info())
  File "/Users/scott/.virtualenvs/audience/lib/python3.4/site-packages/tornado_mysql/connections.py", line 731, in query
    yield self._read_query_result(unbuffered=unbuffered)
  File "/Users/scott/.virtualenvs/audience/lib/python3.4/site-packages/tornado/gen.py", line 807, in run
    value = future.result()
  File "/Users/scott/.virtualenvs/audience/lib/python3.4/site-packages/tornado/concurrent.py", line 209, in result
    raise_exc_info(self._exc_info)
  File "<string>", line 3, in raise_exc_info
  File "/Users/scott/.virtualenvs/audience/lib/python3.4/site-packages/tornado/gen.py", line 810, in run
    yielded = self.gen.throw(*sys.exc_info())
  File "/Users/scott/.virtualenvs/audience/lib/python3.4/site-packages/tornado_mysql/connections.py", line 858, in _read_query_result
    yield result.init_unbuffered_query()
  File "/Users/scott/.virtualenvs/audience/lib/python3.4/site-packages/tornado/gen.py", line 807, in run
    value = future.result()
  File "/Users/scott/.virtualenvs/audience/lib/python3.4/site-packages/tornado/concurrent.py", line 209, in result
    raise_exc_info(self._exc_info)
  File "<string>", line 3, in raise_exc_info
  File "/Users/scott/.virtualenvs/audience/lib/python3.4/site-packages/tornado/gen.py", line 810, in run
    yielded = self.gen.throw(*sys.exc_info())
  File "/Users/scott/.virtualenvs/audience/lib/python3.4/site-packages/tornado_mysql/connections.py", line 1084, in init_unbuffered_query
    yield self._get_descriptions()
  File "/Users/scott/.virtualenvs/audience/lib/python3.4/site-packages/tornado/gen.py", line 807, in run
    value = future.result()
  File "/Users/scott/.virtualenvs/audience/lib/python3.4/site-packages/tornado/concurrent.py", line 209, in result
    raise_exc_info(self._exc_info)
  File "<string>", line 3, in raise_exc_info
  File "/Users/scott/.virtualenvs/audience/lib/python3.4/site-packages/tornado/gen.py", line 212, in wrapper
    yielded = next(result)
  File "/Users/scott/.virtualenvs/audience/lib/python3.4/site-packages/tornado_mysql/connections.py", line 1188, in _get_descriptions
    for i in range_type(self.field_count):
TypeError: 'NoneType' object cannot be interpreted as an integer

What's the suitable way to know whether the sql is executed successfully or not?

I am not familiar with futures mechanism. so ask for help...
If the sql don't execute successfully, how can i get err-no and err-message from mysql?
thanks~

Now I do it by this way:

import logging
from tornado import ioloop, gen
from tornado_mysql import pools

dbpool = pools.Pool(...)

@gen.coroutine
def run_sql(sql):
    success = False
    try:
        cur = yield dbpool.execute(sql)
        success = True
    except:
        logging.error('mysql execution error')

    if success:
        # do sth.

DictCursor causing AttributeError: 'Connection' object has no attribute 'errorhandler'

Tried passing to pool.execute and also via connect_kwargs to the pool constructor.

HTTPServerRequest(protocol='http', host='127.0.0.1:8888', method='GET', uri='/huddle/news', version='HTTP/1.1', remote_ip='127.0.0.1', headers={'Accept-Language': 'en-US,en;q=0.8,pl;q=0.6', 'Accept-Encoding': 'gzip, deflate, sdch', 'Host': '127.0.0.1:8888', 'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8', 'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.84 Safari/537.36', 'Connection': 'keep-alive', 'Cookie': 'rg_cookie_session_id=2073182434', 'Cache-Control': 'max-age=0', 'Upgrade-Insecure-Requests': '1'})
Traceback (most recent call last):
  File "/Library/Python/2.7/site-packages/tornado/web.py", line 1445, in _execute
    result = yield result
  File "/Library/Python/2.7/site-packages/tornado/gen.py", line 1008, in run
    value = future.result()
  File "/Library/Python/2.7/site-packages/tornado/concurrent.py", line 232, in result
    raise_exc_info(self._exc_info)
  File "/Library/Python/2.7/site-packages/tornado/gen.py", line 1014, in run
    yielded = self.gen.throw(*exc_info)
  File "/Users/kkoston/dev/thehuddle/api/huddle/news.py", line 15, in get
    cur = yield db.HUDDLE.execute(sql)
  File "/Library/Python/2.7/site-packages/tornado/gen.py", line 1008, in run
    value = future.result()
  File "/Library/Python/2.7/site-packages/tornado/concurrent.py", line 232, in result
    raise_exc_info(self._exc_info)
  File "/Library/Python/2.7/site-packages/tornado/gen.py", line 1017, in run
    yielded = self.gen.send(value)
  File "/Library/Python/2.7/site-packages/Tornado_MySQL-0.5.1-py2.7.egg/tornado_mysql/pools.py", line 125, in execute
    cur = conn.cursor(cursor)
  File "/Library/Python/2.7/site-packages/Tornado_MySQL-0.5.1-py2.7.egg/tornado_mysql/connections.py", line 693, in cursor
    return self.cursorclass(self)
  File "/Library/Python/2.7/site-packages/MySQLdb/cursors.py", line 86, in __init__
    self.errorhandler = connection.errorhandler
AttributeError: 'Connection' object has no attribute 'errorhandler'
ERROR:tornado.access:500 GET /huddle/news (127.0.0.1) 205.46ms

How to use Pool? Please get me sample

my code :

#!/usr/bin/env python
from __future__ import print_function
from tornado import ioloop, gen
from tornado_mysql import pools
pools.DEBUG = True
POOL = pools.Pool(
    dict(host='127.0.0.1', port=3306, user='root', passwd='123456', db='lottery'),
    max_idle_connections=1,
    max_recycle_sec=3)

@gen.coroutine
def getAll():
    cur=yield POOL.execute("SELECT redball1 FROM lottery")
    print(cur.fetchall())
@gen.coroutine
def main():
    getAll()

ioloop.IOLoop.current().run_sync(main)
print(POOL._opened_conns)

But print is "0".why

Python3.6 Tornado5.0 runtests tornado_mysql.err.OperationalError: (2006, 'MySQL server has gone away (Stream is closed)')

python3.6
tornado 5.0

runtests

Traceback (most recent call last):
  File "E:\work\projects\interview\service\user\venv\lib\site-packages\tornado_mysql\connections.py", line 833, in _read_packet
    packet_header = yield self._stream.read_bytes(4)
  File "E:\work\projects\interview\service\user\venv\lib\site-packages\tornado\iostream.py", line 418, in read_bytes
    self._try_inline_read()
  File "E:\work\projects\interview\service\user\venv\lib\site-packages\tornado\iostream.py", line 857, in _try_inline_read
    self._check_closed()
  File "E:\work\projects\interview\service\user\venv\lib\site-packages\tornado\iostream.py", line 1058, in _check_closed
    raise StreamClosedError(real_error=self.error)
tornado.iostream.StreamClosedError: Stream is closed

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "E:\work\projects\interview\service\user\handlers\user_service\views.py", line 437, in post
    cur = yield self.db.execute(sql, [reward_id, self.cache_user_id])
  File "E:\work\projects\interview\service\user\venv\lib\site-packages\tornado\gen.py", line 1099, in run
    value = future.result()
  File "E:\work\projects\interview\service\user\venv\lib\site-packages\tornado\gen.py", line 1107, in run
    yielded = self.gen.throw(*exc_info)
  File "E:\work\projects\interview\service\user\venv\lib\site-packages\tornado_mysql\pools.py", line 124, in execute
    yield cur.execute(query, params)
  File "E:\work\projects\interview\service\user\venv\lib\site-packages\tornado\gen.py", line 1099, in run
    value = future.result()
  File "E:\work\projects\interview\service\user\venv\lib\site-packages\tornado\gen.py", line 1107, in run
    yielded = self.gen.throw(*exc_info)
  File "E:\work\projects\interview\service\user\venv\lib\site-packages\tornado_mysql\cursors.py", line 132, in execute
    yield self._query(query)
  File "E:\work\projects\interview\service\user\venv\lib\site-packages\tornado\gen.py", line 1099, in run
    value = future.result()
  File "E:\work\projects\interview\service\user\venv\lib\site-packages\tornado\gen.py", line 1107, in run
    yielded = self.gen.throw(*exc_info)
  File "E:\work\projects\interview\service\user\venv\lib\site-packages\tornado_mysql\cursors.py", line 283, in _query
    yield conn.query(q)
  File "E:\work\projects\interview\service\user\venv\lib\site-packages\tornado\gen.py", line 1099, in run
    value = future.result()
  File "E:\work\projects\interview\service\user\venv\lib\site-packages\tornado\gen.py", line 1107, in run
    yielded = self.gen.throw(*exc_info)
  File "E:\work\projects\interview\service\user\venv\lib\site-packages\tornado_mysql\connections.py", line 731, in query
    yield self._read_query_result(unbuffered=unbuffered)
  File "E:\work\projects\interview\service\user\venv\lib\site-packages\tornado\gen.py", line 1099, in run
    value = future.result()
  File "E:\work\projects\interview\service\user\venv\lib\site-packages\tornado\gen.py", line 1107, in run
    yielded = self.gen.throw(*exc_info)
  File "E:\work\projects\interview\service\user\venv\lib\site-packages\tornado_mysql\connections.py", line 865, in _read_query_result
    yield result.read()
  File "E:\work\projects\interview\service\user\venv\lib\site-packages\tornado\gen.py", line 1099, in run
    value = future.result()
  File "E:\work\projects\interview\service\user\venv\lib\site-packages\tornado\gen.py", line 1107, in run
    yielded = self.gen.throw(*exc_info)
  File "E:\work\projects\interview\service\user\venv\lib\site-packages\tornado_mysql\connections.py", line 1062, in read
    first_packet = yield self.connection._read_packet()
  File "E:\work\projects\interview\service\user\venv\lib\site-packages\tornado\gen.py", line 1099, in run
    value = future.result()
  File "E:\work\projects\interview\service\user\venv\lib\site-packages\tornado\gen.py", line 315, in wrapper
    yielded = next(result)
  File "E:\work\projects\interview\service\user\venv\lib\site-packages\tornado_mysql\connections.py", line 845, in _read_packet
    raise OperationalError(2006, "MySQL server has gone away (%s)" % (e,))
tornado_mysql.err.OperationalError: (2006, 'MySQL server has gone away (Stream is closed)')

my testcase

class BaseHandlerTestCase(WebTestCase):
    """
        构建测试本项目的测试案例
        1. 将所有的handler进行处理
        2. 初始化数据库
        3. 初始化redis环境
        4. 结束时,销毁数据库以及redis环境
    """

    def set_db_info(self):
        """
        测试案例,默认创建数据库的参数
        可根据自身需求,重写方法或者定制
        :return:
        """
        settings.MYSQL_DB_DBNAME = "testing"

    def set_redis_info(self):
        """
        测试案例,默认创建redis的参数
        可根据自身需求,重写方法或者定制
        :return:
        """
        settings.REDIS_DB = 8

    def get_app(self):
        """
        1. 读取系统的App
        2. load所有的Handlers
        :return:
        """
        app = MyApplication()
        return app

    def create_db(self):
        """
        创建数据库 database
        :return:
        """
        with pymysql.connect(**dict(host=settings.MYSQL_DB_HOST,
                                    port=settings.MYSQL_DB_PORT,
                                    user=settings.MYSQL_DB_USER,
                                    passwd=settings.MYSQL_DB_PASSWORD,
                                    charset="utf8mb4"
                                    )) as cursor:
            # 警告不提示
            cursor._defer_warnings = True
            sql = """
                  DROP DATABASE IF EXISTS `{}`;
            """.format(settings.MYSQL_DB_DBNAME)
            cursor.execute(sql)
            sql = """
              CREATE DATABASE IF NOT EXISTS {} DEFAULT CHARSET utf8mb4 COLLATE utf8mb4_general_ci;
            """.format(settings.MYSQL_DB_DBNAME)
            cursor.execute(sql)

    def create_tables(self):
        """
        创建数据库表
        :return:
        """
        with pymysql.connect(**dict(host=settings.MYSQL_DB_HOST,
                                    port=settings.MYSQL_DB_PORT,
                                    user=settings.MYSQL_DB_USER,
                                    passwd=settings.MYSQL_DB_PASSWORD,
                                    database=settings.MYSQL_DB_DBNAME,
                                    charset="utf8mb4"
                                    )) as cursor:
            cursor._defer_warnings = True
            file_path = os.path.join(settings.BASE_PATH, 'doc', 'user.sql')
            with open(file_path, 'r', encoding='utf-8') as f:
                sql = "".join(f.readlines()).replace("\n", "")
                sqls = sql.split('--')
            for sql_ in sqls:
                if sql_.strip():
                    cursor.execute(sql_)

    def clear_db(self):
        """
        删除数据库
        :return:
        """
        with pymysql.connect(**dict(host=settings.MYSQL_DB_HOST,
                                    port=settings.MYSQL_DB_PORT,
                                    user=settings.MYSQL_DB_USER,
                                    passwd=settings.MYSQL_DB_PASSWORD,
                                    charset="utf8mb4"
                                    )) as cursor:
            sql = "DROP DATABASE IF EXISTS `{}`;".format(settings.MYSQL_DB_DBNAME)
            cursor._defer_warnings = True
            cursor.execute(sql)


    def clear_redis(self):
        """
        清除redis数据
        :return:
        """
        rc = redis.Redis(
            host=settings.REDIS_HOST,
            port=settings.REDIS_PORT,
            db=settings.REDIS_DB,
            password=settings.REDIS_PASS)
        rc.flushdb()

    def setUp(self):
        self.set_db_info()
        self.set_redis_info()
        self.create_db()
        self.create_tables()
        super(BaseHandlerTestCase, self).setUp()

    def tearDown(self):
        super(BaseHandlerTestCase, self).tearDown()
        self.clear_redis()
        self.clear_db()

    @property
    def db(self):
        return pymysql.connect(**dict(host=settings.MYSQL_DB_HOST,
                                      port=settings.MYSQL_DB_PORT,
                                      user=settings.MYSQL_DB_USER,
                                      passwd=settings.MYSQL_DB_PASSWORD,
                                      database=settings.MYSQL_DB_DBNAME,
                                      charset="utf8mb4"
                                      ))

    @property
    def redis(self):
        return self._app.redis.get_client()

class RecieveRewardTest(BaseHandlerTestCase):
    def init_req_data(self):
        self.user_id = 12345
        access_token = "TEST"
        data = {
            "user_id": self.user_id
        }
        self.redis.set("a:{}".format(access_token), json_encode(data))
        headers = {
            "Content-Type": "application/json",
            "X-Access-Token": access_token
        }
        return access_token, headers

    def execute_sql(self, sql, *args):
        try:
            with self.db as cursor:
                rowcount = cursor.execute(sql, args)
            return rowcount, cursor.lastrowid
        except:
            traceback.print_exc()
        finally:
            self.db.close()

    @gen_test
    def query_sql(self):
        pass

    # 参数不正确
    def test_param_error(self):
        req_data = {
            "reward_id": ""
        }
        access_token, headers = self.init_req_data()
        resp = self.fetch("/user_service/receive_reward", method="POST", body=json_encode(req_data), headers=headers)
        res = json_decode(resp.body)
        self.assertEqual(resp.code, 200)
        self.assertEqual(res['code'], 400)
        self.assertEqual(res['message'], '参数不正确')

    # 正在领取
    def test_process_reward(self):
        req_data = {
            "reward_id": "123456"
        }
        access_token, headers = self.init_req_data()
        # 构造正在处理的数据
        key = "RecieveReward:%s" % req_data['reward_id']
        self.redis.set(key, 1)

        # 准备测试
        resp = self.fetch("/user_service/receive_reward", method="POST", body=json_encode(req_data), headers=headers)
        res = json_decode(resp.body)
        self.assertEqual(resp.code, 200)
        self.assertEqual(res['code'], 400)
        self.assertEqual(res['message'], '奖励在其他处已经领取,请重新查询')
        self.assertIsNotNone(self.redis.get(key))

    # 数据库中指定的领取奖励信息不存在
    def test_not_found(self):
        req_data = {
            "reward_id": "123456"
        }
        access_token, headers = self.init_req_data()
        # 构造正在处理的数据
        key = "RecieveReward:%s" % req_data['reward_id']
        # 准备测试
        resp = self.fetch("/user_service/receive_reward", method="POST", body=json_encode(req_data), headers=headers)
        res = json_decode(resp.body)
        self.assertEqual(resp.code, 200)
        self.assertEqual(res['code'], 400)
        self.assertEqual(res['message'], '奖励信息不存在')
        self.assertIsNone(self.redis.get(key))

    # 数据库中是否可领取状态
    def test_is_interview_status_error(self):
        access_token, headers = self.init_req_data()
        sql = """
            insert into client_user_share_award_info(
              user_id, position_stage_id, name, school, job, short_job, position, 
              money, interview_status, is_interview_status)
            value(%s, 1, 'test', 'test', '公司名字', '公司简称', 1, 12.5, 5, 0)
        """
        rowcount, lastrowid = self.execute_sql(sql, self.user_id)
        req_data = {
            "reward_id": lastrowid
        }
        # 构造正在处理的数据
        key = "RecieveReward:%s" % req_data['reward_id']
        # 准备测试
        resp = self.fetch("/user_service/receive_reward", method="POST", body=json_encode(req_data),
                          headers=headers)
        res = json_decode(resp.body)
        self.assertEqual(resp.code, 200)
        self.assertEqual(res['code'], 400)
        self.assertEqual(res['message'], '奖励非可领取状态,请刷新重试')
        self.assertIsNone(self.redis.get(key))

    # 数据库中是否可领取状态
    def test_interview_status_error(self):
        access_token, headers = self.init_req_data()
        sql = """
            insert into client_user_share_award_info(
              user_id, position_stage_id, name, school, job, short_job, position,
              money, interview_status, is_interview_status)
            value(%s, 1, 'test', 'test', '公司名字', '公司简称', 1, 12.5, 4, 3)
        """
        rowcount, lastrowid = self.execute_sql(sql, self.user_id)
        req_data = {
            "reward_id": lastrowid
        }
        # 构造正在处理的数据
        key = "RecieveReward:%s" % req_data['reward_id']
        # 准备测试
        resp = self.fetch("/user_service/receive_reward", method="POST", body=json_encode(req_data),
                          headers=headers)
        res = json_decode(resp.body)
        self.assertEqual(resp.code, 200)
        self.assertEqual(res['code'], 400)
        self.assertEqual(res['message'], '奖励非可领取状态,请刷新重试')
        self.assertIsNone(self.redis.get(key))

A single test case can be executed alone, but the execution of a group is wrong.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.