I'm use the gunicorn_thrift like last issues,some time is run well ,but when it run a while ,I got a error

1741648 [2017-12-06 12:15:39 +0000] [5727] [CRITICAL] WORKER TIMEOUT (pid:7438)
1741649 [2017-12-06 12:15:39 +0000] [5727] [CRITICAL] WORKER TIMEOUT (pid:7445)
1741650 [2017-12-06 12:15:39 +0000] [5727] [CRITICAL] WORKER TIMEOUT (pid:7447)
1741651 [2017-12-06 12:15:39 +0000] [5727] [CRITICAL] WORKER TIMEOUT (pid:7448)
1741652 [2017-12-06 12:15:39 +0000] [5727] [CRITICAL] WORKER TIMEOUT (pid:7449)
1741653 [2017-12-06 12:15:39 +0000] [5727] [CRITICAL] WORKER TIMEOUT (pid:7451)
1741654 [2017-12-06 12:15:39 +0000] [5727] [CRITICAL] WORKER TIMEOUT (pid:7453)
1741655 [2017-12-06 12:15:39 +0000] [5727] [CRITICAL] WORKER TIMEOUT (pid:7456)
1741656 [2017-12-06 12:15:39 +0000] [5727] [CRITICAL] WORKER TIMEOUT (pid:7457)
1741657 [2017-12-06 12:15:39 +0000] [5727] [CRITICAL] WORKER TIMEOUT (pid:7458)
1741658 [2017-12-06 12:15:39 +0000] [5727] [CRITICAL] WORKER TIMEOUT (pid:7459)
1741659 [2017-12-06 12:15:40 +0000] [5727] [CRITICAL] WORKER TIMEOUT (pid:7434)
1741660 [2017-12-06 12:15:40 +0000] [5727] [CRITICAL] WORKER TIMEOUT (pid:7437)
1741661 [2017-12-06 12:15:40 +0000] [5727] [CRITICAL] WORKER TIMEOUT (pid:7443)
1741662 [2017-12-06 12:15:40 +0000] [7506] [INFO] Booting worker with pid: 7506
1741663 [2017-12-06 12:15:40 +0000] [7507] [INFO] Booting worker with pid: 7507
1741664 [2017-12-06 12:15:40 +0000] [7508] [INFO] Booting worker with pid: 7508
1741665 [2017-12-06 12:15:40 +0000] [7509] [INFO] Booting worker with pid: 7509
1741666 [2017-12-06 12:15:40 +0000] [7510] [INFO] Booting worker with pid: 7510
1741667 [2017-12-06 12:15:40 +0000] [7511] [INFO] Booting worker with pid: 7511
1741668 [2017-12-06 12:15:40 +0000] [7512] [INFO] Booting worker with pid: 7512
1741669 [2017-12-06 12:15:40 +0000] [7513] [INFO] Booting worker with pid: 7513
1741670 [2017-12-06 12:15:40 +0000] [7514] [INFO] Booting worker with pid: 7514
1741671 [2017-12-06 12:15:40 +0000] [7515] [INFO] Booting worker with pid: 7515
1741672 [2017-12-06 12:15:40 +0000] [7514] [ERROR] Exception in worker process:
1741673 Traceback (most recent call last):
1741674   File "/usr/local/lib/python2.7/site-packages/gunicorn/arbiter.py", line 507, in spawn_worker
1741675     worker.init_process()
1741676   File "/usr/local/lib/python2.7/site-packages/gunicorn_thrift/thriftpy_gevent_worker.py", line 98, in init_process
1741677     return super(GeventThriftPyWorker, self).init_process()
1741678   File "/usr/local/lib/python2.7/site-packages/gunicorn/workers/ggevent.py", line 192, in init_process
1741679     super(GeventWorker, self).init_process()
1741680   File "/usr/local/lib/python2.7/site-packages/gunicorn/workers/base.py", line 118, in init_process
1741681     self.wsgi = self.app.wsgi()
1741682   File "/usr/local/lib/python2.7/site-packages/gunicorn/app/base.py", line 67, in wsgi
1741683     self.callable = self.load()
1741684   File "/usr/local/lib/python2.7/site-packages/gunicorn_thrift/thriftapp.py", line 39, in load
1741685     self.thrift_app = self.load_thrift_app()
1741686   File "/usr/local/lib/python2.7/site-packages/gunicorn_thrift/thriftapp.py", line 33, in load_thrift_app
1741687     return utils.load_obj(self.app_uri)
1741688   File "/usr/local/lib/python2.7/site-packages/gunicorn_thrift/utils.py", line 16, in load_obj
1741689     mod = importlib.import_module(module)
1741690   File "/usr/local/lib/python2.7/importlib/__init__.py", line 37, in import_module
1741691     __import__(name)
1741692   File "/usr/local/lib/python2.7/site-packages/gevent-1.1rc2-py2.7-linux-x86_64.egg/gevent/builtins.py", line 58, in __import__
1741693     result = _import(*args, **kwargs)
1741694   File "/home/zhouyi/heybox_script/release/pubg_thriftpy/pubg_account_server.py", line 13, in <module>
1741695     pg_account = psycopg2.connect(database="account", user='postgres', password="", host="10.10.10.54", port='19934')
1741696   File "/usr/local/lib/python2.7/site-packages/psycopg2/__init__.py", line 164, in connect
1741697     conn = _connect(dsn, connection_factory=connection_factory, async=async)
1741698 OperationalError: FATAL:  sorry, too many clients already

it seems like the WORKER got timeout but not exit so the master booting another worker , finally the worker connect to pg got the ' too many clients' error, anyone know what's the problem ?

Thrift server 本身不就有多进程模型吧 ，你让gunicorn 兼容thrift的协议是为了使用gunicorn的进程管理模型么？ 好奇...

写了一个thrift文件
service PingPong {
string ping()
}
service Hello{
string hello2015()
}

然后使用thriftpy

import thriftpy
from thriftpy.thrift import TProcessor
from thriftpy.thrift import TMultiplexedProcessor

class PingPongDispatcher(object):
def ping(self):
return "pong"

class HelloDispatcher(object):
def hello2015(self):
return "hello2015"

pingpong_thrift = thriftpy.load("pingpong.thrift")
app1 = TProcessor(pingpong_thrift.PingPong, PingPongDispatcher())
app2 = TProcessor(pingpong_thrift.Hello, HelloDispatcher())
processor = TMultiplexedProcessor()
processor.register_processor("pingpong", app1)
processor.register_processor("hello", app2)

然后使用gunicorn_thrift启动
gunicorn_thrift app:processor -k thrift_gevent --thrift-protocol-factory thriftpy.protocol:TBinaryProtocolFactory --thrift-transport-factory thriftpy.transport:TBufferedTransportFactory --error-log -

客户端使用的是java、访问python服务器端时出现错误

[2015-08-29 14:09:49 +0800] [719] [ERROR] 'TCyBufferedTransportFactory' object has no attribute 'getTransport'
Traceback (most recent call last):
File "/Library/Python/2.7/site-packages/gunicorn_thrift/gevent_worker.py", line 42, in handle
self.get_thrift_transports_and_protos(result)
File "/Library/Python/2.7/site-packages/gunicorn_thrift/gevent_worker.py", line 25, in get_thrift_transports_and_protos
itrans = self.app.tfactory.getTransport(result)
AttributeError: 'TCyBufferedTransportFactory' object has no attribute 'getTransport'
Traceback (most recent call last):
File "/Library/Python/2.7/site-packages/gevent/greenlet.py", line 327, in run
result = self._run(_self.args, *_self.kwargs)
File "/Library/Python/2.7/site-packages/gunicorn_thrift/gevent_worker.py", line 59, in handle
itrans.close()
UnboundLocalError: local variable 'itrans' referenced before assignment
<Greenlet at 0x104e74870: <functools.partial object at 0x104ea4ec0>(<socket at 0x104f24750 fileno=11 sock=127.0.0.1:80, ('127.0.0.1', 49831))> failed with UnboundLocalError

请问这是怎么回事？本人新手学习thriftpy和使用gunicorn_thrift、跪请大神回复

I'm going to take a crack at this, so here's an issue for me to reference later:

TODO:

• Implement a worker for gthread
• Make some tests
• Throw some crap at it

项目使用 gunicorn_thrift + thriftpy2，接口正常请求时间400ms左右，但是持续请求测试时，客户端打印请求时间发现，偶尔会出现十几秒甚至几十秒的请求时间（我timeout设置比较大）。接口本身逻辑简单且服务端（业务代码）打印请求时间正常。线上请求量不算大，QPS最高也就300r/s
不知道你们有没有遇到过类似问题......

您好！ 请问关于使用thriftpy2 + gunicorn_thrift + gevent worker-class 使用场景来再生产环境里，增强thriftpy server吞吐性能使用场景，我遇到如下几个问题，特来请教大神：
(1) 对于使用thrfitpy2框架，并业务handler里，处理一条请求里有较多rpc访问，我理解上述 gunicorn 多worker进程 + 每个worker使用gevent类型，应该是性能最佳的组合？ 当然我没有实践，跟@ethe 确认
(2) 我看worker-class = gevent 只适用于protocol=TBinaryProtocol 和 transport=TBufferedTransport， transport不支持TFramedTransport么？
(3) 我原来使用thriftpy2 make_server接口，采用的TthreadServer, 多线程模式并发处理请求，业务处理handler里，需要在处理并发请求时，加锁，做一些同步处理，这种场景，改为gunicorn_thrift fork多worker进程模式，应当如何保障在多个worker进程处理请求时，共享一个全局的进程锁，多worker同步了？

@ethe 能否在百忙之中，及时帮我们答疑解惑了？ 万分感谢！ 以及是否方便微信联系了？ 我们在较大规模应用thriftpy2 以及相关组件

Gunicorn added support in 19.4 and gevent did in 1.1.

I tested gunicorn_thrift and the gevent worker with those versions, and it seems to work fine.

I noticed installing from pypi and using a thriftpy_gevent worker is broken in the latest version 0.2.21.

Installing directly from this repo seems to solve the problem:
git+https://github.com/eleme/gunicorn_thrift.git

After further inspection the code looks pretty different on pypi
pypi 0.2.21
thriftpy_gevent_worker.py line 100

this repo:

Any chance of updating pypi distribution?

Thanks!

Hey Eleme.

For a few days I've been trying to use New Relic monitoring on my gunicorn_thrift app. I've had no issues with my traditional gunicorn apps. I can launch my gunicorn_thrift application with no issues, but new relic gets no data.

I can launch it with Gunicorn like so:

NEW_RELIC_CONFIG_FILE=newrelic.ini newrelic-admin run-program [gunicorn commands here]

But if I use:

NEW_RELIC_CONFIG_FILE=newrelic.ini newrelic-admin run-program gunicorn_thrift tests.thriftpy_app:app -k thriftpy_sync \
  --thrift-protocol-factory \
    thriftpy.protocol:TCyBinaryProtocolFactory \
  --thrift-transport-factory \
    thriftpy.transport:TCyBufferedTransportFactory

I get no errors, but no communication to new relic.

https://docs.newrelic.com/docs/agents/python-agent/installation-configuration/python-agent-admin-script

Any advice for me?

I would recommend changing the requirement gunicorn version to 19.9.0. Because older version has issue: benoitc/gunicorn#1822
Later version has six import problem.

In my situation, I just set client timeout as low as possible described in #45, but i always got a timeout message like Thriftpy/thrift_connector#48 said.

At last, I found that in https://github.com/eleme/gunicorn_thrift/blob/master/gunicorn_thrift/thriftpy_sync_worker.py#L75, it always log exception with full error stack.

But I think we should first identify to exception like below:

if isinstance(e, socket.timeout):
    self.log.info("timeout")
else:
    self.log.exception(e)

from thriftpy2.rpc import make_server
make_server(): this way start app, it can accept several client connection. but when i change to use gunicorn_thrift ,one worker just can accept one client connection.

--worker-connections INT
                        The maximum number of simultaneous clients. [1000]

i have add the parameter “--worker-connections” explicitly.

cmd:

rpc.rpc_server:app -k thriftpy_sync -b 0.0.0.0:6000 -w 1 --threads 1 --worker-connections 2 -t 5 --thrift-protocol-factory thriftpy2.protocol:TCyBinaryProtocolFactory --thrift-transport-factory thriftpy2.transport:TCyBufferedTransportFactory --log-level debug

Gevent 1.2 supports 3.4+ according to their documentation (3.3+ less officially), so theoretically this library shouldn't have any problems supporting thriftpy_gevent in 3.4+ (3.3+). I propose, then, that it should (and if it does, the README should reflect that)

when I use gunicorn_thrift -c gunicorn.py pythriftServer:app to launch my thrifypy2 app, import error occured like follow

(monitor) ➜  pythriftCoco gunicorn_thrift -c gunicorn.py pythriftServer:app
Traceback (most recent call last):
  File "/home/cicada/.pyenv/versions/monitor/bin/gunicorn_thrift", line 7, in <module>
    from gunicorn_thrift.thriftapp import run
  File "/home/cicada/.pyenv/versions/3.6.5/envs/monitor/lib/python3.6/site-packages/gunicorn_thrift/thriftapp.py", line 11, in <module>
    from . import config
  File "/home/cicada/.pyenv/versions/3.6.5/envs/monitor/lib/python3.6/site-packages/gunicorn_thrift/config.py", line 4, in <module>
    from gunicorn import six
ImportError: cannot import name 'six'

maybe gunicorn_thrift version is not matched with gunicorn

Hi. Is there some way to do more grained configuration in gunicorn_thrift just like gunicorn? E.g. worker number or thread number.

i use this to create a service that provide RPC interface.
but when it was in pressure test,
cpu usage suddenly falled for a moment(about 10 seconds),then lots of requests were time out.
then all of these were back to normal.but after several minutes.it repeated again.
i use gevent model and 6 workers.
can you help me ?

使用了reuse_port=True 起了8个worker进程
但是 netstat -tunlp的时候 发现只有一个进程在监听端口
是不是SO_REUSEPORT没有效果，还是哪有什么问题？

The version of setproctitle being used is broken on alpine linux. It was fixed dvarrazzo/py-setproctitle#44 and the new 1.1.10 version contains this fix. Can we bump the version?

thriftpy2, 在client退出后，服务端检查还有这个连接的socket在ESTABLISHED状态。这个有什么方法可以主动关闭么？
目前使用了thrift_client_timeout， 但是依然不理想

I'v installed gunicorn_thrift， but can't find the command gunicorn_thrift？

eleme/gunicorn looks live have been removed. Could not open https://github.com/eleme/gunicorn/commit/0ac40b460149f757cbbc04288f4764c3e261a645 any more.

cc @wooparadog

Throw import error when using high version gunicorn as gunicorn has already remove six from it's package, in gunicorn>=20.0。Remove six dependency。 hoping gunicorn_thrift can catch up with the new version, Thank you

Here is the situation I ran into:

In my project requirements.txt, I use gunicorn==19.1.1 and gunicorn_thrift master branch via git.

Then I run pip install -r requirements.txt, pip freeze, and gunicorn is 19.1.1. After that I try to run gunicorn_thrift, but error occur:

Traceback (most recent call last):
  File "/virtualenv/bin/gunicorn_thrift", line 5, in <module>
    from pkg_resources import load_entry_point
  File "/virtualenv/lib/python2.7/site-packages/pkg_resources.py", line 3011, in <module>
    parse_requirements(__requires__), Environment()
  File "/virtualenv/lib/python2.7/site-packages/pkg_resources.py", line 626, in resolve
    raise DistributionNotFound(req)
pkg_resources.DistributionNotFound: gunicorn==19.1.0

Maybe a better way to write requirements.txt should be using gunicorn>=19.1.0, instead of gunicorn==19.1.0?
Also by doing this, if we got some issues caused by upstream changes, we can fix it in time.

@wooparadog

Hello,

I was trying to bringup gunicorn_thrift with http instead of roc to make it work under heroku load balancers.

I am facing issues when pushed gunicorn_thrift with rpc.
How can I change that to http?

Right now, I am bringing up app with

app = TProcessor(thrift_service,thrift_service_handler)

Hi，
I did a test and found a problem that bothered me.
I start gunicorn_thrift server and set --worker with the number of my cpu cores, it is 8.
Then I did the stress testing to the multiple workers server.
I found the total cpu usage of these 8 worker process is less than 100%, and the stress testing is broken.
I doubt if multiple worker server can use multiple cpu cores. Is it really a multi process with python server? Whether or not related to GIL with python.
Or, it is totally my program problem...

The following is some data of my testing:
    THREAD_NUM = 100    // thread number per round
    ONE_WORKER_NUM = 1000    // loop number of each thread
    LOOP_SLEEP = 0.1    // time interval of each loop

    PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                                                                                                                                  
    20   0  221136  30468   7380 S  23.9  0.2   0:40.24 gunicorn: worker [PythonServer:processor]
    20   0  221196  30456   7380 S  20.9  0.2   0:33.67 gunicorn: worker [PythonServer:processor]
    20   0 2626964  18972   3564 S  19.6  0.1   0:03.06 python stress_test.py                                                                                                                  
    20   0  221116  30444   7380 S  17.0  0.2   0:29.29 gunicorn: worker [PythonServer:processor]
    20   0  221104  30440   7380 S  14.0  0.2   0:24.07 gunicorn: worker [PythonServer:processor]                                                                                       
    20   0  221100  30428   7380 S  11.3  0.2   0:18.64 gunicorn: worker [PythonServer:processor]
    20   0  221476  30416   7380 S   7.6  0.2   0:14.29 gunicorn: worker [PythonServer:processor]
    20   0  221332  30408   7380 S   6.6  0.2   0:11.93 gunicorn: worker [PythonServer:processor]
    20   0  221008  30144   7308 S   5.7  0.2   0:10.69 gunicorn: worker [PythonServer:processor]

and how to config self-define proto_factory or modiefied TBinaryProtocolFactory ?

I was run thriftpy server as the readme suggest, and i used gunicorn_thrift as follow config

gunicorn_thrift account_server:app -w 60 \
  --worker-connections 2000 -k thriftpy_sync --thrift-protocol-factory thriftpy.protocol:TCyBinaryProtocolFactory \
    --thrift-transport-factory thriftpy.transport:TCyBufferedTransportFactory --threads 5  -D \
    --error-logfile /homex2/zhouyi/log/account_thrift_error.log --access-logfile /homex2/zhouyi/log/account_thrift_access.log\
      -b  0.0.0.0:16796 --reload  --forwarded-allow-ips 10.10.10.55

the test code was

import time
from multiprocessing import Process
import thriftpy
account_thrift = thriftpy.load("account.thrift", module_name="account_thrift")
from thriftpy.rpc import make_client

def main():
    thread_list = []
    start_time = time.time()
    for i in xrange(0,800):
        th = Process(target=process_func)
        th.start()
        thread_list.append(th)
    for i in thread_list:
        i.join()
    print time.time() - start_time

def process_func():
    th = MyThread()
    th.process_func()


class MyThread:
    def __init__(self):
        self.client = make_client(account_thrift.AccountRpc, '127.0.0.1', 16796)

    def process_func(self):
        for i in xrange(1,10000):
            print self.client.rpc_account_sid('account.5e4c769046c54b37a6f441fed028186d')
            time.sleep(0.5)


if __name__ == '__main__':
    r = main()

when i run the test code i got error with block
and i can't connect to the rpc server with other client too i got error
thriftpy.transport.TTransportException: TTransportException(message="Could not connect to ('127.0.0.1', 16796)", type=1)
, Is there any config error and what't the problem?

I just run command like

gunicorn_thrift tests.thriftpy_app:app -k thriftpy_sync \
  --thrift-protocol-factory \
    thriftpy.protocol:TCyBinaryProtocolFactory \
  --thrift-transport-factory \
    thriftpy.transport:TCyBufferedTransportFactory

,when I run on my computer, works well.
But on a server, some request always be hang on a long time.
i can't figure out which part got problem, thriftpy ? thrift_connector? gunicorn_thrift or even sqlalchemy.
finally, I found that one Process or called one worker only work for one connection. so i run into gunicorn documents find -w out.

so could you please add this parameter to Readme.md. so, someone else will not run in this trouble.

btw, could you please just share how to choose best client timeout and service timeout?

thanks.

When I try to install dependencies for an application that uses gunicorn_thrift, I get errors from pip complaining about not finding a version that satisfies my given requirement.

Looking on PyPI, the only available distribution for gunicorn_thrift 0.2.1 is a Python 2 wheel.

Can you guys please upload an sdist and a Python 3 wheel?

Thanks!