Git Product home page Git Product logo

msgpack-rpc-python's Introduction

MessagePack RPC for Python

MessagePack RPC implementation based on Tornado.

Installation

% pip install msgpack-rpc-python

or

% python setup.py install

Module dependency

  • msgpack-python (>= 0.3)
  • tornado (>= 3)

Example

Server

import msgpackrpc

class SumServer(object):
    def sum(self, x, y):
        return x + y

server = msgpackrpc.Server(SumServer())
server.listen(msgpackrpc.Address("localhost", 18800))
server.start()

Client

import msgpackrpc

client = msgpackrpc.Client(msgpackrpc.Address("localhost", 18800))
result = client.call('sum', 1, 2)  # = > 3

Run test

In test directory:

% PYTHONPATH=../ python test/test_msgpackrpc.py

Run with timeout test(Timeout test takes about 5 seconds)

% PYTHONPATH=../ python test/test_msgpackrpc.py --timeout-test

Performance

OS: Mac OS X ver 10.8.3
CPU: Intel Core i7 2.7 GHz
Memory: 16 GB 1600 MHz DDR3

call(QPS)async(QPS)notify(QPS)
2.7.35903604024877
3.3.05493581223634
PyPy 1.9.0 with GCC 4.2.15519972946406

Test code are available in example directory(bench_client.py and bench_server.py).

TODO

  • Add advanced return to Server.
  • UDP, UNIX Domain support
  • Utilities (MultiFuture, SessionPool)

Copyright

AuthorMasahiro Nakagawa
CopyrightCopyright (c) 2011- Masahiro Nakagawa
LicenseApache License, Version 2.0

msgpack-rpc-python's People

Contributors

blep avatar conovaloffjustforfun avatar hufman avatar ianlewis avatar kazupon avatar ksato9700 avatar kuenishi avatar methane avatar repeatedly avatar youtalk avatar z888z avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

msgpack-rpc-python's Issues

about strategy of socket closed

HI,
when ClientSocket was closed, I think the Session._request_table were no longer necessary.
future under current strategy will be wait until timeout, but if Client set timeout None, when socket closed, there will be endless wait.

    def on_close(self, sock):
        # Avoid calling self.on_connect_failed after self.close called.
        if self._closed:
            return

        # if sock in self._sockets:
        #     self._sockets.remove(sock)
        #     self._connecting = 0
        # else:
        # Tornado does not have on_connect_failed event.
        self.on_connect_failed(sock)

What do you think?

Need maintainers

I can't maintain this project now because I don't have enough time.
Does anyone interested in working on this project?

msgpackrpc.error.TimeoutError: Request timed out

Hi. How can I extend the default timeout? In my app the rpc call invokes some process that performs a job that takes several seconds.How can I modify the timeout time so that this errors won't throw?

Thanks.

avoid importing tornado if only using the client

maybe i'm missing something, but when using the client, tornado shouldn't be needed right?
right now msgpackrpc/init.py imports Loop which imports the tornado loop, which is a little annoying.

Tornado 5.0 not supported (unexpected keyword argument 'io_loop')

Tornado 3 and 4 works ok. Tornado 5 server.py fails with:

Traceback (most recent call last):
  File "server.py", line 8, in <module>
	server.listen(msgpackrpc.Address("localhost", 18800))
  File "/usr/local/lib/python2.7/dist-packages/msgpackrpc/server.py", line 24, in listen
	listener.listen(self)
  File "/usr/local/lib/python2.7/dist-packages/msgpackrpc/transport/tcp.py", line 170, in listen
	self._mp_server = MessagePackServer(self, io_loop=self._server._loop._ioloop, encodings=self._encodings)
  File "/usr/local/lib/python2.7/dist-packages/msgpackrpc/transport/tcp.py", line 157, in __init__
	tcpserver.TCPServer.__init__(self, io_loop=io_loop)
TypeError: __init__() got an unexpected keyword argument 'io_loop'

Unfortunately pip defaults to version 5. Suggest we get pip to install 4.5.3 until version 5 is supported.

How to reinitialize msgpckrpc (python)? (to create a new client)

Hello,

I am having trouble with Microsoft AirSim, which uses msgpckrpc as its communication backend. AirSim does this in a python object:

self.client = msgpackrpc.Client(msgpackrpc.Address(ip, port), timeout = timeout_value, pack_encoding = 'utf-8', unpack_encoding = 'utf-8')

Let's say at some point AirSim freezes and I get a TimeoutError from msgpckrpc (which I do, see microsoft/AirSim-NeurIPS2019-Drone-Racing#104 ). So I want to instantiate a new version of this python object (with another msgpackrpc.Address(ip, port) ).

However, doing this apparently simple thing is actually a nightmare and fails because msgpackrpc/tornado don't get properly reinitialized in the process. How can I properly reinitialize msgpckrpc please ?

I get an exception under heavy load

When the server is heavily in use I occasionally get this error:

'NoneType' object has no attribute 'send_message'

Any ideas?

full stack trace:

  File "requests_service/client.py", line 30, in _client_call
    return client.call(method, args, kwargs)
  File "msgpackrpc/session.py", line 41, in call
    return self.send_request(method, args).get()
  File "msgpackrpc/session.py", line 51, in send_request
    self._transport.send_message([message.REQUEST, msgid, method, args])
AttributeError: 'NoneType' object has no attribute 'send_message'

AsyncResult is located in msgpack.server

Currently a non-blocking server would look like this:

import multiprocessing, time, threading
import msgpackrpc

# Why is this core thing buried deep in a random module?
from msgpackrpc.server import AsyncResult


def run_threaded(result, x, y):
    time.sleep(1)
    result.set_result(x + y)

class SumServer(object):
    def sum(self, x, y):
        result = AsyncResult()
        thread = threading.Thread(target=run_threaded, args=(result, x, y))
        thread.start()
        return result

def server():
    server = msgpackrpc.Server(SumServer())
    server.listen(msgpackrpc.Address("localhost", 4000))
    server.start()

def client():
    client = msgpackrpc.Client(msgpackrpc.Address("localhost", 4000))
    future = client.call_async('sum', 1, 2)
    print(future.get())

def main():
    proc_server = multiprocessing.Process(target=server)
    proc_server.start()

    time.sleep(1)

    try:
        client()
    finally:
        proc_server.terminate()

if __name__ == '__main__':
    main()

In my opinion, this is a design problem: AsyncResult, which is essential to writing asynchronous servers, is contained in a non-top-level module. Is it possible to move it to the top level, for example, by importing from __init__.py?

Use a more recent version of Tornado?

Are there any plans to move towards a more recent version of Tornado?
Airsim uses msgpack-rpc-python (related issue), which is not compatible with Tornado >= v5 (deliberate decision here).
Unfortunately, the jupyter extension of VS Code needs Tornado >= v5. Are there any plans to update tornado to a more recent version? (Or are there not enough maintainers?)
Thx!

Example of using notify()

I couldn't get client.notify() to work. It looks like the call is going nowhere. The following code prints nothing:

import multiprocessing, time
import msgpackrpc

class SumServer(object):
    def notification(self, s):
        print(s)

def server():
    server = msgpackrpc.Server(SumServer())
    server.listen(msgpackrpc.Address("localhost", 4000))
    server.start()

def client():
    client = msgpackrpc.Client(msgpackrpc.Address("localhost", 4000))
    client.notify('notification', 'boo')

def main():
    proc_server = multiprocessing.Process(target=server)
    proc_server.start()

    time.sleep(0.5)

    try:
        client()
    finally:
        proc_server.terminate()

if __name__ == '__main__':
    main()

Are there any examples of using .notify()?

Automatically created Loops should be closed automatically

Constructor of msgpackrpc.Session class has an optional argument loop=None. When loop is not specified, new Loop will be automatically created in the constructor.

However, as automatically-created Loops are not closed automatically, it uses more file descriptors than expected. This becomes an issue for clients that connects/disconnects frequently.
I think automatically-created Loops should be closed (i.e., session._loop._ioloop.close()) when its associated Session is closed.

I have two ideas:

  1. If Loop instance is not specified by user (i.e., automatically created in Session constructor), close the Loop when session.close() method is called.
  2. Add close_loop=True to the Session constructor argument. Close the Loop only if close_loop is set to True when session.close() method is called. This behavior is similar to closefd option of io.FileIO.

In addition, it is better to add close() method to msgpackrpc.Loop class, to allow users to close the internal tornado IOLoop.

Separate timeout for connect() and call()

My use-case:
If I connect to server, I want to know that there is something wrong immediately.
However when I call my client, I want to wait longer, as my calculation in RPC server logic takes a while.
The best would be to be able to provide timeout kwarg to call() method, so that a developer can specify separate timeouts for each call.
Question is, how timeout parameter in call method should be set as a default. 2 ideas:

  1. whatever, like 10 sec
  2. None. In that case timeout which was used for creating Client is used for call as it is done right now - backward compatibility provided.

I may try to develop something and PR, but let me know first what do you think plz.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.