pyrlang / pyrlang Goto Github PK

View Code? Open in Web Editor NEW

586.0 41.0 55.0 1012 KB

Erlang node implemented in Python 3.5+ (Asyncio-based)

Home Page: https://pyrlang.github.io/Pyrlang/

License: Apache License 2.0

Makefile 2.60% Python 91.09% Erlang 5.46% Batchfile 0.56% CSS 0.14% HTML 0.16%

python erlang distributed erlang-node library gevent rpc message-passing

pyrlang's Introduction

Pyrlang - Erlang node in Python

This is a drop-in Erlang node implementation in Python 3, implementing a network Erlang node protocol. It was designed to allow interoperation between existing Python projects and BEAM languages: Erlang, Elixir, Gleam, Luaerl, LFE, Clojerl and such.

With just a few lines of startup code your Python program becomes an Erlang network node, participating in the Erlang cluster.

Features

Erlang distribution protocol for Erlang versions 19 to 25
Registry of Python 'processes', which have an Erlang-compatible process identifier and an optional registered name
Send and receive messages locally and remotely by pid or name
Can link and monitor Erlang from Python and Python from Erlang
net_adm pings supported
RPC calls to Python (Erlang rpc:call). Exceptions are propagated from Python back to Erlang;
pyrlang.gen.server.GenServer descendant from pyrlang.process.Process allows accepting generic calls mapped to Python class members

Documentation

Browse at https://pyrlang.github.io/Pyrlang/

Or build your own by running make docs (generated by Sphinx).

Installing

From PyPI

Local from Sources

Clone Pyrlang and Term repositories
Install Term from source: Go to Term directory and pip install -e .
Install Pyrlang from source: Go to Pyrlang directory and pip install -e .

Support & Questions

For those times when you absolutely need assistance and email is too slow, here's a Discord channel: https://discord.gg/pWWe7Wx and there is a Slack channel #pyrlang on https://erlanger.slack.com/

pyrlang's People

Contributors

Stargazers

Watchers

pyrlang's Issues

Support for other Async options in Python

There are https://github.com/python-trio/trio and https://github.com/dabeaz/curio which simplifies Async, and there is https://github.com/agronholm/anyio for compatibility.

Create on demand supervised python nodes from erlang/elixir

Is it possible to start a python node on demand (from erlang/elixir code) and add it to a supervision tree?

I tried to find something about it in the docs without success.

Thanks

Batch remote evaluation on Python side

As an performance boost by reducing the amount of calls back and forth between the erlang and python node, we should implement a lazy evaluation chain for #29.

This can either be a function within the same erlang module py (or whatever name we use) or a new one py_lazy. The call should not require a context while using. example:

lazy_test() ->
  B0 = py:batch_new(),
  {B1, Now1} = py:batch_call(B0, ["datetime", <<"datetime.now">>], []),
  {B2, Now2} = py:batch_call(B1, [DT, datetime, <<"now">>], [], #{}),
  {B3, Res} =  py:batch_call(B2, [Now2, '__sub__'], [Now1], #{}),
  C1 = py:new_context('[email protected]'),
  C2 = py:new_context('[email protected]'),
  C3 = py:new_context('[email protected]'),
  TS = <<"total_seconds">>,
  [py:batch_run(C, B3, []) || C <- [C1, C2, C3]].

Make pluggable asyng engine abstractions, one for gevent and one for asyncio

(the work is done in pluggable-async-engines branch)

Create abstract Gevent adapter
Abstract away socket client for EPMD
Abstract away socket server for Distribution protocol
Abstract away socket client for Distribution protocol
Create abstract asyncio adapter

Native (Rust) high performance term encoding/decoding

Currently have moved my ETF implementation to a separate Rust library

Write a decoder
Write decode tests
Write an encoder
Write encode tests
Encode hook functionality, inspiration: https://github.com/discordapp/erlpack#how-to-use-an-encode-hook
Pack it as a Python library
Release nicely

Ability to use configured port for server2server connections

by Gordon Guthrie in Slack:

Hello
Trying to set up a cluster using Pyrlang in docker
So I need to have predictable port numbers - which I would normally do
by starting Erlang with the inet_dist_listen_min and inet_dist_listen_max
set to the same.
How do I do this with Pyrlang

In master the listener which needs to be configured, is located here:

Pyrlang/pyrlang/dist/distribution.py

Lines 53 to 57 in b39ac4b

 (self.in_srv_, self.in_port_) = self.engine_.listen_with( 

 protocol_class=InDistProtocol, 

 protocol_args=[], 

 protocol_kwargs=proto_kwargs 

 )

Cannot communicate with Pyrlang process: handle_inbox function does not run

I have followed the documentation to create a node and a Pyrlang process that inherits from the Process class:

import gevent
from gevent import monkey
monkey.patch_all()
import Pyrlang
from Pyrlang import Atom
from Pyrlang import term
from Pyrlang import Process

class MyProcess(Process):
    def __init__(self, node) -> None:
        Process.__init__(self, node)
        node.register_name(self, term.Atom('my_process'))  # optional
        self.play()

    def handle_inbox(self):
        while True:
            # Do a selective receive but the filter says always True
            print("-------------> inside handle_inbox <---------------")
            msg = self.inbox_.receive(filter_fn = lambda _: True)
            if msg is None:
                break
            print("Incoming", msg)


def main():
    node = Pyrlang.Node("[email protected]", "COOKIE")
    node.start()
    pid = node.register_new_process(None)
    node.send(pid, (Atom('[email protected]'), Atom('shell')), Atom('hello'))
    while True:
        gevent.sleep(0.1)


if __name__ == "__main__":
    main()

However when trying to communicate with it from Erlang, while messages are getting through, they are NOT going to the correct process, because my print("-------------> inside handle_inbox <---------------") statement is never being executed.

From Erlang:

tbrowne@cubietruck ~/c/e/p/p/python> erl -name [email protected] -setcookie COOKIE
Erlang/OTP 20 [erts-9.1] [source] [smp:2:2] [ds:2:2:10] [async-threads:10] [hipe] [kernel-poll:false]

Eshell V9.1  (abort with ^G)
([email protected])1> net_adm:ping('[email protected]').        
pong
([email protected])2> {my_process, '[email protected]'} ! hello.                            
hello
([email protected])3>

So ping/pong no problem, but on Python side I never see my -------> inside handle_inbox <-------- printed even though message is getting through. So I assume my_process is not getting it? Python side output:


Dist: Listening for dist connections on port 46019
EPMD: Connecting 127.0.0.1:4369
EPMD: Socket connected
EPMD: sending ALIVE2 req ('[email protected]', 77, (5, 5))
EPMD: Connected successfully (creation 1)
send -> (atom'[email protected]', atom'shell'): hello
Node._send_remote shell <- hello
Node: connect to node [email protected]
Dist-out: Send_name b'n\x05\x05\x00\x03\x0f\[email protected]'
Connection to ('127.0.0.1', 39413) established
Node: wait for 'node_connected'
Out-connection established with [email protected]
Node: connected
MonitorP: org Pid<2.71.0>@[email protected] targ net_kernel = <NetKernel at 0xb64a8f30>
send -> net_kernel: (atom'$gen_call', (Pid<2.71.0>@[email protected], Ref<2,185901,3780902913,1788095308>@[email protected]), (atom'is_auth', atom'[email protected]'))
send -> Pid<2.71.0>@[email protected]: (Ref<2,185901,3780902913,1788095308>@[email protected], atom'yes')
Node._send_remote Pid<2.71.0>@[email protected] <- (Ref<2,185901,3780902913,1788095308>@[email protected], atom'yes')
send -> my_process: hello

As you can see, it is getting something ("send -> my_process: hello") but it does not seem to be getting handled by my_process handle_inbox function. What's going wrong here. Error in my code? Error in docs? Or bug?

Using Python 3, Debian Stretch on ARM, Erlang 20. Same story on Python 3/Ubuntu 17.10/Erlang 20.1.

A test to run Erlang node and communicate, or kill it if test fails

Examples broken as a consequence of commit 7827635

The signature of Process.init no longer contains 'node_name', but the examples still try to pass it.

py vs native discrepancies

The PR that fixed so that the codec module actually can import the rust implementation (Pyrlang/Term#8), breaks pyrlang due to differences in how things are decoded. One bug I found and fixed but something makes the node be str when py implementation is used and Atom when rust implementation is used.

Row 336 in pyrlang/node.py will fail since the node is stored with key as str but the receiver_node is an Atom. This got me thinking that maybe we should allow for Atom to be inherited from str as done in erlpack. I would guess that more often then not having equality between Atom and str would be the expected behaviour in most use cases. And since it's a type we can guarantee that a roundtrip back to Erlang land will convert it back to an Atom.

Add Cloudi to the docs

Somebody in IRC tells me, that Cloudi is suitable in order to handle throughput inbetween Python and Erlang, because Pyrlang is a single socket connection for distributed Erlang node communication:

This project allows us to have multiple socket connections to python processes (and separate threads if you choose), Some comparison to cnodes is at http://cloudi.org/faq.html#4_Erlang

Docs do not explain how to send a message from Python to a non-named Erlang/Elixir process

http://pyrlang.readthedocs.io/en/latest/getting_started.html#send-from-python-to-a-remote

In the above documentation, it is clear how to send a message from a Pyrlang process back to Erlang or Elixir if the process in Erlang/Elixir is already named (example shows with names "shell"). But it is not clear how to format on the Python side if one wants to send to a regular pid such as #PID<0.264.0>.

Does one send "#PID<0.264.0>" as a string? Does one use 264 as an integer? Is there a pid constructor in Pyrlang? Nobody knows.

I am happy to update the docs myself but first I would need to know how the above is done.

Improved Erlang process representation in Python

Better feature mirroring from Erlang in Python:

Better defined API for Python handlers of Pyrlang events in Process class
Exception handling/crash imitation with links/monitors triggering and process ending

Python 2 is not working due to presence of Python 3 type hints

Need to devise a clever way to retain python3 type hints and make it compatible with python2.

For ETF codec: implement atom table

Large terms with a lot of atoms would benefit from having atom table transferred as a part of the encoded term.

Implemented for Python encoder
Implemented for Python decoder
Implemented for native (Rust) encoder
Implemented for native (Rust) decoder

ModuleNotFoundError when making example 1

Hi there,

I'm using tag 0.9 on the master branch and after installing requirements with pip install -r requirements.txt I try running make example1 but I get the following error:

PYTHONPATH=/home/icub/user_files/project/bgvenv/Pyrlang:/home/icub/user_files/project/bgvenv/Pyrlang/Term PYRLANG_ENABLE_LOG_FORMAT=1 PYRLANG_LOG_LEVEL=DEBUG python3 examples/e01_simple_node.py
Traceback (most recent call last):
  File "examples/e01_simple_node.py", line 14, in <module>
    from pyrlang import Node
  File "/home/icub/user_files/project/bgvenv/Pyrlang/pyrlang/__init__.py", line 15, in <module>
    from pyrlang.node import Node, NodeException
  File "/home/icub/user_files/project/bgvenv/Pyrlang/pyrlang/node.py", line 20, in <module>
    from pyrlang.dist.distribution import ErlangDistribution
ModuleNotFoundError: No module named 'pyrlang.dist'
Makefile:12: recipe for target 'example1' failed
make: *** [example1] Error 1

I've also tried running other examples but I get the same error.

Wrong parameter for GenServer inside elixir/e10.py example

First of all, thank you for your effort.
While evaluating your library for Python-Elixir communication I've found a bug inside: https://github.com/Pyrlang/Pyrlang/blob/master/examples/elixir/e10.py

GenServer requires node_name (string) as parameter:

class GenServer(Process):
    def __init__(self, node_name: str, ...

but inside example you are passing node object:

class MyProcess(GenServer):
    def __init__(self, node) -> None:
        GenServer.__init__(self, node,
...
    node = Node(node_name="[email protected]", cookie="COOKIE", engine=event_engine)
    MyProcess(node)

this results in:

Traceback (most recent call last):
  File "/home/ute/auto/elixir/Pyrlang/examples/elixir/e10.py", line 52, in <module>
    main()
  File "/home/ute/auto/elixir/Pyrlang/examples/elixir/e10.py", line 46, in main
    MyProcess(node)
  File "/home/ute/auto/elixir/Pyrlang/examples/elixir/e10.py", line 26, in __init__
    accepted_calls=['hello', 'hello_again'])
  File "/home/ute/auto/elixir/Pyrlang/pyrlang/gen_server.py", line 57, in __init__
    super().__init__(node_name=node_name)
  File "/home/ute/auto/elixir/Pyrlang/pyrlang/process.py", line 58, in __init__
    node_obj = Node.all_nodes[node_name]
KeyError: <pyrlang.node.Node object at 0x7fd01f2b3c18>

To fix it just change to:

class MyProcess(GenServer):
    def __init__(self, node) -> None:
        GenServer.__init__(self, node_name=node.node_name_,
                           accepted_calls=['hello', 'hello_again'])

Regards
Grzegorz Latuszek

Python node crashes when using Elixir's GenServer.call() with a python pid

Python node crashes when using Elixir's GenServer.call() with a python pid.

Python code

import gevent
from gevent import monkey
monkey.patch_all()
import Pyrlang
from Pyrlang import Atom, Process, gen


class MyProcess(Process):
    def __init__(self, node) -> None:
        Process.__init__(self, node)
        node.register_name(self, Atom('my_process'))
        print("registering process - 'my_process'")

    def handle_one_inbox_message(self, msg):
        print("Incoming to", self.pid_, type(self.pid_), msg)
        gencall = gen.parse_gen_message(msg)

        if isinstance(gencall, str):
            print("MyProcess:", gencall)
            return

        print('replying pid')
        gencall.reply(local_pid=self.pid_, result=self.pid_)


def main():
    node = Pyrlang.Node("[email protected]", "COOKIE")
    node.start()

    mp = MyProcess(node)

    while True:
        # Sleep gives other greenlets time to run
        gevent.sleep(0.5)


if __name__ == "__main__":
    main()

IEX session

iex(erl@127.0.0.1)1> my_process = {:"my_process", :"[email protected]"}
iex(erl@127.0.0.1)2> my_process_pid = GenServer.call(my_process, "hello")
#PID<9956.0.3>
iex(erl@127.0.0.1)3> my_process_pid = GenServer.call(my_process_pid, "hello")
** (exit) exited in: GenServer.call(#PID<9956.0.3>, "hello", 5000)
    ** (EXIT) time out
    (elixir) lib/gen_server.ex:831: GenServer.call/3

Python output

Dist: Listening for dist connections on port 61516
EPMD: Connecting 127.0.0.1:4369
EPMD: Socket connected
EPMD: sending ALIVE2 req ('[email protected]', 77, (5, 5))
EPMD: Connected successfully (creation 3)
register_new_process pid: Pid<3.0.2>@[email protected]
Node: connect to node [email protected]
Dist-out: send_name b'n\x05\x05\x00\x03\x0f\[email protected]' ([email protected])
Connection to ('127.0.0.1', 54704) established
Node: wait for 'node_connected'
Out-connection established with [email protected]
Node: connected
registering process - 'my_process'
MonitorP: orig=Pid<2.90.0>@[email protected] targ=my_process -> <MyProcess at 0x107332210>
Node: send local reg=my_process receiver=<MyProcess at 0x107332210> msg=(atom'$gen_call', (Pid<2.90.0>@[email protected], Ref<2,198356,1675886593,4288514664>@[email protected]), b'hello')
<MyProcess at 0x107332210>: handle_inbox (atom'$gen_call', (Pid<2.90.0>@[email protected], Ref<2,198356,1675886593,4288514664>@[email protected]), b'hello')
Incoming to Pid<3.0.3>@[email protected] <class 'Pyrlang.Term.pid.Pid'> (atom'$gen_call', (Pid<2.90.0>@[email protected], Ref<2,198356,1675886593,4288514664>@[email protected]), b'hello')
replying
MonitorP: orig=Pid<2.90.0>@[email protected] targ=Pid<3.0.3>@[email protected] -> <MyProcess at 0x107332210>
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/gevent/greenlet.py", line 536, in run
    result = self._run(*self.args, **self.kwargs)
  File "/usr/local/Cellar/python3/3.6.3/Frameworks/Python.framework/Versions/3.6/bin/src/pyrlang/Pyrlang/Dist/helpers.py", line 55, in _handle_socket_read
    collected1 = handler.consume(collected)
  File "/usr/local/Cellar/python3/3.6.3/Frameworks/Python.framework/Versions/3.6/bin/src/pyrlang/Pyrlang/Dist/base_connection.py", line 108, in consume
    if self.on_packet(packet):
  File "/usr/local/Cellar/python3/3.6.3/Frameworks/Python.framework/Versions/3.6/bin/src/pyrlang/Pyrlang/Dist/out_connection.py", line 73, in on_packet
    return self.on_packet_connected(data)
  File "/usr/local/Cellar/python3/3.6.3/Frameworks/Python.framework/Versions/3.6/bin/src/pyrlang/Pyrlang/Dist/base_connection.py", line 254, in on_packet_connected
    self.on_passthrough_message(control_term, msg_term)
  File "/usr/local/Cellar/python3/3.6.3/Frameworks/Python.framework/Versions/3.6/bin/src/pyrlang/Pyrlang/Dist/base_connection.py", line 159, in on_passthrough_message
    receiver=control_term[3],
IndexError: tuple index out of range
Thu Jan 25 19:59:17 2018 <Greenlet at 0x107332178: _handle_socket_read(<Pyrlang.Dist.out_connection.OutConnection object , <gevent._socket3.socket object, fd=8, family=2, ty)> failed with IndexError

When running GenServer.call({:"my_process", "[email protected]"}, "hello"), Pyrlang/Dist/base_connection.py@159 control_term is (6, Pid<2.90.0>@[email protected], atom'', atom'my_process').

When running GenServer.call(a_pid, "hello"), Pyrlang/Dist/base_connection.py@159 control_term is (2, atom'', Pid<1.0.3>@[email protected]).

Sending integers to Python RPC is not possible?

Sending integers to Python RPC does not seem to work. Is that on purpose? Must I serialize non-strings? Examples below from Elixir.


tbrowne@cubietruck ~/c/e/p/p/p/Pyrlang> iex --sname erl --cookie COOKIE
Erlang/OTP 20 [erts-9.1] [source] [smp:2:2] [ds:2:2:10] [async-threads:10] [hipe] [kernel-poll:false]

Interactive Elixir (1.5.2) - press Ctrl+C to exit (type h() ENTER for help)
iex(erl@localhost)1> :rpc.call(:'py@localhost', :'test2', :'printit2', ['1', '2'])
:undefined
iex(erl@localhost)2> :rpc.call(:'py@localhost', :'test2', :'printit2', [1, 2])    
{:badrpc,
 {:EXIT,
  {{:AttributeError,
    %{args: {'\'str\' object has no attribute \'elements_\''},
      traceback: 'Traceback (most recent call last):\n  File "/home/tbrowne/code/elix/pyrr/priv/python/Pyrlang/Pyrlang/rex.py", line 62, in handle_one_inbox_message\n    args = gencall.get_args()\n  File "/home/tbrowne/code/elix/pyrr/priv/python/Pyrlang/Pyrlang/gen.py", line 72, in get_args\n    return self.args_.elements_\nAttributeError: \'str\' object has no attribute \'elements_\'\n'}},
   {:gen_server, :call,
    [{:rex, :py@localhost}, {:call, :test2, :printit2, [1, 2], #PID<0.60.0>},
     :infinity]}}}}
iex(erl@localhost)3>

Herewith the Python side:

tbrowne@cubietruck ~/c/e/p/p/p/Pyrlang> python3 test2.py
Dist: Listening for dist connections on port 35315
EPMD: Connecting 127.0.0.1:4369
EPMD: Socket connected
EPMD: sending ALIVE2 req ('py@localhost', 77, (5, 5))
EPMD: Connected successfully (creation 3)
send -> (atom'erl@localhost', atom'shell'): hello
Node._send_remote shell <- hello
Node: connect to node erl@localhost
Dist-out: Send_name b'n\x05\x05\x00\x03\x0f\xbcpy@localhost'
Connection to ('127.0.0.1', 35985) established
Node: wait for 'node_connected'
Out-connection established with erl@localhost
Node: connected
MonitorP: org Pid<3.95.0>@erl@localhost targ rex = <Rex at 0xb650e030>
send -> rex: (atom'$gen_call', (Pid<3.95.0>@erl@localhost, Ref<3,50579,3058171905,2856406562>@erl@localhost), (atom'call', atom'test2', atom'printit2', ['1', '2'], Pid<3.60.0>@erl@localhost))
1
2
send -> Pid<3.95.0>@erl@localhost: (Ref<3,50579,3058171905,2856406562>@erl@localhost, None)
Node._send_remote Pid<3.95.0>@erl@localhost <- (Ref<3,50579,3058171905,2856406562>@erl@localhost, None)
MonitorP: org Pid<3.95.0>@erl@localhost targ rex = <Rex at 0xb650e030>
send -> rex: (atom'$gen_call', (Pid<3.95.0>@erl@localhost, Ref<3,50619,3058171905,2856406562>@erl@localhost), (atom'call', atom'test2', atom'printit2', '\x01\x02', Pid<3.60.0>@erl@localhost))

and here is test2.py

import gevent
from gevent import monkey
monkey.patch_all()

import Pyrlang
from Pyrlang import Atom


def addit(x, y):
    return(x + y)


def printit(x):
    print(x)

def printit2(x, y):
    print(x)
    print(y)


def main():
    node = Pyrlang.Node("py@localhost", "COOKIE")
    node.start()

    pid = node.register_new_process(None)
    node.send(pid, (Atom('erl@localhost'), Atom('shell')), Atom('hello'))


    while True:
        gevent.sleep(0.5)


if __name__ == "__main__":
    main()

Native python object as return crashes node

If you create a function that returns a object that is not parable into erlang terms, the node crashes.

An example, create module tdf.py:

def to_die_for(cant_handle_empty_args_yet=None):
    import datetime
    return datetime.datetime.now()

start everything up and from erlang call:

rpc:call('[email protected]', 'tdf', 'to_die_for', [<<"foo">>]).

node dies with RecursionError maximum depth

ETF native codec doesn't work with pypy

Hello,

I am trying to run Pyrlang with the native term ETF library.

I have installed setuptools-rust and I see the output from Rust compiler:

(test_pypy) mik@lt-1:~/dev_python/Term$ LD_LIBRARY_PATH=/home/mik/dev_python/Term/target/release python setup.py develop
running develop
Checking .pth file support in /home/mik/dev_python/envs/test_pypy/site-packages/
/home/mik/dev_python/envs/test_pypy/bin/python -E -c pass
TEST PASSED: /home/mik/dev_python/envs/test_pypy/site-packages/ appears to support .pth files
running egg_info
writing pyrlang_term.egg-info/PKG-INFO
writing dependency_links to pyrlang_term.egg-info/dependency_links.txt
writing top-level names to pyrlang_term.egg-info/top_level.txt
reading manifest file 'pyrlang_term.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
writing manifest file 'pyrlang_term.egg-info/SOURCES.txt'
running build_ext
running build_rust
cargo rustc --lib --manifest-path Cargo.toml --features cpython/python3-sys cpython/extension-module --verbose -- --crate-type cdylib
       Fresh version_check v0.1.4
       Fresh unicode-xid v0.1.0
       Fresh ucd-util v0.1.1
       Fresh cc v1.0.25
       Fresh libc v0.2.43
       Fresh utf8-ranges v1.0.1
       Fresh rustc-serialize v0.3.24
       Fresh cfg-if v0.1.5
       Fresh rustc-demangle v0.1.9
       Fresh byteorder v0.4.2
       Fresh void v1.0.2
       Fresh byteorder v1.2.6
       Fresh byte v0.2.3
       Fresh regex-syntax v0.6.6
       Fresh rand v0.4.3
       Fresh log v0.4.5
       Fresh empty v0.0.4
       Fresh memchr v2.2.0
       Fresh num-traits v0.2.5
       Fresh proc-macro2 v0.4.19
       Fresh rand v0.3.22
       Fresh lazy_static v1.1.0
       Fresh log v0.3.9
       Fresh aho-corasick v0.7.3
       Fresh num-integer v0.1.39
       Fresh quote v0.6.8
       Fresh backtrace-sys v0.1.24
       Fresh num-complex v0.1.43
       Fresh thread_local v0.3.6
       Fresh syn v0.14.9
       Fresh num-bigint v0.1.44
       Fresh num-iter v0.1.37
       Fresh backtrace v0.3.9
       Fresh regex v1.1.6
       Fresh num-rational v0.1.42
       Fresh synstructure v0.9.0
       Fresh failure_derive v0.1.2
       Fresh num v0.1.42
       Fresh python3-sys v0.2.1
       Fresh failure v0.1.2
       Fresh compress v0.1.2
       Fresh cpython v0.2.1
       Fresh native_codec_impl v0.1.0 (/home/mik/dev_python/Term)
    Finished dev [unoptimized + debuginfo] target(s) in 0.05s
Creating /home/mik/dev_python/envs/test_pypy/site-packages/pyrlang-term.egg-link (link to .)
Adding pyrlang-term 1.2 to easy-install.pth file

Installed /home/mik/dev_python/Term
Processing dependencies for pyrlang-term==1.2
Finished processing dependencies for pyrlang-term==1.2
(test_pypy) mik@lt-1:~/dev_python/Term$

After that I tried to install Pyrlang but I get the same warning about ETF:

(test_pypy) mik@lt-1:~/dev_python/Pyrlang$ LD_LIBRARY_PATH=/home/mik/dev_python/Term/target/release python setup.py develop
running develop
Checking .pth file support in /home/mik/dev_python/envs/test_pypy/site-packages/
/home/mik/dev_python/envs/test_pypy/bin/python -E -c pass
TEST PASSED: /home/mik/dev_python/envs/test_pypy/site-packages/ appears to support .pth files
running egg_info
writing pyrlang.egg-info/PKG-INFO
writing dependency_links to pyrlang.egg-info/dependency_links.txt
writing top-level names to pyrlang.egg-info/top_level.txt
reading manifest file 'pyrlang.egg-info/SOURCES.txt'
writing manifest file 'pyrlang.egg-info/SOURCES.txt'
running build_ext
Creating /home/mik/dev_python/envs/test_pypy/site-packages/pyrlang.egg-link (link to .)
Adding pyrlang 0.9 to easy-install.pth file

Installed /home/mik/dev_python/Pyrlang
Processing dependencies for pyrlang==0.9
Finished processing dependencies for pyrlang==0.9
(test_pypy) mik@lt-1:~/dev_python/Pyrlang$ cd ../pyrlang_pypy/
(test_pypy) mik@lt-1:~/dev_python/pyrlang_pypy$ python start_node.py
Native term ETF codec library import failed, falling back to slower Python impl
^C
KeyboardInterrupt
(test_pypy) mik@lt-1:~/dev_python/pyrlang_pypy$

What must I do to get the native codec ?

Split implementations for Gevent and Asyncio and deprecate Gevent

Migrate logging to standard Python logger
Offer implementation of backend for Gevent
Offer implementation of backend for Asyncio

Documentation update: Cookbook needs review

The code in the Cookbook looks old.
Needs to be updated to the new API.

EPMD implementation

To allow Pyrlang run on nodes where Erlang is not installed and EPMD is not present, we want a reasonable solution for running our own EPMD daemon, or to reuse existing EPMD from Erlang/OTP source. Come up with ideas.

Choose EPMD implementation or write our own
Apply the solution (have EPMD available for nodes where Erlang is missing)

Release Pyrlang as two separate libraries with proper setuptools scripts

setuptools script for term lib
setuptools script for pyrlang lib
renaming stuff from Term to term
renaming stuff from Pyrlang to pyrlang
releasing together as subdirectories
documentation update for ETF codec

Elixir-friendly docs?

I'd be happy to help contribute nicely comprehensive Elixir-friendly documentation. Personally believe #13 and #18 would need to be addressed first however. Think Elixir users will find Pyrlang extremely useful for integrating Python's scientific / data science capabilities.

Can't import pyrlang Atom

OTP 22
code main.py :
from pyrlang import Node, Atom, GeventEngine # or AsyncioEngine
run python main.py
I get

Traceback (most recent call last):
  File "main.py", line 1, in <module>
    from pyrlang import Node, Atom, GeventEngine # or AsyncioEngine
ImportError: cannot import name 'Atom' from 'pyrlang' (/usr/local/lib/python3.7/site-packages/pyrlang/__init__.py)

Implement outgoing distribution protocol

Currently only incoming distribution connections are accepted. I.e. you can only initiate cluster link from another node but never from Python.

Make Node's network code real asyncio `async def`

Revisit all code in the library and make it asyncio only, and make it proper compatible with async def/await code style.
Remove asyncio adapter as we use the library directly now
Remove gevent dependency and gevent adapter
Engine argument to constructors now becomes event_loop from asyncio with default None (means take asyncio.get_event_loop())

This will allow closer simulation of Erlang execution model. Also this will allow inline receive (block thread until a message comes) and they are required to perform gen_server calls from Python.

Monitors, links and supervision - see how to make it work + document it

http://pyrlang.readthedocs.io/en/latest/node.html

There are a lot of juicy hints in this series of docs about how one might run a supervision strategy for a Pyrlang process. But it is not very user friendly for a beginner, especially considering a large proportion of the python population is not very advanced, and also, a lot of "newbies" (like me) are coming into Erlang world via Elixir. May I propose that one of the core contributors to Pyrlang explain to me exactly how one would run a one-for-one supervision strategy from Erlang master to Pyrlang node, and then I will try to write user-friendly documentation? The core idea is that supervision is one of the main things that Erlang world offers and it should therefore be very clear how this benefit is available in Pyrlang.

The idea is:

If Erlang node crashes, Pyrlang node must terminate, but must have a callback for cleanup
if Pyrlang node or process crashes, then Erlang side must reliably be notified.

This is as simple as it gets, but would allow us to "compete" against other strategies of interop between Elixir/Erlang and Python, such as zeromq and/or grpc which, apart from supervision strategies, can do most of what pyrlang does (I think).

Add support for {Node, Name} or {Node, Pid} in Node.send()

Atom.text_ not implemented

So, I'm not sure if this is an issue with my particular environment, which seems likely, but when attempting to use gen_server.call from elixir to invoke a procedure in Python, I got the following error:

Traceback (most recent call last):
  File "src/gevent/greenlet.py", line 766, in gevent._greenlet.Greenlet.run
  File "/Users/claytoncrockett/Documents/Podium/pyrlang_test/Pyrlang/pyrlang/async_support/gevent_engine.py", line 206, in _read_loop
    collected1 = proto.on_incoming_data(collected)
  File "/Users/claytoncrockett/Documents/Podium/pyrlang_test/Pyrlang/pyrlang/dist/base_dist_protocol.py", line 154, in on_incoming_data
    if self.on_packet(packet):
  File "/Users/claytoncrockett/Documents/Podium/pyrlang_test/Pyrlang/pyrlang/dist/out_dist_protocol.py", line 50, in on_packet
    return self.on_packet_connected(data)
  File "/Users/claytoncrockett/Documents/Podium/pyrlang_test/Pyrlang/pyrlang/dist/base_dist_protocol.py", line 370, in on_packet_connected
    self.on_passthrough_message(control_term, msg_term)
  File "/Users/claytoncrockett/Documents/Podium/pyrlang_test/Pyrlang/pyrlang/dist/base_dist_protocol.py", line 209, in on_passthrough_message
    message=msg_term)
  File "/Users/claytoncrockett/Documents/Podium/pyrlang_test/Pyrlang/pyrlang/node.py", line 305, in send
    return self._send_local_registered(receiver, message)
  File "/Users/claytoncrockett/Documents/Podium/pyrlang_test/Pyrlang/pyrlang/node.py", line 248, in _send_local_registered
    receiver_obj.deliver_message(msg=message)
  File "/Users/claytoncrockett/Documents/Podium/pyrlang_test/Pyrlang/pyrlang/process.py", line 148, in deliver_message
    self.handle_one_inbox_message(msg)
  File "/Users/claytoncrockett/Documents/Podium/pyrlang_test/Pyrlang/pyrlang/rex.py", line 59, in handle_one_inbox_message
    gencall = gen.parse_gen_call(msg, node_name=self.node_name_)
  File "/Users/claytoncrockett/Documents/Podium/pyrlang_test/Pyrlang/pyrlang/gen.py", line 127, in parse_gen_call
    if not isinstance(msg[0], Atom) or msg[0].text_ != '$gen_call':
AttributeError: 'Atom' object has no attribute 'text_'

The relevant line of code is this:

    if not isinstance(msg[0], Atom) or msg[0].text_ != '$gen_call':

Changing it to:

    if not isinstance(msg[0], Atom) or str(msg[0]) != '$gen_call':

works flawlessly. I intend to submit a PR around this, but I'm wondering if this is likely to cause issues for others who are using this library.

setuptools

We should implement setuptools to help with an easy installation.

Also we should make sure it works directly for python 2 and 3.

Dependencies
Python 2 support
Python 3 support
Wheels (also C extensions for major dists?), or maybe python only fallback

Review and harden input checks in native ETF decoder

Doesn't work on OTP 22

Starting erlang node with iex and trying to connect to pyrlang node:

iex --sname b --cookie 123
iex(b@alex-pc)1> :net_adm.ping(:'c@alex-pc')
:pang

Logs from pyrlang:

ERL_NODE=c@alex-pc ERL_COOKIE=123 PYRLANG_ENABLE_LOG_FORMAT=1 PYRLANG_LOG_LEVEL=DEBUG python main.py                                                                                                                         1 ↵  17:30:59
2019-06-21 17:31:00,076 [pyrlang] gevent_engine:113: Listening on ('0.0.0.0', 0) (40783)
2019-06-21 17:31:00,076 [pyrlang.dist] distribution:58: Listening for dist connections on port 40783
2019-06-21 17:31:00,076 [pyrlang] epmd:89: Connecting 127.0.0.1:4369
2019-06-21 17:31:00,081 [pyrlang] epmd:100: Socket connected
2019-06-21 17:31:00,081 [pyrlang] epmd:167: sending ALIVE2 req n=c@alex-pc (77) vsn=(5, 5)
2019-06-21 17:31:00,082 [pyrlang] epmd:119: Connected successfully (creation 3)
2019-06-21 17:31:00,082 [pyrlang] process:99: Spawned process <3.0.0 @ c@alex-pc>
2019-06-21 17:31:00,082 [pyrlang] process:99: Spawned process <3.0.1 @ c@alex-pc>
2019-06-21 17:31:00,082 [pyrlang] process:99: Spawned process <3.0.2 @ c@alex-pc>
2019-06-21 17:31:00,082 [root] main:17: Registering process - 'my_process'
2019-06-21 17:31:07,437 [pyrlang] gevent_engine:168: Client ('127.0.0.1', 47601) connected
2019-06-21 17:31:07,448 [pyrlang.dist] in_dist_protocol:76: RECV_NAME: (0, 5) b@alex-pc
2019-06-21 17:31:07,449 [pyrlang.dist] in_dist_protocol:117: Sending challenge (our number is 1732719476) c@alex-pc
2019-06-21 17:31:07,470 [pyrlang.dist] in_dist_protocol:95: challengereply: peer's challenge 1761004289
2019-06-21 17:31:07,471 [pyrlang.dist] base_dist_protocol:380: Connected to b@alex-pc
2019-06-21 17:31:07,471 [pyrlang.dist] in_dist_protocol:111: Incoming established with b@alex-pc
2019-06-21 17:31:07,472 [pyrlang] node:162: Node b@alex-pc connected
2019-06-21 17:31:07,504 [pyrlang] gevent_engine:176: Exception: Traceback (most recent call last):
  File "/home/alex/Sites/py_dedup/env/lib/python3.7/site-packages/pyrlang/async_support/gevent_engine.py", line 173, in _serve_loop
    _read_loop(proto=proto, sock=sock)
  File "/home/alex/Sites/py_dedup/env/lib/python3.7/site-packages/pyrlang/async_support/gevent_engine.py", line 206, in _read_loop
    collected1 = proto.on_incoming_data(collected)
  File "/home/alex/Sites/py_dedup/env/lib/python3.7/site-packages/pyrlang/dist/base_dist_protocol.py", line 154, in on_incoming_data
    if self.on_packet(packet):
  File "/home/alex/Sites/py_dedup/env/lib/python3.7/site-packages/pyrlang/dist/in_dist_protocol.py", line 50, in on_packet
    return self.on_packet_connected(data)
  File "/home/alex/Sites/py_dedup/env/lib/python3.7/site-packages/pyrlang/dist/base_dist_protocol.py", line 370, in on_packet_connected
    self.on_passthrough_message(control_term, msg_term)
  File "/home/alex/Sites/py_dedup/env/lib/python3.7/site-packages/pyrlang/dist/base_dist_protocol.py", line 230, in on_passthrough_message
    ref=ref)
  File "/home/alex/Sites/py_dedup/env/lib/python3.7/site-packages/pyrlang/node.py", line 440, in monitor_process
    return self._monitor_local_process(origin_pid, target_pid, m_ref)
  File "/home/alex/Sites/py_dedup/env/lib/python3.7/site-packages/pyrlang/node.py", line 466, in _monitor_local_process
    target_proc.add_monitored_by(pid=origin_pid, ref=ref)
  File "/home/alex/Sites/py_dedup/env/lib/python3.7/site-packages/pyrlang/process.py", line 237, in add_monitored_by
    self._monitored_by[ref] = pid
  File "/home/alex/Sites/py_dedup/env/lib/python3.7/site-packages/term/reference.py", line 81, in __hash__
    return hash((REF_MARKER, self.node_name_, self.id_, self.creation_))
TypeError: unhashable type: 'list'

2019-06-21 17:31:07,504 [pyrlang] gevent_engine:179: Client ('127.0.0.1', 47601) disconnected
2019-06-21 17:31:07,505 [pyrlang] node:169: Node b@alex-pc disconnected

Tried with both asyncio and gevent engines - same error

can't start start GenServer process via rpc call

So when I try to create a server through an RPC call, code something like this:

class Server(GenServer):
    def __init__(self):
        super().__init__(['test'])

    def test(self, *args):
        print("test call with args: {}".format(args))


def init_server():
    return Server()

Then on the erlang side I get

([email protected])4> rpc:call('[email protected]', jso, init_server, []).
=WARNING REPORT==== 4-Oct-2019::14:09:09.175699 ===
'[email protected]' got a corrupted external term from '[email protected]' on distribution channel 8172
<<...,131,104,2,114,0,3,119,20,112,121,114,108,97,110,103,101,114,108,64,49,50,55,46,48,46,48,46,49,3,0,1,118,127,32,164,0,5,107,196,249,243>>
ATOM_CACHE_REF translations: none

{badrpc,nodedown}
([email protected])5>

Is it the use of ATOM_CACHE thing referenced in #35 or something else perhaps.

pyrlang v.0.7 build fails with Python 3.7.1

First, thanks for your work in this! I am trying to orchestrate heavy computations with Python and this looks very promising.

Installing Python dependencies using pip install -r requirements.txt fails when trying to build gevent and greenlet because of compilation errors. This is likely due to the version of Cython used for gevent v1.2.1
and greenlet v.0.4.12 is too old for Python 3.7.

Upgrading greenlet to 0.14.5 and gevent to 1.3.7 (latest releases) solves the issue.

I can test other versions of python and submit a PR if that helps.

I don't seem to be able to multithread using gevent

I am trying to run multiple concurrent processes in Pyrlang using a class that inherits from Process.

So I start up my elixir node:

tbrowne@calculon:~/scratch$ iex --name [email protected] --cookie COOKIE
Erlang/OTP 20 [erts-9.1] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:10] [hipe] [kernel-poll:false]

Interactive Elixir (1.5.2) - press Ctrl+C to exit (type h() ENTER for help)
iex([email protected])1>

And here is my code which connects to above node, and then launches 2 processes.

import socket
import gevent
from gevent import monkey
monkey.patch_all()
import Pyrlang
from Pyrlang import Atom
from Pyrlang import Process
import pdb

def myip():
    s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
    try:
        s.connect(('google.com', 0))
        myip = s.getsockname()[0]
    except: # usually osx fails so try another method
        print("failed using UDP socket google.com...trying gethostbyname...")
        myip = socket.gethostbyname(socket.gethostname())
    return myip



class t3master(Process):
    """ thread which continuously looks for master IP address """

    def __init__(self, node, tname):
        Process.__init__(self, node)
        node.register_name(self, Atom(tname))
        self.tname = tname
        self.looper()

    def handle_inbox(self) -> None:
        msg = self.inbox_.receive(filter_fn = lambda x: x is not None)
        if msg is not None:
            print("Incoming handle_inbox", msg)

    def looper(self):
        while True:
            print(self.tname)
            gevent.sleep(1)


if __name__ == "__main__":
    masterip = myip()
    sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
    sock.bind(('', 8477))
    node = Pyrlang.Node("py@" + masterip, "COOKIE")
    node.start()
    pid = node.register_new_process(None)
    node.send(pid, (Atom("erl@" + masterip), Atom("shell")), Atom(masterip))
    t3 = t3master(node, 't3master')
    t4 = t3master(node, 't4master')

    while True:
        print("dorunrun")
        gevent.sleep(2)

However only one of said processes actually does anyting with its looper function. It connects fine, but I only see "t3master" there being printed every second. I was expecting also to see "t4 master" and "dorunrun" being printed at regular intervals:

tbrowne@calculon:~/scratch$ python3 geventno.py 
Dist: Listening for dist connections on port 46557
EPMD: Connecting 127.0.0.1:4369
EPMD: Socket connected
EPMD: sending ALIVE2 req ('[email protected]', 77, (5, 5))
EPMD: Connected successfully (creation 2)
send -> (atom'[email protected]', atom'shell'): 192.168.1.117
Node._send_remote shell <- 192.168.1.117
Node: connect to node [email protected]
Dist-out: Send_name b'n\x05\x05\x00\x03\x0f\[email protected]'
Connection to ('192.168.1.117', 43563) established
Node: wait for 'node_connected'
Out-connection established with [email protected]
Node: connected
t3master
t3master
t3master
t3master
t3master
t3master
t3master
t3master
t3master
.......etc

Am I using the Process class correctly?

Support Erlang/OTP 20

Seems to be getting this error:

Connection attempt to node '' aborted since it cannot handle ["UTF8_ATOMS"]

Cannot ping to python node if full domain address eg 192.168.1.117 whereas 127.0.0.1 works

Code:

import gevent
from gevent import monkey
monkey.patch_all()

import Pyrlang

def main():
    node = Pyrlang.Node("[email protected]", "COOKIE")
    node.start()

    while True:
        # Sleep gives other greenlets time to run
        gevent.sleep(0.1)

if __name__ == "__main__":
    main()

Ifconfig:

tbrowne@calculon:~$ ifconfig
eno1      Link encap:Ethernet  HWaddr b8:ca:3a:a6:a5:96  
          inet addr:192.168.1.117  Bcast:192.168.1.255  Mask:255.255.255.0
          inet6 addr: fe80::cb77:85a:a878:d0d6/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:112497668 errors:0 dropped:0 overruns:0 frame:0
          TX packets:126299904 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:67358966716 (67.3 GB)  TX bytes:107202418020 (107.2 GB)
          Interrupt:20 Memory:f3100000-f3120000 

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:504849 errors:0 dropped:0 overruns:0 frame:0
          TX packets:504849 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:100673650 (100.6 MB)  TX bytes:100673650 (100.6 MB)

Ping from Elixir (no problem):

tbrowne@calculon:~$ iex --name [email protected] --cookie COOKIE
Erlang/OTP 20 [erts-9.1] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:10] [hipe] [kernel-poll:false]

Interactive Elixir (1.5.2) - press Ctrl+C to exit (type h() ENTER for help)
iex([email protected])1> :net_adm.ping(:'[email protected]')
:pong
iex([email protected])2>

Now change to full domain name ie from 127.0.0.1, to 192.168.1.117 as per ifconfig above

Code:

import gevent
from gevent import monkey
monkey.patch_all()

import Pyrlang

def main():
    node = Pyrlang.Node("[email protected]", "COOKIE")
    node.start()

    while True:
        # Sleep gives other greenlets time to run
        gevent.sleep(0.1)

if __name__ == "__main__":
    main()

Python side seems to connect fine:

tbrowne@calculon:~/Dropbox/code/scratch/Pyrlang$ python test1.py
Dist: Listening for dist connections on port 33351
EPMD: Connecting 127.0.0.1:4369
EPMD: Socket connected
EPMD: sending ALIVE2 req ('[email protected]', 77, (5, 5))
EPMD: Connected successfully (creation 2)

But problem: get pang from Elixir:


tbrowne@calculon:~$ iex --name [email protected] --cookie COOKIE
Erlang/OTP 20 [erts-9.1] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:10] [hipe] [kernel-poll:false]

Interactive Elixir (1.5.2) - press Ctrl+C to exit (type h() ENTER for help)
iex([email protected])1> :net_adm.ping(:'[email protected]') 
:pang
iex([email protected])2>

This works fine Elixir to Elixir for example:


tbrowne@calculon:~/Dropbox/code/scratch/Pyrlang$ iex --name [email protected] --cookie COOKIE
Erlang/OTP 20 [erts-9.1] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:10] [hipe] [kernel-poll:false]

Interactive Elixir (1.5.2) - press Ctrl+C to exit (type h() ENTER for help)
iex([email protected])1>

tbrowne@calculon:~$ iex --name [email protected] --cookie COOKIE
Erlang/OTP 20 [erts-9.1] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:10] [hipe] [kernel-poll:false]

Interactive Elixir (1.5.2) - press Ctrl+C to exit (type h() ENTER for help)
iex([email protected])1> :net_adm.ping(:'[email protected]')
:pong
iex([email protected])2>

Using Python 3.6.3, Elixir 1.5.2, Erlang 20.1, Ubuntu 16.04. Also tested on Armbian Stretch with identical results.

Links and monitors

Process links: link/unlink API, link triggering on exit
Process links: Local processes exit propagation
Process links: Remote processes exit propagation
Process monitors: monitor/demonitor API, monitor triggering on exit
Guard process interactions with try/catch and trigger Process.exit
Example for links and monitors
Example of Erlang supervisor herding Python processes
Documentation

Does not work with OTP/23's updated node communication set up

Hello, I just came across this project and am looking to use it for something I'm working on but I can't seem to get any of the examples working. When I try running example 10, through make example10a and then make example10b in another terminal tab, I get this error on the elixir side:

16:52:10.878 [error] ** :"[email protected]": Connection attempt to node :"[email protected]" aborted since it cannot handle ['BIG_CREATION'].**

I've tried looking around on the internet for this issue but wasn't able to find anything describing this issue. My installation of elixir came with Erlang/OTP 23, and I see in the README it only lists support up to 21, could that be an issue?

Does not work under Windows due to dependency on term / tty

Windows PowerShell
Copyright (C) Microsoft Corporation. All rights reserved.

PS C:\Users\thoma> e:
PS E:\> cd code
PS E:\code> cd bbg
PS E:\code\bbg> ls


    Directory: E:\code\bbg


Mode                LastWriteTime         Length Name
----                -------------         ------ ----
d-----       12/15/2017   8:36 AM                Pyrlang
------       10/18/2017   3:48 PM           1127 bbsubout.py
------       10/18/2017   3:48 PM          44814 z3.pkl
------       10/30/2017  10:20 AM           3299 bbcl.py
------       10/18/2017   3:48 PM            560 bbkex.py
------       10/18/2017   3:48 PM          82436 z.pkl
------       10/18/2017   3:48 PM            914 eltest.py
------       10/18/2017   3:48 PM            839 bbsub.py
------       10/18/2017   3:48 PM            923 bbpub.py
------       10/18/2017   3:47 PM             22 README.md
------       12/19/2017   1:00 PM          32099 xxnode.py
------       12/19/2017  11:34 AM          31744 bbnode.py
------       12/19/2017  12:40 PM           6059 bbmaster.py


PS E:\code\bbg> cd Pyrlang
PS E:\code\bbg\Pyrlang> python test1.py
Traceback (most recent call last):
  File "test1.py", line 5, in <module>
    import Pyrlang
  File "E:\code\bbg\Pyrlang\Pyrlang\__init__.py", line 15, in <module>
    from Pyrlang.node import *
  File "E:\code\bbg\Pyrlang\Pyrlang\node.py", line 24, in <module>
    from Pyrlang.term import *
  File "E:\code\bbg\Pyrlang\Pyrlang\term.py", line 26, in <module>
    from Pyrlang.Dist import util
  File "E:\code\bbg\Pyrlang\Pyrlang\Dist\__init__.py", line 15, in <module>
    from Pyrlang.Dist.distribution import ErlangDistribution
  File "E:\code\bbg\Pyrlang\Pyrlang\Dist\distribution.py", line 27, in <module>
    from Pyrlang.Dist.out_connection import OutConnection
  File "E:\code\bbg\Pyrlang\Pyrlang\Dist\out_connection.py", line 25, in <module>
    from Pyrlang.Dist.base_connection import *
  File "E:\code\bbg\Pyrlang\Pyrlang\Dist\base_connection.py", line 25, in <module>
    from Pyrlang import logger, mailbox, term
ImportError: cannot import name 'term'
PS E:\code\bbg\Pyrlang>
PS E:\code\bbg\Pyrlang>
PS E:\code\bbg\Pyrlang> python
Python 3.4.5 |Anaconda custom (64-bit)| (default, Jul  5 2016, 14:53:07) [MSC v.1600 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import term
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Anaconda3\lib\site-packages\term\__init__.py", line 9, in <module>
    from termios import *
ImportError: No module named 'termios'
>>>

As you will see from this google search, termios is Linux/Unix only.

Personally I am wishing to create a Windows-domain Erlang node because the Bloomberg terminal, from which I subscribe to data, only supports Windows for the Desktop licence. So my use case is Erlang node in Python talks to Bloomberg, and transmits data to Linux side.

Elixir support

The documentation seems to suggest, that only Erlang is supported, while Elixir, LFE and so and are for sure also?

Runtime error when using send_remote

I have set up a scenario with 2 python processes that run as separate (remote) instances to test communication between two python Pyrlang processes registered as sender and receiver

From controller, I am executing

self.node.send_remote(sender=self.pid_, dst_node="[email protected]", receiver="receiver", message=Atom("try"))

Whenever I execute this command, I get the following traceback on sender:

Registering process - [email protected]
Task exception was never retrieved
future: <Task finished coro=<_generic_async_loop() done, defined at /home/icub/user_files/project/venv/Pyrlang/pyrlang/async/asyncio_engine.py:169> exception=RuntimeError('This event loop is already running',)>
Traceback (most recent call last):
  File "/home/icub/user_files/project/venv/Pyrlang/pyrlang/async/asyncio_engine.py", line 170, in _generic_async_loop
    while loop_fn():
  File "/home/icub/user_files/project/project/project/BaseGlue.py", line 61, in process_loop
    self.module_loop()
  File "/home/icub/user_files/project/project/project/Modules/Controller.py", line 75, in module_loop
    self.__node__._send_remote(sender=self.pid_, dst_node="[email protected]", receiver="receiver", message=Atom("try"))
  File "/home/icub/user_files/project/venv/Pyrlang/pyrlang/node.py", line 314, in _send_remote
    message=m)
  File "/home/icub/user_files/project/venv/Pyrlang/pyrlang/node.py", line 350, in _dist_command
    self.engine_.sleep(0.1)
  File "/home/icub/user_files/project/venv/Pyrlang/pyrlang/async/asyncio_engine.py", line 64, in sleep
    self.loop_.run_until_complete(asyncio.sleep(seconds))
  File "/usr/lib/python3.6/asyncio/base_events.py", line 455, in run_until_complete
    self.run_forever()
  File "/usr/lib/python3.6/asyncio/base_events.py", line 409, in run_forever
    raise RuntimeError('This event loop is already running')
RuntimeError: This event loop is already running
Socket closed

and the following error in receiver:

Error: Dist protocol version have: (5, 5) got: (5, 5) (state recvname)
Protocol requested to disconnect the socket

Any feedback would be greatly appreciated!

gen_server support for cleaner handling of gen messages

Create a mix-in class to combine with Pyrlang.Process which will map gen:call's to local calls to self.

RPC call without arguments not handled correct

When doing an rpc call without arguments like

rpc:call('[email protected]', 'Pyrlang.logger', 'tty', []).

The args variable in GenIncomingCall is a list. If you call with any argument it's a Pyrlang.term.List. in gen.py row 71
the code assumes it to be a Pyrlang.term.List causing an exception.

One way would be to check type here and return accordinlgy, but it's probably better to ensure that it's always a Pyrlang.term.List, even when it's empty.

Cannot connect to remote node from Python, after reboot

After rebooting a Python-side node, it will not connect to a remote node on the first attempt. One must first connect to an erlang/elixir node on the same physical machine, then after disconnecting, one can then connect to remote nodes.

Remote node running Erlang/Elixir: 192.168.1.199
Local node running Pyrlang: 192.168.1.192

code:

import gevent
from gevent import monkey
monkey.patch_all()
import pdb

import Pyrlang
from Pyrlang import Atom

def main():
    node = Pyrlang.Node("[email protected]", "COOKIE")
    node.start()

    # Attempt to send something will initiate a connection before sending
    pid = node.register_new_process(None)
    # To be able to send to Erlang shell by name first give it a registered
    # name: `erlang:register(shell, self()).`
    node.send(pid, (Atom('[email protected]'), Atom('shell')), Atom('hello'))

    while True:
        # Sleep gives other greenlets time to run
        gevent.sleep(0.1)

if __name__ == "__main__":
    main()

Start Elixir on remote node:

pi@flotilla:~/www/mspiex --name [email protected] --cookie COOKIE

Run code on local node:

tbrowne@cubietruck ~/c/e/p/p/python> python3 test6.py
Dist: Listening for dist connections on port 36539
EPMD: Connecting 127.0.0.1:4369
EPMD: connection error: [Errno 111] Connection refused
EPMD: Connecting 127.0.0.1:4369
EPMD: connection error: [Errno 111] Connection refused

Now change code so remote node is same computer so both 192.168.1.192:

tbrowne@cubietruck ~/c/e/p/p/python> cat test6.py
import gevent
from gevent import monkey
monkey.patch_all()
import pdb

import Pyrlang
from Pyrlang import Atom

def main():
    node = Pyrlang.Node("[email protected]", "COOKIE")
    node.start()

    # Attempt to send something will initiate a connection before sending
    pid = node.register_new_process(None)
    # To be able to send to Erlang shell by name first give it a registered
    # name: `erlang:register(shell, self()).`
    node.send(pid, (Atom('[email protected]'), Atom('shell')), Atom('hello'))

    while True:
        # Sleep gives other greenlets time to run
        gevent.sleep(0.1)

if __name__ == "__main__":
    main()

Start local machine Elixir node:


tbrowne@cubietruck ~> iex --name [email protected] --cookie COOKIE
Erlang/OTP 20 [erts-9.1] [source] [smp:2:2] [ds:2:2:10] [async-threads:10] [hipe] [kernel-poll:false]

Interactive Elixir (1.5.3) - press Ctrl+C to exit (type h() ENTER for help)
iex([email protected])1>

Start Pyrlang on local machine:

tbrowne@cubietruck ~/c/e/p/p/python> python3 test6.py
Dist: Listening for dist connections on port 41409
EPMD: Connecting 127.0.0.1:4369
EPMD: Socket connected
EPMD: sending ALIVE2 req ('[email protected]', 77, (5, 5))
EPMD: Connected successfully (creation 1)
send -> (atom'[email protected]', atom'shell'): hello
Node._send_remote shell <- hello
Node: connect to node [email protected]
Dist-out: Send_name b'n\x05\x05\x00\x03\x0f\[email protected]'
Connection to ('192.168.1.192', 34369) established
Node: wait for 'node_connected'
Out-connection established with [email protected]
Node: connected

Connected. Ping and RPC all work as expected. Now try again with first version of code, ie back to 192.168.1.199


tbrowne@cubietruck ~/c/e/p/p/python> python3 test6.py
Dist: Listening for dist connections on port 40479
EPMD: Connecting 127.0.0.1:4369
EPMD: Socket connected
EPMD: sending ALIVE2 req ('[email protected]', 77, (5, 5))
EPMD: Connected successfully (creation 2)
send -> (atom'[email protected]', atom'shell'): hello
Node._send_remote shell <- hello
Node: connect to node [email protected]
Dist-out: Send_name b'n\x05\x05\x00\x03\x0f\[email protected]'
Connection to ('192.168.1.199', 41883) established
Node: wait for 'node_connected'
Out-connection established with [email protected]
Node: connected

Connected. Ping and RPC work as expected. I have tried this on Linux, ARM instructions set and Intel instruction set, and even on Linux to Windows. Always behaves the same way. The python node will never connect to an Elixir or Erlang node on another IP address, unless you have first, once, connected to the same IP address as the Pyrlang node is running on. Thereafter you can stop all processes and start again to remote node and it works. Something weird is happening on ports initialization or something and I don't know what.

Basically, on first attempt after any reboot of the Pyrlang side remote-address node, you cannot connect to other-address Elixir/Erlang. You first have to connect to same-address elixir/erlang, and then for rest of life of session remote IP connections will work.

Detect socket errors and disconnections

While everything works on a first try, if you stop remote Erlang node and start it again, the things don't work anymore because it was never de-registered.

Gevent engine must detect socket errors correctly
Asyncio engine must detect socket errors correctly

Simplified flow for python calls

A more pythonistic way of calling functions/objects then MFA is needed. Also when the result of a call is something that can't/shoudn't be sent back to erlang, we need some way of keeping state on the python side and be able to use it later on.

first a calling scheme that fits python world better is

py:call(Node, ObjectPath, Args, KeyWordArgs)

To be able to handle results that we want to be able to use in later calls on python side, but don't want sent over the wire, we need some context on python side

Ctx = py:new_context(Node),
py:call(Ctx, ObjectPath, Arguments, KeywordArguments),

a full example:

test() ->
  Ctx = py:new_context('[email protected]'),
  DT = py:call(Ctx, <<"__import__">>, [<<"datetime">>], #{}),
  Now1 = py:call(Ctx, [DT, <<"datetime.now">>], [], #{}),
  Now2 = py:call(Ctx, [DT, datetime, <<"now">>], [], #{}),
  Res =  py:call(Ctx, [Now2, '__sub__'], [Now1], #{}),
  py:call(Ctx, [Res, <<"total_seconds">>], [], #{}).

Discussion

In python everything is an object. Some object make sense in converting to types that can be sent
back to erlang and some don't. The ones that doesn't make sense should just send back a reference,
that can be used in future calls.

However sometimes you might have data, for example a huge list, that you're not interested in having
in erlang, you want to do more processing on it before returning it. Therefore we need some way of
controlling if a reference or the actual data is sent back to erlang.

	(self.in_srv_, self.in_port_) = self.engine_.listen_with(
	protocol_class=InDistProtocol,
	protocol_args=[],
	protocol_kwargs=proto_kwargs
	)