Python about libzt HOT 30 OPEN

This is true, it's still a TODO item. Sometime this week I'll try to get a simple example in there.

from libzt.

An update on this ticket.

I've uploaded a beta of the Python package to PyPI (https://pypi.python.org/pypi/libzt), it is only available for macOS at the moment but will soon include other platforms.

We will have the following support:

• macOS: binary distribution
• Windows: binary distribution
• Linux: source distribution

An example of usage is as follows:

pip3 install libzt

test.py:

import libzt
import time

nwid = 0x1234567890ABCDEF

# Authorization required via my.zerotier.com if this is a private network).
print('Joining virtual network...')
libzt.zts_startjoin('whatev_config', nwid)

fd = libzt.zts_socket(AF_INET,SOCK_STREAM,0)
if fd < 0:
	print('error creating socket')
print('fd = ' + str(fd))

# Display info about this node
print('I am ' + str(hex(libzt.zts_get_node_id())))
address = None
libzt.zts_get_address(nwid, address, AF_INET)
print('assigned address = ' + str(address))

print('Looping forever, ping me or something')
while True:
    time.sleep(1)

The relevant subdirectory in this project for this package is: https://github.com/zerotier/libzt/tree/master/packages/pypi

I'll keep this ticket open for updates. Let me know if you find any bugs or have any feature requests.

from libzt.

Well that's just about another quarter, we still have a week or two to go, but while I remember might as well ask for an update.

Zerotier 2.0 is looking interesting, does that make this defunct?

from libzt.

Thanks for the compliment, @ciprianiacobescu!

Swig is not often picked up by new projects these days. I'm also not aware of SIP being used outside PyQt itself, but I could be wrong. I think a C++ binding-generator like that is more beneficial for large C++ APIs like Qt, though.

Depending on the size of the API, the current best / most popular tools I know of to wrap libraries and expose them to Python are:

1. Cython, which I use the most in pyzmq. Great for low-level integrations of C and Python, e.g. mapping a numpy array to an underlying buffer in C without copying, or carefully managing the GIL. The Cython syntax is also extra friendly for Python folks who want to wrap a C library (me), as opposed to C developers who want to expose their library to Python.
2. cffi, used by pyzmq when running with PyPy. Great, simple tool if your main task is to expose an API that takes simple arguments and returns simple types (it can do way more, but that's where it shines, I think).
3. pybind11 for wrapping modern C++ libraries. I have less experience with this one, but know several C++ folks on projects of different scales who are very happy with it.

All of these tools let you define standard Python Extensions in setup.py so that all the pip-install, wheel-building infrastructure/automation tools should work for you without much or any customization. Tip: please build your Python bindings with a setup.py, not a custom build system.

Based on skimming ZeroTierSockets.h, my first inclination would be to start with cffi.

The biggest challenge if you want a pip-installable package is how to deal with the wrapped library itself. pyzmq chooses the most-widely-installable route of bundling libzmq's source so that it can pip install from source without installing lizmq first. This is a lot of work to make function everywhere and may not be worth it, depending on your audience. We built most of that before wheels were allowed on PyPI, and probably wouldn't have bothered today. More common these days is to require the library to be installed to build from source, and then vendor the library in your wheels using something like the auditwheel/manylinux infrastructure on linux or delocate on mac. The nice thing about these tools is that they don't require any customization - you build and install the library as normal, then also build a wheel as normal. Running these tools will then unpack, detect, patch, and rebundle the wheel with any dependencies they find in a portable way.

from libzt.

Thanks again. For those of you lurking, a new batch of 1.3.4b1 wheels will be out tomorrow reflecting @linsomniac's improvements.

Already responded elsewhere but I figured an answer to your question has value here too:

The reason for the error:

ImportError: dynamic module does not define module export
function (PyInit__libzt)

is that we use swig to generate part of the wrapper and its conventions are a bit different.

In order to generate a wheel to test your changes locally you can do the following:

If you modify native sources, first generate a new native wrapper:

swig -c++ -python -o src/bindings/python/zt_wrap.cpp -Iinclude src/bindings/python/zt.i

Then, from pkg/pypi:

./build.sh ext # copies source to local directory and builds extension module
./build.sh wheel # package wheel and places it in `pkg/pypi/dist/`

From there you can pip install the *.whl and test.

from libzt.

Hello all! I'd like to finally take care of this. I've simplified the C API so that it's easier to wrap in something like Python and I've made a little bit of progress on getting a working extension module. I've made an attempt to wrap the library via SWIG and had some success. Everything seems to work internally and most of the API works. Unfortunately I'm not a Python expert and thus I have reached an impasse and am hoping one or more of you knows something that I do not. If we can solve this callback issue it will be trivial to generate an extension module.

I've written a minimal piece of code which exhibits the issue. The problem point is in passing a structure from C to the callback function in Python:

zt.c

#include "zt.h"

static struct zts_callback_msg object = {'a', 'b', 12345, NULL};

void zts_start(void(*callback)(struct zts_callback_msg *))
{
  callback(&object);
}

zt.h

#include <stdint.h>
#include <stdlib.h>

struct zts_callback_msg {
  uint8_t a;
  uint8_t b;
  uint32_t c;
  uint8_t* d;
};

void zts_start(void(*callback)(struct zts_callback_msg *));

This swig interface file includes the C API header, and provides a typemap for a callback function.

/* zt.i */

%include <stdint.i>

%module libzt
%{
#define SWIG_FILE_WITH_INIT
#include "zt.h"
%}


// a typemap for the callback, it expects the argument to be an integer
// whose value is the address of an appropriate callback function
%typemap(in) void (*f)(struct zts_callback_msg *) {
    $1 = (void (*)(struct zts_callback_msg *))PyLong_AsVoidPtr($input);
}

%pythoncode
%{

import ctypes

# a ctypes callback prototype
py_callback_type = ctypes.CFUNCTYPE(None, ctypes.c_void_p)

def zts_start(py_callback):

    # wrap the python callback with a ctypes function pointer
    f = py_callback_type(py_callback)

    # get the function pointer of the ctypes wrapper by casting it to void* and taking its value
    f_ptr = ctypes.cast(f, ctypes.c_void_p).value

    _libzt.zts_start(f_ptr)

%}

%include "zt.h"

When swig -python -py3 zt.i is issued, it successfully generates a zt_wrap.c and a libzt.py. I can now compile the necessary _libzt.so linked with the Python libraries and the aforementioned zt_wrap.c.

Makefile

all:
	swig -python -py3 zt.i
	clang -dynamiclib *.c `python3-config --include` `python3-config --ldflags` `python3-config --lib` -o _libzt.so

clean:
	rm -rf __pycache__
	rm -rf *.out *.dylib *.so *.a *.pyc libzt.py zt_wrap.c

So far so good!

app.py

import libzt

from ctypes import *

# Mock callback structure
class zts_callback_msg(Structure):
    _fields_ = [
        ('a', c_uint8),
        ('b', c_uint8),
        ('c', c_uint32),
        ('d', POINTER(c_uint8)),
    ]

callback_t = CFUNCTYPE(None, POINTER(zts_callback_msg))

zts_start = libzt.zts_start
zts_start.argtypes = [callback_t]
zts_start.restype   = None

def mycallback(obj):
    print("Received callback from ZT!")
    print(obj.contents.a)
    print(obj.contents.b)
    print(obj.contents.c)
    print(obj.contents.d)

# Example of how a user app would set a callback for libzt
zts_start(callback_t(mycallback))

When I place _libzt.so in the same directory as app.py and issue python3 app.py the following happens:

Traceback (most recent call last):
  File "app.py", line 28, in <module>
    zts_start(callback_t(mycallback))
  File "/Users/joseph/op/zt/libzt/latest_swig_effort/min_working/libzt.py", line 128, in zts_start
    return _libzt.zts_start(callback)
TypeError: in method 'zts_start', argument 1 of type 'void (*)(struct zts_callback_msg *)'

from libzt.

An update:

I've been spending some time on this and have pushed some new wheels for you all to try. @lpereira1's comment was particularly insightful. A director class was required.

You can now pip install libzt to get 1.3.4b0 on mac/linux (windows is being a pain but support will be added). Supported versions are 3.5, 3.6, 3.7, 3.8, and 3.9 on x86_64, and i686. arm is planned. The socket portion of the API is modeled after the Python low-level socket api. Currently ipv6 isn't supported but is planned to be included soon. Take note that any portion of that API that deals with file descriptors is irrelevant to libzt since it doesn't use OS facilities.

We now have a rudimentary build pipeline set up to emit wheels fairly easily so you can expect more regular updates going forward.

Example usage: examples/python
Language Bindings (swig generated stuff isn't committed yet due to sheer size): src/bindings/python

Thanks for your patience and suggestions, everyone! Please let me know if something is broken or not Pythonic enough.

P.S. These wheels are based on ZeroTier 1.4.6 but will be upgraded to 1.6.X soon.
P.S.S.: I'd take a look at src/bindings/python/sockets.py to get a feeling for what is implemented so far and what is not.

from libzt.

Another 3 months passes, I bother the devs again ;p

I probably ought to set up a script to remind me or something, but I figure I'm less of an asshole if I genuinely have to remember it and come back to it.

Is there an example of the provisional package someone could hack away at, if they found themselves with some free time?

from libzt.

New 1.3.4b1 packages are up. You can get it with pip3 install libzt There's still some stuff in the C API that isn't exposed but I'm working on that.

I also updated the .github/workflows/wheels.yml script so it should generate wrappers automatically making the previously-mentioned local build instructions unnecessary.

from libzt.

Great news. Looking forward ...

from libzt.

Bump

from libzt.

@joseph-henry - first of all thanks for all the good work! I was wondering when will the Linux distribution would be available?

Thank you!

from libzt.

Not really going to be testing this if it's only available on macOs, sadly. Perhaps provide a source distrobution to start with?

from libzt.

but will soon include other platforms.

Bump. Even a source distribution would be plenty, actually probably preferable if we want to be able to target things like arm/android.

from libzt.

it is only available for macOS at the moment but will soon include other platforms.

So it's been a year, any updates on this feature? I've had a specific hobby project in mine that could definitely use that feature.

from libzt.

Sorry to bump again, but any plans for this project? I also have a project that could really usea version of libzt in Python.

Thanks!

from libzt.

And another bump.

You can say "We're too busy to work on this". That's fine. Bus some kind of update would be nice.

https://github.com/zerotier/libzt/tree/master/packages/pypi now points to a dead link. Has this actually slid backwards?

I'll keep asking every 3 months or so.

from libzt.

A python version would vastly expand the number of users who need a paid account to administer more than 100 connections

from libzt.

from libzt.

It's definitely beyond me, I've been trying to learn the cppyy bindings generator but it's still pretty early days for that library.

When you get to the point where you need to be generating documentation, building wheels, and otherwise setting up packaging infrastructure, that's something that I could help out with. Here's hoping someone more experienced can jump in.

from libzt.

Maybe @minrk from pyzmq (the best python binding I've seen) could give some pointers.

from libzt.

@joseph-henry I would like to give it a try with SIP. Which version of the libzt should I use?

Could there be licensing problems ? https://www.riverbankcomputing.com/static/Docs/sip/introduction.html#license

from libzt.

I've had a fair bit of luck with cppyy despite my inexperience with bindings, it's definitely worth considering. It uses cmake to automatically generate a setup.py file.

from libzt.

I'm a bit late in my quarterly bothering-people.

It is very close to being finished

Yay!

from libzt.

I happened to run into this thread looking for a way to use libzt in python. Looked at swig documentation and I think you may want to look at this. https://rawgit.com/swig/swig/master/Doc/Manual/SWIGPlus.html#SWIGPlus_target_language_callbacks

It says you need director feature for it to work.

from libzt.

Last night I just happened to decide to play with libzt in Python while watching an Ada Lovelace documentary... Looks like I stumbled across this at just the right time (considering the release 4 days ago).

I'm having problems building the _libzt.so to test the changes I've made, the resulting _libzt.so I'm getting under Ubuntu 20.04 is:

ImportError: dynamic module does not define module export function (PyInit__libzt)

I had built it using "./build.sh host-python release", which seemed to build a _libzt.so but it produced that error. I even tried using github actions to build the wheels, and the result seems to be the same.

I've submitted #101 which I think significantly improves the recv() function, but I have no way to test it.

from libzt.

Awesome to see this.

from libzt.

I'm going to test it this weekend. @joseph-henry thanks for remembering the lurkers! You're the man!

from libzt.

For those following, I just submitted #104 which fixes some leaking None references, adds some more thread abilities, and fixes a few bugs. Hopefully without adding any. ;-) I'd also like some feedback on my making the docstrings more like the socket module with a function prototype.

from libzt.

I'm quite new to python so sorry for possiblly basic question, but how to build libzt for python on windows?
I've tired to install visual studio build tools 2019 (they work with python on windows) and to compile libzt on windows according to instructions, but I've got error:

C:\Users\graf0\code\libzt\ext\ZeroTierOne\node\SHA512.cpp(124,27): error C2039: 'min': is not a member of 'std' [C:\Users\graf0\code\libzt\cache\win-x64-host-release\zto_obj.vcxproj]

from libzt.