Git Product home page Git Product logo

pybind11_json's Introduction

Tests

pybind11_json

pybind11_json is an nlohmann::json to pybind11 bridge, it allows you to automatically convert nlohmann::json to py::object and the other way around. Simply include the header, and the automatic conversion will be enabled.

C++ API: Automatic conversion between nlohmann::json and pybind11 Python objects

#include "pybind11_json/pybind11_json.hpp"

#include "nlohmann/json.hpp"

#include "pybind11/pybind11.h"

namespace py = pybind11;
namespace nl = nlohmann;
using namespace pybind11::literals;

py::dict obj = py::dict("number"_a=1234, "hello"_a="world");

// Automatic py::dict->nl::json conversion
nl::json j = obj;

// Automatic nl::json->py::object conversion
py::object result1 = j;
// Automatic nl::json->py::dict conversion
py::dict result2 = j;

Making bindings

You can easily make bindings for C++ classes/functions that make use of nlohmann::json.

For example, making a binding for the following two functions is automatic, thanks to pybind11_json

void take_json(const nlohmann::json& json) {
    std::cout << "This function took an nlohmann::json instance as argument: " << s << std::endl;
}

nlohmann::json return_json() {
    nlohmann::json j = {{"value", 1}};

    std::cout << "This function returns an nlohmann::json instance: "  << j << std::endl;

    return j;
}

Bindings:

PYBIND11_MODULE(my_module, m) {
    m.doc() = "My awesome module";

    m.def("take_json", &take_json, "pass py::object to a C++ function that takes an nlohmann::json");
    m.def("return_json", &return_json, "return py::object from a C++ function that returns an nlohmann::json");
}

You can then use your functions Python side:

import my_module

my_module.take_json({"value": 2})
j = my_module.return_json()

print(j)

Example

You can find an example of simple Python bindings using pybind11_json here: https://github.com/martinRenou/xjson

Installation

Using conda

You can install pybind11_json using conda

conda install -c conda-forge pybind11 nlohmann_json pybind11_json

From sources

We encourage you to use conda for installing dependencies, but it is not a requirement for pybind11_json to work

conda install cmake nlohmann_json pybind11 -c conda-forge

Then you can install the sources

cmake -D CMAKE_INSTALL_PREFIX=your_conda_path
make install

Header only usage

Download the "pybind11_json.hpp" single file into your project, and install/download pybind11 and nlohmann_json or use as git submodule.

Run tests

You can compile and run tests locally doing

cmake -D CMAKE_INSTALL_PREFIX=$CONDA_PREFIX -D DOWNLOAD_GTEST=ON ..
make
./test/test_pybind11_json

Dependencies

pybind11_json depends on

pybind11_json nlohmann_json pybind11
master >=3.2.0,<4.0 >=2.2.4,<3.0
0.2.* >=3.2.0,<4.0 >=2.2.4,<3.0

License

We use a shared copyright model that enables all contributors to maintain the copyright on their contributions.

This software is licensed under the BSD-3-Clause license. See the LICENSE file for details.

pybind11_json's People

Contributors

anthony-celus avatar burgholzer avatar chronitis avatar gslotman avatar henryiii avatar johanmabille avatar martinrenou avatar mlund avatar noahamsel avatar pengwyn avatar qingfengxia avatar sylvaincorlay avatar t-kawai0922 avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

pybind11_json's Issues

Allow automatic cast from nlohmann::json to py::object in return type of functions

I have a class function that returns a nlohmann::json object that I would like to call directly in python.

Is there any way to use this function without having to explicitly do the conversion myself?

What I am currently doing is:

py::class_<myClass> (m, "myClass")
    .def (py::init<const char*> ())  
    .def ("getJSON", [] (const myClass& my_class) -> py::object {
        nl::json json_obj = my_class.getJSON ();
        py::object py_obj = json_obj; // convert to type already bound
        return py_obj;
    })
    ;

I am trying to achieve something like:

py::class_<myClass> (m, "myClass")
    .def (py::init<const char*> ())
    .def ("getJSON", &myClass::getJSON)
        ;

I tried to register a py::implicitly_convertible<py::object, nl::json> (); but it just works from python to C++, not the other way around.

I also tried defining a json binding class with:

py::class_<nl::json> (m, "json")
    .def (py::init<> ())
    .def (py::init<py::object> ())
     ;
py::implicitly_convertible<py::object, nl::json> ();

I also know that what would probably work is to return a py::object from myClass::getJSON as mentioned in #18. However, I would like to avoid myClass having to depend on a pybind11 implementation.

[question] Python + C++ full example?

Hi,

sorry for being lame but I can't put together a complete working example with pybind11_json.
Here is what I need to do:

1.) I have a native C++ class like
class Foo {
public:
int member1;
int member2;
};

2.) I have the corresponding pybind11 class_ mapping which defines a "Foo" class and does ".def_readwrite" on "member1" and "member2"

3.) Now I need to have to create something which automatically makes it possible that from Python I could:

foo = Foo()
foo.member1 = 1
foo.member2 = 2
jsonFoo = json(foo)
print(jsonFoo) // This would print '{ member1 : 1, member2 : 2 }'

// ???
foo2 = json('{ member1: 4, member2 : 5}')
print(foo2.member1) // this would print 4
print(foo2.member2) // this would print 5

Is it possible at all? Or do I have to create Nlohmann boilerplate code to be able to "Jsonify" my "Foo" class?
What is the full workflow here to be able to serialize back and forther PyBind11 classes in Python?
Thanks,

p.s. I have tried using the "implicitly_convertible" trick

pythonbind::class_nl::json(mod, "Json")
.def(py::init<>())
.def(py::initpy::object());
py::implicitly_convertible<py::object, nl::json>();

and then from Python:

foo = Foo()
jsonFoo = Json(foo)
json.dumps(jsonFoo)

but that gives me "to_json not implemented for this type of object: xxxx_pybind11.Foo object at XXXXX"
(which I suppose is not a surprise, as the C++ PyBind classes are not iterable objects, hence you need to write
for each and every Pybind class its own Json converter, I suppose?)

Redundant Move in from_json

In some paths within py::object from_json(const nl::json& j) the returned object is returned via std::move(obj);. This triggers a redundant-move warning in GCC 12.1, as the move prevents copy elision.

local variable 'obj' will be copied despite being returned by name

During my app compilation, I got a multiple repetition of 2 warnings:

/Users/studio/maialib/core/external/pybind11_json/pybind11_json.hpp:57:20: note: 
      call 'std::move' explicitly to avoid copying
            return obj;
                   ^~~
                   std::move(obj)
/Users/studio/maialib/core/external/pybind11_json/pybind11_json.hpp:66:20: warning: 
      local variable 'obj' will be copied despite being returned by name
      [-Wreturn-std-move]
            return obj;
                   ^~~

Please, could you fix it to clean the terminal output compilation?

My System:

  • Apple OS X Catalina 10.15.7
  • Xcode Command Tools 11.5
  • Pybind11 2.9.0
  • Using CMake with C++17 flags enable

Obs.: This 2 warnings appear on OS X system only. I got no warnings on Windows and Linux.

Infinity mishandled in nl::json => py::object conversion

When converting a nl::json object that contains the infinity to Python, infinity gets interpreted as an integer instead of a float. If I bind the following function

#include <cmath>

nl::json test() {
    return nl::json(INFINITY);
}

it returns -9223372036854775808 on the Python side instead of math.inf.

Move to pybind organization?

Hi Martin,

several projects that integrate pybind11 with other dependencies (abseil, basel, protobuf) are slated to move into the pybind organization (https://github.com/pybind/). I thought that this would also make sense for this project, which is both useful and popular. It's completely up to you, of course. If you would like to move, please let me know and I'll invite you to the organization.

Best,
Wenzel

Error: nlohmann::json as input argument

As you can see in this topic #18, I can return a nlohmann::json from a Python function as a pybind11::object type.

Now, I'm trying to send a JSON as a input argument to my_func(), modify it and return it back in my Python script.

Example: C++ side

pybind11::object my_func(nlohmann::json& a)
{
    a["new"] = "new string";

    return a;
}

It compiles Ok, but when I tried to call this function inside my Python module, I got this error below:

Error: Python side

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: test(): incompatible function arguments. The following argument types are supported:
    1. (self: my_lib.MusicXML, arg0: nlohmann::basic_json<std::map, std::vector, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long, unsigned long, double, std::allocator, nlohmann::adl_serializer>) -> object

What I have to do to fix it?

Thank you,

What about enumerations?

Hi,

I must say I really like the idea of this utility to ease my parameters structures usage in Python. I encountered an issue with bound enumerations. I use enumerations quite a lot in C++ mainly in factories. I can easily have a nlohmann converters for them, I can easily bind them, but I'm lost when trying to have the two functionalities working together.

It's quite logic since the nlohmann json bindings expect classic types and I guess enumerations fall into a special structures. I tried to get how I could handle those errors but I struggle finding how I could do that.

module.cpp

#include 

#include "nlohmann/json.hpp"
// This is where I copied pybind11_json.hpp
#include "converters/json.h"

#define MODULE_NAME footest

namespace testing_pybind11_json {

enum class Foo : uint32_t { ZERO, ONE, TWO, THREE };
NLOHMANN_JSON_SERIALIZE_ENUM(Foo,
{{Foo::ZERO, "ZERO"},
{Foo::ONE, "ONE"},
{Foo::TWO, "TWO"},
{Foo::THREE, "THREE"}})

void print_json(const nlohmann::json &j) {
std::cout << std::endl << std::setw(4) << j.dump() << std::endl;
}

void print_dict(const py::dict &d) {
nlohmann::json j;
j = d;
std::cout << std::endl << std::setw(4) << j.dump() << std::endl;
}

} // namespace testing_pybind11_json

PYBIND11_MODULE(MODULE_NAME, m) {
Py_Initialize();
PyEval_InitThreads();

py::enum_<testing_pybind11_json::Foo>(m, "Foo")
    .value("ZERO", testing_pybind11_json::Foo::ZERO)
    .value("ONE", testing_pybind11_json::Foo::ONE)
    .value("TWO", testing_pybind11_json::Foo::TWO)
    .value("THREE", testing_pybind11_json::Foo::THREE);
m.def("print_json", &testing_pybind11_json::print_json);
m.def("print_dict", &testing_pybind11_json::print_dict);

}

footest_pytest.cpp

import footest
import pytest

@pytest.mark.module("binding.utils")
def pytestcase_testing_pybind11_print_json():
footest.print_json({'a': 1, 'b': 2})
footest.print_json({'a': 1, 'b': 2, 'c': footest.Foo.ONE})

@pytest.mark.module("binding.utils")
def pytestcase_testing_pybind11_print_dict():
footest.print_dict({'a': 1, 'b': 2})
footest.print_dict({'a': 1, 'b': 2, 'c': footest.Foo.ONE})

It returns me different type of error depending on the function I'm using.

With print_json: it tries to use directly type_caster but seems to fall in a strange case.

    @pytest.mark.module("binding.utils")
    def pytestcase_testing_pybind11_print_json():
        mci.print_json({'a': 1, 'b': 2})
>       mci.print_json({'a': 1, 'b': 2, 'c': mci.Foo.ONE})
E       TypeError: print_json(): incompatible function arguments. The following argument types are supported:
E           1. (arg0: json) -> None
E       
E       Invoked with: {'a': 1, 'b': 2, 'c': <Foo.ONE: 1>}

sdk/modules/computational_imaging/python/tests/bindings/calibration_pytest.py:148: TypeError

With print_dict: I avoid using type_caster but calls directly the to_json function which goes well until the incompatible enum.

    @pytest.mark.module("binding.utils")
    def pytestcase_testing_pybind11_print_dict():
        mci.print_dict({'a': 1, 'b': 2})
>       mci.print_dict({'a': 1, 'b': 2, 'c': mci.Foo.ONE})
E       RuntimeError: to_json not implemented for this type of object: <Foo.ONE: 1>

sdk/modules/computational_imaging/python/tests/bindings/calibration_pytest.py:154: RuntimeError

I don't know if it's even possible to catch the enumeration case and call the to_json / from_json of the underlying enumeration.

CMake deprecation warning

On newer versions of cmake I get this warning:

CMake Deprecation Warning at CMakeLists.txt:9 (cmake_minimum_required):
  Compatibility with CMake < 3.5 will be removed from a future version of
  CMake.

  Update the VERSION argument <min> value or use a ...<max> suffix to tell
  CMake that the project does not need compatibility with older versions.

Build fails with CMake version < 3.14

This error is caused by using the option ARCH_INDEPENDENT, which was only added in CMake version 3.14 (see the release notes for that version here). Building the library fails with older versions of CMake. For example, with CMake 3.11 I get:

CMake Error at /usr/share/cmake/Modules/WriteBasicConfigVersionFile.cmake:30 (message):
  Unknown keywords given to WRITE_BASIC_CONFIG_VERSION_FILE():
  "ARCH_INDEPENDENT"
Call Stack (most recent call first):
  /usr/share/cmake/Modules/CMakePackageConfigHelpers.cmake:209 (write_basic_config_version_file)
  CMakeLists.txt:116 (write_basic_package_version_file)

This recent commit introduced the use of ARCH_INDEPENDENT, but did not change the minimum required CMake version.

C++ returning nlohmann::json to Python. Is it possible?

I'm building a C++ Pybind11 module to use my C++ functions inside some Python scripts.

I would like to know if is possible to write a C++ function that returns a nlohmann::json type and get this output result in the Python side as JSON...or dict...??

Example: C++ side

nlohmann::json my_func()
{
    nlohmann::json j;

    j["key_a"] = 0;
    j["key_b"] = 1;
    j["tree"]["first"] = "node 1";

    return j;
}

Python side:

import my_module

data = my_module.my_func()

How can I do that?

Bind nlohmann::json::array_t is possible?

As you can see here #18 I understood that I can bind nlohmann::json as a py::object type.

I would like to know if is possible to bind the nlohmann::json::array_t type to some native Python type.

I mean, which Python type should I use to be able to return a nlohmann array of objects from C++ to Python side.

Cannot include dependencies with add_directories

This is the same issue that #15 attempted to solve. In CMake, we don't want to search for the dependencies (pybind11 and nlohmann_json) using find_package if the user is adding them using add_directories. In this case, the find_package command would fail because it would force CMake to look for these dependencies in the wrong places. So #15 added the following guards: only call find_package(nlohmann_json) if the target nlohmann_json doesn't already exist, and only call find_package(pybind11) if the target pybind11 doesn't already exist:

if (NOT TARGET pybind11)
find_package(pybind11 ${pybind11_REQUIRED_VERSION} REQUIRED)
endif()
if (NOT TARGET nlohmann_json)
find_package(nlohmann_json ${nlohmann_json_REQUIRED_VERSION} REQUIRED)
endif()

Unfortunately, the first guard does not work. The problem is that the target pybind11 does not exist even if pybind11 has been properly added. I was using add_directories to add pybind11 to my project and getting the following error from CMake:

CMake Error at build/_deps/pybind11_json-src/CMakeLists.txt:26 (find_package):
  By not providing "Findpybind11.cmake" in CMAKE_MODULE_PATH this project has
  asked CMake to find a package configuration file provided by "pybind11",
  but CMake did not find one.

  Could not find a package configuration file provided by "pybind11"
  (requested version 2.2.4) with any of the following names:

    pybind11Config.cmake
    pybind11-config.cmake

  Add the installation prefix of "pybind11" to CMAKE_PREFIX_PATH or set
  "pybind11_DIR" to a directory containing one of the above files.  If
  "pybind11" provides a separate development package or SDK, be sure it has
  been installed.


-- Configuring incomplete, errors occurred!

This is because pybind11_json/CMakeLists.txt:26 is calling find_package(pybind11) when it shouldn't be, which happens because the target pybind11 does not exist.

AppVeyor badge url

@wjakob last time I ping you, I promise!

The appveyor badge url is wrong on the README, but I don't seem to have the authorization for getting access to the pybind/pybind11_json project on AppVeyor (because I'm not part of the organization). So I am not able to find this badge url.

Would you mind sharing the url here or fixing the README?

How can I debug?

I wrote a Pybind11 module and just one function is not working only when it is called in the Python environment (C++ side works great).

When I call it in the C++ environment, this function takes 30 seconds to return the result and works great.

This function uses 2 external libraries:

  • nlohmann JSON
  • pugi XML parser

When I call it in the Python side, I got a error inside the pugi library.

python3: /src/pugixml.cpp:12202: const pugi::xpath_node& pugi::xpath_node_set::operator[](size_t) const: Assertion `index < size()' failed.

Aborted (core dumped)

How can I debug (line by line) my C++ code in the Python environment?

pybind11_json with c++ class

I am trying to use the pybind11_json with a c++ class.

When I use the example provided in this git: void take_json(const nlohmann::json& json) , it works fine.
But I am trying to use it as a method of my class:

 py::class_<myClass>(module, "myClass", py::buffer_protocol()) 
.def("fromJson", &myClass::fromJson);

I am getting the following error when I call it from python:
obj.fromJson({"a": 1234})

TypeError: fromJson(): incompatible function arguments. The following argument types are supported:

    1. (self: myModule.myClass, arg0: json) -> None
Invoked with: {'a': 1234}

Do I need to do anything else to make it work with methods of a c++ class?

Issues using the lib

I'm having a tough time using this lib. I've copied pybind_json.hpp into my include dir as I don't have access to conda.

Then my main file is:

#include <pybind_json.hpp>

int main()
{
    nlohmann::json j;
    j["a"] = 1;
    j["b"] = 2;

    py::dict d;
    d["a"] = 1;
    d["b"] = 2;

    py::dict d2 = j;
    nlohmann::json j2 = d;
}

The last two lines give compilation errors of:
"no suitable user-defined conversion from "nlohmann::json" to "pybind11::dict" exists"
"no suitable user-defined conversion from "pybind11::dict" to "nlohmann::json" exists"

respectively. So I am unable to do any conversion. I must be missing something simple. I am able to compile the code with the last 2 lines commented out so I know the nlohmann and pybind11 libs are properly being used.

Support for binary arrays

NLohmann JSON version 3.10.4 has binary arrays: https://json.nlohmann.me/api/basic_json/binary/
If such array is encountered in JSON object, pyjson::from_json() crashes with stack overflow because it falls to "else // Object" clause and then the for() loop tries to iterate over non-existing values, since binary array is not an object.

Proposed fix:

  1. Add include file:
    #include "pybind11/numpy.h"

  2. In pyjson::from_json() add:
    ...
    else if (j.is_binary())
    {
    const auto &bin = j.get_binary();
    return py::array( bin.size(), bin.data() );
    }
    else // Object
    {
    ...

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.