ramonhagenaars / jsons Goto Github PK
View Code? Open in Web Editor NEW๐ A Python lib for (de)serializing Python objects to/from JSON
Home Page: https://jsons.readthedocs.io
License: MIT License
๐ A Python lib for (de)serializing Python objects to/from JSON
Home Page: https://jsons.readthedocs.io
License: MIT License
Consider example:
class A(NamedTuple):
a: int
b: str
>>> jsons.dumps(A(0, ""))
'[0, ""]'
>>> jsons.loads('[0, ""]', A)
jsons.exceptions.UnfulfilledArgumentError: No value present in [0, ''] for argument "a"
>>> jsons.loads('[1, ""]', A)
jsons.exceptions.UnfulfilledArgumentError: No value present in [1, ''] for argument "b"
I'm a beginner with Python. So I'm not sure if I use jsons wrong.
The first exemple below works well.
import jsons
from dataclasses import dataclass
@dataclass
class ChatUser:
uid: str
name: str
dumped = jsons.dumps(ChatUser('012451', 'Casimir'), strip_privates=True, strip_microseconds=True, verbose=True)
print(dumped)
instance = jsons.loads(dumped, ChatUser)
print(instance)
Both following examples fail with DeserializationError. Can someone help?
import jsons
from dataclasses import dataclass
import datetime
@dataclass
class ChatUser:
uid: datetime
name: str
dumped = jsons.dumps(ChatUser(datetime.datetime.now(), 'Casimir'), strip_privates=True, strip_microseconds=True, verbose=True)
print(dumped)
instance = jsons.loads(dumped, ChatUser)
print(instance)`
import jsons
from dataclasses import dataclass
from typing import List, NewType
Uid = NewType("Uid", str)
@dataclass
class ChatUser:
uid: Uid
name: str
dumped = jsons.dumps(ChatUser(Uid('012451'), 'Casimir'), strip_privates=True, strip_microseconds=True, verbose=True)
print(dumped)
instance = jsons.loads(dumped, ChatUser)
print(instance)
Hi there,
Loading a 1Go JSON file may take a while. Using the standard JSON API with the same JSON file may take approximatively 35 seconds.
with jsons: time~ 20min
with open(json_path, "r") as json_file:
content = json_file.readlines()
json_data = jsons.loads(content[0], CustomObject)
The readlines
function take 2 or 3 seconds.
with json: time~ 35s
with open(json_path, "r") as json_file:
content = json.load(json_file)
json_data = CustomObject(**content)
What I'm doing wrong ?
Simply transform as follows:
Serializer: uuid_str = str(uuid)
Deserializer: uuid = uuid.UUID(uuid_str)
I am willing to implement if there is a desire to have this in the library.
Hello!
Does the author consider adding support for the TypedDict
from mypy_extensions
in the future?
TypedDict is extremely helpful for type hinting and code completion and its syntax is very concise.
A detailed description can be found here. It is also likely that it will be found in the standard library typing
module in the future versions of Python: python/mypy#5288
As a downside, this would introduce a dependency - mypy_extensions
. A solution to that would be making the dependency optional the same way for example Flask
has an optional dependency SimpleJSON
and only uses it if this package is installed.
They are too useful to be left out. :)
Trying to do:
import uuid
import jsons
def custom_uuid_serializer(obj: uuid.UUID, **kwargs) -> str:
return str(obj)
def custom_uuid_deserializer(obj: str, cls, **kwargs) -> uuid.UUID:
return uuid.UUID(obj)
jsons.set_serializer(custom_uuid_serializer, uuid.UUID) # Failed...
jsons.set_deserializer(custom_uuid_deserializer, uuid.UUID) # Failed...
Exception:
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "json_test.py", line 36, in <module>
data = jsons.dump(hearthbeat) # It will serialize uuid to its bytes
File "/Users/ahmetkucuk/anaconda3/envs/jsons/lib/python3.7/site-packages/jsons/_dump_impl.py", line 52, in dump
raise SerializationError(str(err))
jsons.exceptions.SerializationError: 'uuid.UUID'
I think the issue is here: https://github.com/ramonhagenaars/jsons/blob/master/jsons/_lizers_impl.py#L139
result = lizers[pname.lower()]
solves the issue.
Python 3.8 introduced the TypedDict
:
When trying to deserialize NamedTuple I got following error:
File "/Users/cypreess/.virtualenvs/.../lib/python3.6/site-packages/jsons/deserializers.py", line 83, in default_tuple_deserializer
tuple_types = getattr(cls, '__tuple_params__', cls.__args__)
AttributeError: type object 'MyNamedTuple' has no attribute '__args__'
This is different error than related issue which was solved: #11
I can't easily find in the docs how I can specify some attributes of my JsonSerializable class that I don't want to be serialized to JSON.
class MyClass(JsonSerializable):
def __init__(self):
self.a = 1
self.b = 2
self.log = logging.getLogger(__name__)
In the above case, how can I make self.log get skipped, without having to modify the jsons.dumps
call? I want the control of what gets included in the JSON to be with the class, not the place where dump(s/b)
is called. So I don't want to have to make them private (with '_') and add strip_privates
to dumps
. First of all, that would require modifying all dumps calls throughout my code, and secondly, it forces me to change my datamodel, which has consequences elsewhere in the app.
Aside from this example where self.log clearly isn't serializable, and also wouldn't make sense to serialize, there may be other attributes that I want to skip as well. Is this easily possible with jsons? If so, can we add a FAQ question about it? (I'd be happy to contribute it)
Currently, only datetime.datetime
is supported. All types from the datetime
module should be supported. That includes: date
, time
, timedelta
, timezone
, tzinfo
.
You can create a forked JsonSerializable
and attach attr_getters
to this fork with a single .with_load()
call, and then register deserializers. But those deserializers are ignored when using .from_json()
on the subclass.
Demo:
import dataclasses, datetime, jsons
defaults = {"ham": lambda: "spam"}
forked = jsons.JsonSerializable.with_load(attr_getters=defaults, fork=True)
forked.set_deserializer(
lambda td, cls, **kwargs: datetime.timedelta(seconds=td),
datetime.timedelta
)
@dataclasses.dataclass
class Foo(forked):
bar: datetime.timedelta
ham: str
Foo.from_json({"bar": 42})
The above demo leads to a traceback:
Traceback (most recent call last):
File ".../site-packages/jsons/_load_impl.py", line 85, in load
return deserializer(json_obj, cls, **kwargs_)
File ".../site-packages/jsons/deserializers/default_object.py", line 38, in default_object_deserializer
constructor_args = _get_constructor_args(obj, cls, **kwargs)
File ".../site-packages/jsons/deserializers/default_object.py", line 69, in _get_constructor_args
constructor_args_in_obj = {key: value for key, value in args_gen if key}
File ".../site-packages/jsons/deserializers/default_object.py", line 69, in <dictcomp>
constructor_args_in_obj = {key: value for key, value in args_gen if key}
File ".../site-packages/jsons/deserializers/default_object.py", line 68, in <genexpr>
if sig_key != 'self')
File ".../site-packages/jsons/deserializers/default_object.py", line 84, in _get_value_for_attr
if obj and sig_key in obj:
TypeError: argument of type 'int' is not iterable
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File ".../site-packages/jsons/classes/json_serializable.py", line 123, in from_json
return cls.load(json_obj, **kwargs)
File ".../site-packages/jsons/classes/json_serializable.py", line 93, in _wrapper
return load(inst, cls_, **{**kwargs_, **kwargs})
File ".../site-packages/jsons/_load_impl.py", line 85, in load
return deserializer(json_obj, cls, **kwargs_)
File ".../site-packages/jsons/deserializers/default_object.py", line 38, in default_object_deserializer
constructor_args = _get_constructor_args(obj, cls, **kwargs)
File ".../site-packages/jsons/deserializers/default_object.py", line 69, in _get_constructor_args
constructor_args_in_obj = {key: value for key, value in args_gen if key}
File ".../site-packages/jsons/deserializers/default_object.py", line 69, in <dictcomp>
constructor_args_in_obj = {key: value for key, value in args_gen if key}
File ".../site-packages/jsons/deserializers/default_object.py", line 68, in <genexpr>
if sig_key != 'self')
File ".../site-packages/jsons/deserializers/default_object.py", line 87, in _get_value_for_attr
meta_hints, **kwargs)
File ".../site-packages/jsons/deserializers/default_object.py", line 122, in _get_value_from_obj
value = load(obj[sig_key], cls_, meta_hints=new_hints, **kwargs)
File ".../site-packages/jsons/_load_impl.py", line 89, in load
raise DeserializationError(str(err), json_obj, cls)
jsons.exceptions.DeserializationError: argument of type 'int' is not iterable
because the replacement implementation for .load()
generated by with_load()
does not pass in the forked instance:
jsons/jsons/classes/json_serializable.py
Lines 91 to 95 in 03b5ff9
like the original .load()
implementation does:
jsons/jsons/classes/json_serializable.py
Lines 134 to 143 in 03b5ff9
We can work around this by creating a fork first, then using forked.with_default(..., fork_inst=forked)
, but is less than ideal.
Hi~, I using udatetime and jsons, the udatetime return UTC timezone is +00:00
udatetime.utcnow()
datetime.datetime(2019, 4, 12, 8, 31, 12, 471970, tzinfo=+00:00)
So, is allow to add +00:00 to _datetime_offset_str in _datetime_impl.py ?
if tzone.tzname(None) not in ('UTC', 'UTC+00:00'):
edit to
if tzone.tzname(None) not in ('UTC', 'UTC+00:00', '+00:00'):
Sometimes it might be useful to be able to specify attributes that are not to be dumped.
jsons.dump(obj, strip_attr='attr1')
# Or strip multiple:
jsons.dump(obj, strip_attr=('attr1', 'attr2'))
Hi,
I have just started using your library and it works great, but I have encountered this weird error. I have JSON string with correct datetime and it is being deserialized correctly if type hint is just datetime but if I want to support null
in JSON (None
in Python) I use typing.Optional[datetime]
and it gives me this error: jsons.exceptions.DeserializationError: Invalid type: "datetime.datetime", only arguments of the following types are allowed: str, int, float, bool, list, tuple, set, dict, NoneType
.
Minimal code example:
import jsons
import typing
import dataclasses
import datetime
@dataclasses.dataclass
class WithDateOK:
created: datetime.datetime
@dataclasses.dataclass
class WithDateBad:
created: typing.Optional[datetime.datetime]
withDateStr = '''{"created":"2019-12-19T11:40:48Z"}'''
withDateDict = jsons.loads(withDateStr)
print(withDateDict)
withDateObj = jsons.load(withDateDict, WithDateOK)
withDateObj = jsons.load(withDateDict, WithDateBad)
print(type(withDateObj))
print(withDateObj)
Complete Traceback:
Traceback (most recent call last):
File "/home/vanko/PycharmProjects/playground/jsonpickle_playing.py", line 641, in <module>
withDateObj = jsonsLib.load(withDateDict, WithDate)
File "/home/vanko/PycharmProjects/playground/venv/lib/python3.7/site-packages/jsons/_load_impl.py", line 98, in load
return _do_load(json_obj, deserializer, cls, initial, **kwargs_)
File "/home/vanko/PycharmProjects/playground/venv/lib/python3.7/site-packages/jsons/_load_impl.py", line 110, in _do_load
result = deserializer(json_obj, cls, **kwargs)
File "/home/vanko/PycharmProjects/playground/venv/lib/python3.7/site-packages/jsons/deserializers/default_object.py", line 39, in default_object_deserializer
constructor_args = _get_constructor_args(obj, cls, **kwargs)
File "/home/vanko/PycharmProjects/playground/venv/lib/python3.7/site-packages/jsons/deserializers/default_object.py", line 70, in _get_constructor_args
**kwargs)
File "/home/vanko/PycharmProjects/playground/venv/lib/python3.7/site-packages/jsons/deserializers/default_object.py", line 94, in _get_value_for_attr
meta_hints, **kwargs)
File "/home/vanko/PycharmProjects/playground/venv/lib/python3.7/site-packages/jsons/deserializers/default_object.py", line 134, in _get_value_from_obj
value = load(obj[sig_key], cls_, meta_hints=new_hints, **kwargs)
File "/home/vanko/PycharmProjects/playground/venv/lib/python3.7/site-packages/jsons/_load_impl.py", line 82, in load
json_obj, cls, fork_inst, kwargs.get('_inferred_cls', False))
File "/home/vanko/PycharmProjects/playground/venv/lib/python3.7/site-packages/jsons/_load_impl.py", line 197, in _check_and_get_cls_and_meta_hints
raise DeserializationError(msg, json_obj, cls)
jsons.exceptions.DeserializationError: Invalid type: "datetime.datetime", only arguments of the following types are allowed: str, int, float, bool, list, tuple, set, dict, NoneType
If I change datetime string to null
, it works fine and created is None
as expected.
Here is live demo example demonstrating the error. Is this behaviour correct or is this a bug? Thanks
Regards,
Adam
When running following code
from typing import Tuple
import jsons
from dataclasses import dataclass
@dataclass
class A:
tup: Tuple[str, ...]
a = A(('abc', 'qwe', '123'))
a_ser = jsons.dump(a)
a_deser = jsons.load(a_ser, A)
I get exception AttributeError: 'ellipsis' object has no attribute '__origin__'
on the last line.
I can simply use List[str]
as type, but it would be nice to be able to deserialize into tuple of variable length.
When I try to deserialize with extra parameters, the library creates a new variable in the object. I expect it to not create any new variables and not throw any exceptions.
>>> from dataclasses import dataclass
>>> import jsons
>>>
>>> @dataclass
... class A:
... x: int
...
>>> a = jsons.load({'x': 1, 'y': 'abc'}, A)
>>> a.y
'abc'
>>> import dataclasses
>>> dataclasses.asdict(a)
{'x': 1}
I guess this happens in _set_remaining_attrs
. I suggest making this call optional.
OR
When we serialize it again, it should ignore these fields.
The following construct causes a problem when serializing:
@dataclass
class Narcissus:
@property
def mirror(self):
return self
Another example:
@dataclass
class A:
b: object
@dataclass
class B:
c: object
@dataclass
class C:
a: object
In case of recursion problems during serialization, the attributes that cause the problem should be omitted and a warning should be triggered.
Issue#46 may be related.
As in title, strip_privates
gets "over-written" by strip_properties
. Example:
https://gist.github.com/haluzpav/628477bdeee8e33ed40e07c2b786c2dd
Kinda related to previous issue #3.
Similarly to #18 I start being biased towards thinking that because of "readability counts" NamedTuples should be expressed like to dicts. The whole point of naming tuple positions is because of poor readability of tuples. Then we totally drop it on serialization level when we already decided we go for readability.
set_deserializer()
accepts either a single type or a sequence of types. While it's fine to register a generic type that holds a forward reference (e.g. a string value whose loading is deferred to later, useful when model
would otherwise require a circular import), you can't register just a forward reference.
That's because a forward reference is a string, and strings are sequences. Can an exception be made here? A simple if isinstance(cls, Sequence) and not isinstance(cls, str):
should do.
Say I have a class inheritance structure, and I want to generically be able to dump/load with the correct class types in the inheritance structure.
@dataclass
class A(object):
foo: int
@dataclass
class B(A):
bar: int
Assume that when loading, I do now know which class type it was dumped from. I would have to analyse the content to determine which class to load as, and then pass this to the cls
property of jsons.load
method.
Would it make sense to implement an optional flag when dumping to a dictionary that contains information of what class type it was dumped as, that can then be used for loading?
More clear errors should be raised by jsons
upon problems during serialization. For instance, it should be clear which attributes cause problems when dumping an object.
Related: Issue#46
Given the following example Enum type:
from enum import Enum
class TestEnum(Enum):
KEY = 'value`
jsons.dumps(TestEnum.KEY)
will result in "KEY"
(Deserialization works correctly).
However, this is behavior is a little bit irrational in regards to what enum is (used to be in C :) ). Enums classically are used to map some "codenames" for binary values (like numbers 0, 1, 2, 3). Therefore the value of enum should be binary representation generally used in serialization and keys should be something assigned for it on the high level (on runtime/ in the codebase).
Update: just for a reference standard lib json module does not serialize Enum's at all.
In v1.1.0
I am having trouble serializing an object with optional attributes.
import jsons
from dataclasses import dataclass
from typing import List, Optional
@dataclass
class Color:
name: str
hex: Optional[str]
@dataclass
class Swatch:
name: str
colors: List[Color]
blue = Color('blue', '#0000ff')
green = Color('green', '#00ff00')
swatch = Swatch('swatch 1', [blue, green])
print(swatch)
print(jsons.dumps(swatch))
Result
Swatch(name='swatch 1', colors=[Color(name='blue', hex='hex'), Color(name='green', hex='hex')])
/site-packages/jsons/_common_impl.py:42: UserWarning: Failed to dump attribute "hex" of object of type "__main__.Color". Reason: 'NoneType' object is not callable. Ignoring the attribute. You can suppress warnings like this using jsons.suppress_warnings().
warnings.warn(msg_, *args, **kwargs)
{"colors": [{}, {}], "name": "swatch 1"}
Result when hex
is not optional
Swatch(name='swatch 1', colors=[Color(name='blue', hex='#0000ff'), Color(name='green', hex='#00ff00')])
{"colors": [{"hex": "#0000ff", "name": "blue"}, {"hex": "#00ff00", "name": "green"}], "name": "swatch 1"}
Example:
output = jsons.load(42, str)
Current output:
42
Desired output:
'42'
Currently, there is no check whatsoever as jsons
gladly loads a primitive of some type into an attribute of another primitive type. This can lead to undesired behaviour.
Another example:
output = jsons.load('fortytwo', int)
Current output:
'fortytwo'
Desired output: a DeserializationError
is raised.
When a class has an invalid type hint, deserializing into that class may cause a DeserializationError
that does not help solving the issue. The error should be more descriptive.
Example:
import datetime # <-- Note that the module is imported, not the class.
from dataclasses import dataclass
import jsons
@dataclass
class User:
name: str
birthday: datetime
dumped = { 'name': 'Albert', 'birthday': '1879-03-14T11:30:00+01:00' }
jsons.load(dumped, User)
This causes:
Traceback (most recent call last):
File "C:\Projects\Personal\jsons\jsons\_main_impl.py", line 128, in load
return deserializer(json_obj, cls, **kwargs_)
File "C:\Projects\Personal\jsons\jsons\deserializers\default_object.py", line 31, in default_object_deserializer
constructor_args = _get_constructor_args(obj, cls, **kwargs)
File "C:\Projects\Personal\jsons\jsons\deserializers\default_object.py", line 58, in _get_constructor_args
constructor_args_in_obj = {key: value for key, value in args_gen if key}
File "C:\Projects\Personal\jsons\jsons\deserializers\default_object.py", line 58, in <dictcomp>
constructor_args_in_obj = {key: value for key, value in args_gen if key}
File "C:\Projects\Personal\jsons\jsons\deserializers\default_object.py", line 57, in <genexpr>
in signature_parameters.items() if sig_key != 'self')
File "C:\Projects\Personal\jsons\jsons\deserializers\default_object.py", line 75, in _get_value_for_attr
meta_hints, **kwargs)
File "C:\Projects\Personal\jsons\jsons\deserializers\default_object.py", line 109, in _get_value_from_obj
value = load(obj[sig_key], arg_cls, meta_hints=new_hints, **kwargs)
File "C:\Projects\Personal\jsons\jsons\_main_impl.py", line 119, in load
deserializer = _get_deserializer(cls, fork_inst)
File "C:\Projects\Personal\jsons\jsons\_main_impl.py", line 145, in _get_deserializer
fork_inst._classes_deserializers, fork_inst)
File "C:\Projects\Personal\jsons\jsons\_main_impl.py", line 153, in _get_lizer
cls_name = get_class_name(cls, str.lower, fork_inst=fork_inst)
File "C:\Projects\Personal\jsons\jsons\_common_impl.py", line 51, in get_class_name
module = _get_module(cls)
File "C:\Projects\Personal\jsons\jsons\_common_impl.py", line 104, in _get_module
module = cls.__module__
AttributeError: module 'datetime' has no attribute '__module__'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:/Projects/Personal/jsons/effe.py", line 14, in <module>
jsons.load(dumped, User)
File "C:\Projects\Personal\jsons\jsons\_main_impl.py", line 132, in load
raise DeserializationError(str(err), json_obj, cls)
jsons.exceptions.DeserializationError: module 'datetime' has no attribute '__module__'
Hi!
>>> jsons.load({'1': 2}, Dict[str, int])
Traceback (most recent call last):
File "/home/glen/prywatne/traktpy/venv/lib/python3.7/site-packages/jsons/_main_impl.py", line 132, in load
return deserializer(json_obj, cls, **kwargs_)
TypeError: 'NoneType' object is not callable
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/glen/prywatne/traktpy/venv/lib/python3.7/site-packages/jsons/_main_impl.py", line 136, in load
raise DeserializationError(str(err), json_obj, cls)
jsons.exceptions.DeserializationError: 'NoneType' object is not callable
>>>
Self-explanatory :)
v0.8.5
Hello!
from __future__ import annotations
literally destroys the library.
Quick example:
foo.py
from __future__ import annotations
from dataclasses import dataclass
@dataclass
class Spam:
eggs: int
@dataclass
class Baz:
qux: int
spam: Spam
bar.py
from typing import List
import jsons
from foo import Baz
data = [{"qux": 0, "spam": {"eggs": 1}}]
x = jsons.load(data, List[Baz])
Expected result: [Baz(qux=0, spam=Spam(eggs=1))]
Result:
>>> import bar
Traceback (most recent call last):
File "/home/glen/traktpy/venv/lib/python3.7/site-packages/jsons/_main_impl.py", line 146, in load
return deserializer(json_obj, cls, **kwargs_)
TypeError: 'NoneType' object is not callable
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
...
File "/home/glen/traktpy/venv/lib/python3.7/site-packages/jsons/_main_impl.py", line 150, in load
raise DeserializationError(str(err), json_obj, cls)
jsons.exceptions.DeserializationError: 'NoneType' object is not callable
>>>
If you remove from __future__ import annotations
from foo.py
everything works as expected.
https://www.python.org/dev/peps/pep-0563/
from __future__ import annotations
stops the interpreter from executing type annotations, they are stored as str instead. Types may be revealed during runtime using typing.get_type_hints()
. This is going to be the default behavior in python4.
Related to this feature request, I wanted to try it out, but I can not mange to re-construct the objects given the metadata structure. Check out this script:
import jsons
from dataclasses import dataclass
@dataclass
class BarBase():
pass
@dataclass
class BarA(BarBase):
a: int
@dataclass
class BarB(BarBase):
b: int
@dataclass
class Foo():
bar: BarBase
if __name__ == "__main__":
f = Foo(bar=BarA(a=5))
d = jsons.dump(f, verbose=True)
o = jsons.load(d)
print("original:", f)
print("dump:", d)
print("reconstructed:", o)
It gives the following output (Python 3.7.2 and jsons==0.8.0
):
original: Foo(bar=BarA(a=5))
dump: {'bar': {'a': 5}, '-meta': {'classes': {'/bar': '__main__.BarA', '/': '__main__.Foo'}, 'dump_time': '2019-03-20T12:55:38Z'}}
reconstructed: Foo(bar=BarBase())
It appears that the serialization is done properly, but that the re-construction does not take the classes into account?
Consider the following examples:
>>> class T(NamedTuple):
o: Optional[str]
>>> jsons.load({"o": None}, T)
Traceback (most recent call last):
File "<input>", line 1, in <module>
File "/usr/local/lib/python3.7/site-packages/jsons/_load_impl.py", line 85, in load
return deserializer(json_obj, cls, **kwargs_)
File "/usr/local/lib/python3.7/site-packages/jsons/deserializers/default_tuple.py", line 19, in default_tuple_deserializer
return default_namedtuple_deserializer(obj, cls, **kwargs)
File "/usr/local/lib/python3.7/site-packages/jsons/deserializers/default_tuple.py", line 51, in default_namedtuple_deserializer
raise UnfulfilledArgumentError(msg, field_name, obj, cls)
jsons.exceptions.UnfulfilledArgumentError: No value present in {'o': None} for argument "o"
but also:
>>> class T(NamedTuple):
o: Any
>>> jsons.load({"o": None}, T)
Traceback (most recent call last):
File "<input>", line 1, in <module>
File "/usr/local/lib/python3.7/site-packages/jsons/_load_impl.py", line 85, in load
return deserializer(json_obj, cls, **kwargs_)
File "/usr/local/lib/python3.7/site-packages/jsons/deserializers/default_tuple.py", line 19, in default_tuple_deserializer
return default_namedtuple_deserializer(obj, cls, **kwargs)
File "/usr/local/lib/python3.7/site-packages/jsons/deserializers/default_tuple.py", line 51, in default_namedtuple_deserializer
raise UnfulfilledArgumentError(msg, field_name, obj, cls)
jsons.exceptions.UnfulfilledArgumentError: No value present in {'o': None} for argument "o"
I'd like to throw in the idea that it should be possible to have the option of omitting a field from the dump, if its value is None
. Intended use:
@dataclass
class Foo():
foo: int
bar: bool = None
jsons.dump(Foo(1)) #output: {"foo": 1, "bar": None}
jsons.dump(Foo(1), omit_none=True) #output: {"foo": 1}
jsons.dump(Foo(1, True)) #output: {"foo": 1, "bar": True}
Maybe this is already possible using a serializer?
ฮป .venv/bin/mypy inpai
inpai/params_definition.py:8: error: Cannot find module named 'jsons'
inpai/execution/monitors/base_monitor.py:3: error: Cannot find module named 'jsons'
inpai/execution/monitors/base_monitor.py:3: note: See https://mypy.readthedocs.io/en/latest/running_mypy.html#missing-imports
inpai/execution/monitors/gui_monitor.py:6: error: Cannot find module named 'jsons'
inpai/execution/monitors/elasticsearch_stdout_monitor.py:4: error: Cannot find module named 'jsons'
https://mypy.readthedocs.io/en/latest/running_mypy.html#missing-imports
If the module is a third party library, you must make sure that there are type hints available for that library. Mypy by default will not attempt to infer the types of any 3rd party libraries you may have installed unless they either have declared themselves to be PEP 561 compliant stub package or have registered themselves on typeshed, the repository of types for the standard library and some 3rd party libraries.
Jsons fails on deserializing into a class with optional fields.
class C:
def __init__(self, x: typing.Optional[int] = None):
self.x = x
loaded = jsons.load({"x": 1}, C)
This raises:
AttributeError: '_SpecialForm' object has no attribute '__name__'
Let's consider the following example:
In [30]: @dataclass
...: class Test:
...: a: int
...: b: int
...:
In [31]: Test(1,2)
Out[31]: Test(a=1, b=2)
In [32]: jsons.dumps(Test(1,2))
Out[32]: '{"a": 1, "b": 2}'
In [33]: jsons.loads('{"a": 1, "b": 2, "c": 3}', Test)
Out[33]: Test(a=1, b=2)
I would expect jsons to fail in the last statement, as the protocol is not strictly the same.
Is there any rationale for such behavior? If yes, that would be great to have a strict=True
argument for jsons.
jsons should support NewType
so you could do:
Uid = NewType('uid', str)
@dataclass
class User:
uid: Uid
name: str
And them dump and load instances without DeserializationErrors
.
Inherited class is lost while deserialization. In the example below, the object chmsg3 is serialized as a class ChatMessageSection with the weight attribute ('weight': 17, '-cls': 'main.ChatMessageSection'}]). But after deserialization the object becomes a ChatMessage.
import jsons
from dataclasses import dataclass
from typing import List
@dataclass
class ChatUser:
name: str
@dataclass
class ChatMessage:
user: ChatUser
msg_text: str
@dataclass
class ChatMessageSection(ChatMessage):
weight: int
@dataclass
class ChatMessages:
msg_list: List[ChatMessage]
chmsg = ChatMessage(ChatUser("Thierry"), "Bonjour")
chmsg2 = ChatMessage(ChatUser("Casimir"), "Hello")
chmsg3 = ChatMessageSection(ChatUser("Leonard"), "Coucou", weight=17)
chat_msgs = ChatMessages([chmsg, chmsg2, chmsg3])
print(chat_msgs, end="\n\n")
dumped = jsons.dump(chat_msgs, strip_microseconds=True, verbose=jsons.Verbosity.WITH_EVERYTHING)
print(dumped, end="\n\n")
instance = jsons.load(dumped)
print(instance, end="\n\n")
When an abstract base class is inherited from, dumping an instance of that implementation will include private abc attributes.
Example:
class A(ABC):
pass
@dataclass
class B(A):
x: int
dumped = jsons.dump(B(42))
Output:
{'_abc_cache': [], '_abc_negative_cache': [], '_abc_negative_cache_version': 47, '_abc_registry': [], 'x': 42}
By default, they should not be included.
It is possible to get following anomaly:
utc = datetime.datetime.now(tz=datetime.timezone.utc)
local = datetime.datetime.now()
print(jsons.dump(utc)) # >>> 2019-02-17T20:14:32Z
print(jsons.dump(local)) # >>> 2019-02-18T00:14:32-20:00
I set my timezone to +04:00
, jsons
interprets it as -20:00
. This suggests I am a day in future.
It is caused by the way you calculate datetime offset
hrs_delta = datetime.now().hour - gmtime().tm_hour
Is it possible declaring a serializer for a single dump command, instead of declaring it globally?
Currently:
jsons.set_serializer(my_datetime_serializer, datetime)
jsons.dump(foo)
Suggested:
jsons.dump(foo, serializers={datetime: my_datetime_serializer})
Motivation:
I want to have only a particular dump pass datetimes along without string parsing, since the output should be populated in a MongoDB. However, I do other dump's in the script that should use regular behaviour.
jsons.set_serializer(my_datetime_serializer, datetime)
jsons.dump(foo)
jsons.dump(bar) # this should not have my_datetime_serializer
Obviously I could set serializer before and after, but this appears hacky-hacky, and assumes that I know the previous serializer:
jsons.set_serializer(my_datetime_serializer, datetime)
jsons.dump(foo)
jsons.set_serializer(custom_datetime_serializer, datetime)
jsons.dump(bar) # this should not have my_datetime_serializer
When deserializing, jsons
accepts None
when not in "strict-mode". It should not accept None
at all, except when the target class is one of Any
, NoneType
, Optional[...]
, Union[None, ...]
.
Rationale:
# Suppose some_obj happens to be None:
loaded_obj = jsons.load(some_obj, SomeClass) # loaded_obj is now None.
...
# Somewhere else in your code, you do:
loaded_obj.some_method() # This raises.
The above example shows how accepting a None
may be confusing and possibly lead a developer in the wrong direction by falsely declaring the deserialization to be successful. If jsons.load(some_obj, SomeClass)
raised an error already, the developer may sooner realize the cause of the problem.
To summarize, here is an example:
jsons.load(None, str)
Current output:
None
Desired output:
DeserializationError
is raised.
This was also mentioned in Issue#56.
This issue is possibly related Issue#26.
Textual type hints cannot be deserialized as jsons
does not handle string type hints well. This makes it impossible to deserialize into a recursive structure.
Example:
from dataclasses import dataclass
from typing import Optional
import jsons
@dataclass
class Node:
value: object
next: Optional['Node']
linked_list = Node(10, Node(20, Node(30, None)))
dumped = jsons.dump(linked_list)
So far so good. The value of dumped
is now:
{'next': {'next': {'next': None, 'value': 30}, 'value': 20}, 'value': 10}
Now to deserialize:
jsons.load(dumped, Node)
This raises the following error:
Traceback (most recent call last):
File "C:\Projects\Personal\jsons\jsons\_main_impl.py", line 128, in load
return deserializer(json_obj, cls, **kwargs_)
File "C:\Projects\Personal\jsons\jsons\deserializers\default_object.py", line 31, in default_object_deserializer
constructor_args = _get_constructor_args(obj, cls, **kwargs)
File "C:\Projects\Personal\jsons\jsons\deserializers\default_object.py", line 58, in _get_constructor_args
constructor_args_in_obj = {key: value for key, value in args_gen if key}
File "C:\Projects\Personal\jsons\jsons\deserializers\default_object.py", line 58, in <dictcomp>
constructor_args_in_obj = {key: value for key, value in args_gen if key}
File "C:\Projects\Personal\jsons\jsons\deserializers\default_object.py", line 57, in <genexpr>
in signature_parameters.items() if sig_key != 'self')
File "C:\Projects\Personal\jsons\jsons\deserializers\default_object.py", line 72, in _get_value_for_attr
if obj and sig_key in obj:
TypeError: argument of type 'int' is not iterable
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:/Projects/Personal/jsons/effe.py", line 20, in <module>
jsons.load(dumped, Node)
File "C:\Projects\Personal\jsons\jsons\_main_impl.py", line 128, in load
return deserializer(json_obj, cls, **kwargs_)
File "C:\Projects\Personal\jsons\jsons\deserializers\default_object.py", line 31, in default_object_deserializer
constructor_args = _get_constructor_args(obj, cls, **kwargs)
File "C:\Projects\Personal\jsons\jsons\deserializers\default_object.py", line 58, in _get_constructor_args
constructor_args_in_obj = {key: value for key, value in args_gen if key}
File "C:\Projects\Personal\jsons\jsons\deserializers\default_object.py", line 58, in <dictcomp>
constructor_args_in_obj = {key: value for key, value in args_gen if key}
File "C:\Projects\Personal\jsons\jsons\deserializers\default_object.py", line 57, in <genexpr>
in signature_parameters.items() if sig_key != 'self')
File "C:\Projects\Personal\jsons\jsons\deserializers\default_object.py", line 75, in _get_value_for_attr
meta_hints, **kwargs)
File "C:\Projects\Personal\jsons\jsons\deserializers\default_object.py", line 109, in _get_value_from_obj
value = load(obj[sig_key], arg_cls, meta_hints=new_hints, **kwargs)
File "C:\Projects\Personal\jsons\jsons\_main_impl.py", line 132, in load
raise DeserializationError(str(err), json_obj, cls)
jsons.exceptions.DeserializationError: argument of type 'int' is not iterable
Parameter strip_properties
of dump
function works only for top-level object. @property
of nested objects are always included in json. See my gist example:
https://gist.github.com/haluzpav/b578dad460cb3670d55305dc65d9d2bf
Is there a way to also strip properties of nested objects?
Hi, I have a JSON containing objects that contain field named self
and it crashes due to this error: jsons.exceptions.DeserializationError: Could not deserialize value "{'self': 'api/v3/message/0'}" into "__main__.MyClass". __init__() got multiple values for argument '__dataclass_self__'
. I understand the error and the cause, but is there a way how to bypass this limition other than renaming the field in the original JSON string before deserializing and also the class attribute? Thanks
Regards,
Adam
Example code (live demo here):
import jsons
import dataclasses
@dataclasses.dataclass
class MyClass:
self: str
myclassObj = jsons.loads('''{"self":"api/v3/message/0"}''', MyClass)
print(myclassObj)
Hello!
0.8.5 version introduced 2 deprecation warnings:
.tox/py37/lib/python3.7/site-packages/jsons/__init__.py:88
/home/glen/prywatne/traktpy/.tox/py37/lib/python3.7/site-packages/jsons/__init__.py:88: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
from collections import deque, Mapping
.tox/py37/lib/python3.7/site-packages/jsons/deserializers/default_iterable.py:1
/home/glen/prywatne/traktpy/.tox/py37/lib/python3.7/site-packages/jsons/deserializers/default_iterable.py:1: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
from collections import Mapping, Iterable
-- Docs: https://docs.pytest.org/en/latest/warnings.html
Have a good Sunday :)
from datetime import datetime
import jsons
from jsons._common_impl import get_class_name
print(get_class_name(datetime))
# >>> datetime
jsons.dumps(datetime.utcnow())
print(get_class_name(datetime))
# >>> datetime.datetime
It is caused by the fact that classes are automatically announced during dumping, in fully qualified form. If a class is announced, it's name in get_class_name
is taken directly from _announced_classes
, without taking into account fully_qualified: bool
argument.
import jsons
import numpy as np
jsons.dumps(np.array([0]))
causes the following error message to repeat:
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/anaconda3/lib/python3.7/site-packages/jsons/_main_impl.py", line 63, in dump
return serializer(obj, cls=cls, **kwargs_)
File "/usr/local/anaconda3/lib/python3.7/site-packages/jsons/serializers/default_object.py", line 56, in default_object_serializer
**kwargs_)
File "/usr/local/anaconda3/lib/python3.7/site-packages/jsons/serializers/default_dict.py", line 25, in default_dict_serializer
strip_nulls=strip_nulls, **kwargs)
File "/usr/local/anaconda3/lib/python3.7/site-packages/jsons/_main_impl.py", line 65, in dump
raise SerializationError(str(err))
jsons.exceptions.SerializationError: maximum recursion depth exceeded in comparison
The datetime instance is wrongly serialized if initialized from utcnow()
. Example:
dt_local = datetime.datetime.now()
print(dt_local)
# >>> 2019-02-16 17:48:34.714603
print(jsons.dump(dt_local))
# >>> 2019-02-16T17:48:34+01:00
dt_utc = datetime.datetime.utcnow()
print(dt_utc)
# >>> 2019-02-16 16:48:34.715108
print(jsons.dump(dt_utc))
# >>> 2019-02-16T16:48:34+01:00
# this last one is clearly wrong
It seems like serialization using jsons.dump(obj)
is at least 10 times slower than writing custom serialization method to each class. In my case, obj was a nested dataclasses
and data was in the order of 100MBs.
I can pull up some numbers but I just wanted to start a discussion if this is a known issue and if so, what might cause this?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.