surenkov / django-pydantic-field Goto Github PK

The docs we have at the moment only cover essential use cases, with lots of details missing. This indeed should be expanded.
What should be added, to my knowledge:

• Specify a minimal required python version.
• Explicitly state supported pydantic features.
• Describe field internal behavior, demonstrating behavior of model wrapping.
• Also describe model's config resolution, as maybe it's not what you expect to see.
• Add Working with migrations section -- I had to introdice some internal machinery to support Generic Aliases.
• Expand DRF support section. At the moment it contains almost only code samples.

I have a pydantic model like this with a few issues preventing form rendering in the Admin UI:

class Binary(BaseModel):
    model_config = ConfigDict(extra='ignore', populate_by_name=True)

    name: BinName = 'binary-name'  # I wish I could use None here / have no default, but that's also not supported

	# provider_overrides: Dict[str, str] = Field(default={}, alias='overrides') 
	# ^ this also breaks, see separate issue: https://github.com/surenkov/django-pydantic-field/issues/64
    
    loaded_provider: Optional[str] = Field(default=None, alias='provider')
    loaded_abspath: Optional[str] = Field(default=None, alias='abspath')
    
    # loaded_version: Optional[Tuple[int, int, int]] = Field(default=None, alias='version')
    # this also doesn't work: Cannot read properties of undefined (reading 'hasOwnProperty')

When it gets rendered in the Admin UI it looks like this though:

I assume because the schema defines two allowed types string and null, so this becomes two "options" in the UI with a dropdown?

Potential solutions:

1. If the only options are <something> and null (a very common case with Optional[] fields), just show one field in the UI and treat leaving it empty as null
2. Generate better labels for the options dropdown based on the type, e.g. show string and null instead of Option 1 and Option 2
3. ⭐️ Show a radio selector to the left of the text field to choose the type, e.g. string/null/int etc. This would allow it to support more Union types than just Optional, as it could swap out the text field widget with widgets for other types depending on which radio button is clicked.

I like option 3 best as it's the most flexible for all Union types, but option 1 may be the least amount of code to implement to just get nullable types working fast.

If we have a pydantic model with any Pydantic specific fields, then the model will not be properly serialized and the crash will appear.

For instance:

import pydantic
import django_pydantic_field

class MyModel(pydantic.BaseModel):
     my_url: pydantic.HttpUrl = pydantic.HttpUrl('https://example.com/')

obj = MyModel()
field = django_pydantic_field.SchemaField(schema=MyModel)
field.get_prep_value(obj)

And the output will be:

TypeError: Object of type Url is not JSON serializable

It can be fixed if we use the mode=json when we perform _prepare_raw_value inside get_prep_value like this:

def get_prep_value(self, value: t.Any):
        value = self._prepare_raw_value(value, mode='json')
        return super().get_prep_value(value)

Or am I missing something?

I ran into an issue when I upgraded from Django 4.1 to 4.2. This issue did not occur previously.

I have a model that looks like this:

class Organization(models.Model):
    domain: models.CharField()
    integrations: list[Integration] = SchemaField(default=list, blank=True)

I can create new Organization records correctly, but when I try to read one, e.g. by Organization.objects.get(name="foo"), I get the following error:

  File "/Users/tao/dev/cinder/kosa/accounts/middleware.py", line 24, in __call__
    organization = Organization.objects.get(domain=hostname)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/tao/dev/cinder/.venv/lib/python3.11/site-packages/django/db/models/manager.py", line 87, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/tao/dev/cinder/.venv/lib/python3.11/site-packages/django/db/models/query.py", line 633, in get
    num = len(clone)
          ^^^^^^^^^^
  File "/Users/tao/dev/cinder/.venv/lib/python3.11/site-packages/django/db/models/query.py", line 380, in __len__
    self._fetch_all()
  File "/Users/tao/dev/cinder/.venv/lib/python3.11/site-packages/django/db/models/query.py", line 1881, in _fetch_all
    self._result_cache = list(self._iterable_class(self))
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/tao/dev/cinder/.venv/lib/python3.11/site-packages/django/db/models/query.py", line 121, in __iter__
    for row in compiler.results_iter(results):
  File "/Users/tao/dev/cinder/.venv/lib/python3.11/site-packages/django/db/models/sql/compiler.py", line 1500, in apply_converters
    value = converter(value, expression, connection)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/tao/dev/cinder/.venv/lib/python3.11/site-packages/django/db/models/fields/json.py", line 94, in from_db_value
    return json.loads(value, cls=self.decoder)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/[email protected]/3.11.2_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/json/__init__.py", line 359, in loads
    return cls(**kw).decode(s)
           ^^^^^^^^^^^^^^^^^^^
  File "/Users/tao/dev/cinder/.venv/lib/python3.11/site-packages/django_pydantic_field/base.py", line 70, in decode
    value = self.schema.parse_raw(obj).__root__  # type: ignore
            ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "pydantic/main.py", line 550, in pydantic.main.BaseModel.parse_raw
  File "pydantic/main.py", line 527, in pydantic.main.BaseModel.parse_obj
  File "pydantic/main.py", line 342, in pydantic.main.BaseModel.__init__
pydantic.error_wrappers.ValidationError: 1 validation error for FieldSchema[list[kosa.accounts.pydantic_models.Integration]]
__root__
  value is not a valid list (type=type_error.list)

By putting a breakpoint in SchemaDecoder.decode() in base.py, I can see that obj looks wrong:

(Pdb) obj
'"[]"'

It's the string "[]" -- but it seems like it should just be the string [].

Looking at my database, I can see that the integrations JSON column does not contain an empty list, but the string "[]".

Going further, if I set a breakpoint in SchemaEncoder.encode(), I can see that the obj argument is also a string (not an empty list). This should raise a ValidationError, but self.raise_errors is false.

I'm running

• Python 3.11.1
• Django 4.2.2
• django-pydantic-field 0.2.4
• pydantic 1.10.4

Is there a way to disable validation on retrieval? Currently Django will throw a 500 error if I have an object with an invalid schema in the database. Even when calling Model.objects.all().delete() throws a ValidationError. Thanks for any ideas!.

Figures out, current field implementation does not support for deferred type annotations (aka forward referencing annotations). The example below won't work:

class FooModel(models.Model):
    # This field definition will fail, since FooSchema is not defined 
    # in module scope at the moment of FooModel finalisation 
    field: "FooSchema" = SchemaField()

class FooSchema(pydantic.BaseModel):
    foo: int = ...

At the moment forward references are only resolved if the corresponding type is defined BEFORE model itself. E.g. this is a valid code, even though type annotation is quoted in string literal:

class FooSchema(pydantic.BaseModel):
    foo: int = ...

class FooModel(models.Model):
    field: "FooSchema" = SchemaField()  # <<- and now this is ok

I'm unsure how to deal with this without intruding into model inheritance chain with custom methods like Pydantic's #.update_forward_refs(). I already tried to postpone the resolution for such annotations to the field descriptor level, but in this case it breaks migrations generation.

Due to this, I'm considering such behavior as expected, unless we find a good solution: you should always declare your schemas prior to field definition, either in models.py or, preferrably, in a separate module.

When using a default and creating migrations it always detects the field as changed and regenerates migrations.

class BuildingTypes(str, Enum):
    FRAME = "frame"
    BRICK = "brick"
    STUCCO = "stucco"


class BuildingMeta(BaseModel):
    type: Optional[BuildingTypes]


default_meta = BuildingMeta(type=BuildingTypes.FRAME)


class Building(models.Model):
    meta = PydanticSchemaField(schema=BuildingMeta, default=default_meta)

This example should continue to create new migrations every time makemigrations is run for the app.

Hi @surenkov

I have another question regarding your pydantic field :)

I'm creating a BaseModel like so:

I18NSchema = create_model(
    "I18NField",
    __config__={"extra": Extra.forbid},
    **{
        lang[0]: (
            str,
            Field(default=None, description=f"Translation in {lang[0].upper()}"),
        )
        for lang in settings.LANGUAGES
    },
)

which basically results in a:

class I18NSchema(BaseModel):
    en: str = None
    fr: str = None
    ...

(I initially tried it with default="" too)

Is there a way to not have this in the database in the end:

{"af": null, "ar": null, "en": "hello", "es": null, "fr": "bonjour", "km": null, "lo": null, "mn": null, "pt": null, "ru": null, "th": null, "zh": null}
// respectively:
{"af": "", "ar": "", "en": "hello", "es": "", "fr": "bonjour", "km": "", "lo": "", "mn": "", "pt": "", "ru": "", "th": "", "zh": ""}

but only the keys that were actually set, i.e. in this case: {"en": "hello", "fr": "bonjour"}

thanks a bunch!

If I define a model like the example:

class MyModel(models.Model):
    # Infer schema from field annotation
    foo_field: Foo = SchemaField()

Then I go ahead and create an empty instance:

model = MyModel()

I get an error:

ValidationError: ['[{"type":"model_type","loc":[],"msg":"Input should be a valid dictionary or instance of Foo","input":null,"ctx":{"class_name":"Foo"},"url":"https://errors.pydantic.dev/2.6/v/model_type"}]']

This behavior is not consistent with other field types where there's no issue on creation, but saving will give an error. Django's model constructor tries to set the field to None and it fails as the field is not nullable. I can try to fix this and submit a PR but I wanted to get other opinions before spending time on this.
What do you think should be the behavior?

Trying to use this lib with pydantic==1.9.1 leads to import error.
ImportError: cannot import name 'get_config' from 'pydantic.config'
In 1.9.1 there is no this function, you can check it here https://github.com/pydantic/pydantic/releases/tag/v1.9.1

If you don't want to fix backward compatibility so far, you could just fix required version of pydantic.

Currently it's not possible to automatically infer corresponding serializer field type for PydanticSchemaField.
Some third-party libraries patch ModelSerializer field registry to achieve such behaviour.

What I'd like to know is:

1. How ModelSerializer treats model's PydanticSchemaField by default, since it is a subclass of JSONField.
2. Is it possible to simply patch ModelSerializer registry with corresponding serializer field type.

This is a separate issue to add NamedTuple/Tuple support, broken out from #65

I have this Pydantic model that I want to allow users to edit in the UI:

from typing import NamedTuple

class SemVer(NamedTuple):
    major: int
    minor: int = 0
    patch: int = 0
    
    def __new__(cls, *args, **kwargs):
		# allow creating from string SemVer('1.2.3') or SemVer(1, 2, 3)
		if len(args) == 1 and isinstance(args, str):
			args = (int(chunk) for chunk in args[0].split('.'))
        return cls(*args, **kwargs)
	
class Dependency(models.Model):
    min_version: SemVer = SchemaField(default=(0,0,1))

Currently trying to render the Admin UI produces a number of errors in the UI, but the key one for this issue is:

(!) Error: Error while creating EditorState: Invalid schema: Schema of type 'array' must have a key called 'items'

I have also seen this error when trying to nest SemVer inside a different Pydantic BaseModel:

Investigate the possibility of pydantic v2 usage as long as it goes to RC

Ahoy there,

first of: thanks for this package 🖤

I have a situation which might be pebkac, but at this point I figured I'd better ask:

Situation:

class CurrentDateAreaSchema(pydantic.BaseModel):
    current: bool
    date: str
    area: Decimal

contract_size = SchemaField(
    schema=list[CurrentDateAreaSchema], blank=True, default=list
)

It works nicely and saved the right information into the database; however when I return it via normal DRF ModelViewset and ModelSerializer logic I get this:

"contract_size": [
  [
    ["current",false],
    ["date","2022"],
    ["area",1104]
  ],
  [
    ["current",true],
    ["date","2024"],
    ["area",3306.9]
  ]
],

I would rather like to have

"contract_size": [
  {
   "current":false,
   "date":"2022",
   "area":1104
  },
  {
    "current":true,
    "date":"2024",
    "area":3306.9
  }
],

Is the former intended behaviour? Or am I missing something?

Thanks!

I cannot use Django's bulk_update on a model that uses django-pydantic-field SchemaField.
I'm not sure if this is django-pydantic-field or django problem.

Repro:

class ExampleSchemaField(BaseModel):
    count: int

class Example(models.Model):
    example_field: ExampleSchemaField = SchemaField(default=ExampleSchemaField)

e = Example.objects.create({"count": 1})
Example.objects.bulk_update([e], ["example_field"])

This throws TypeError: Object of type Case is not JSON serializable.

It works fine when updating without bulk_update:

e = Example.objects.create({"count": 1})
e.example_field = {"count": 2}
e.save()

And bulk_create seems to work as well.

I have a pydantic model that has a field like so:

class MySchemaModel(BaseModel):
    my_mapping: Dict[str, str] = Field(default={})

DEFAULT = MySchemaModel()

class Model(models.Model):
    test_field: MySchemaModel = SchemaField(default=DEFAULT)

This field contains a mapping of str to str and I Want users to be able to add their own entries , but the default contains no pre-defined keys/properties, so it throws an error right now:

Error: Error while creating EditorState: Invalid schema: Schema of type 'object' must have at least one of these keys: ['properties' or 'keys' or 'oneOf' or 'anyOf' or 'allOf']

I think the issue can be fixed by adding an empty properties: {} entry to the schema generation /conversion code before it gets passed to django-jsonform:
MySchemaModel().model_json_schema()

{
    "properties": {
        "my_mapping": {
            "additionalProperties": {
                "type": "string"
            },
            "default": {},
            "title": "My Mapping",
+           "properties": {},
            "type": "object"
        }
    },
    "title": "MySchemaModel",
    "type": "object"
}

I think django-jsonform should also natively support objects with no properties/keys defined if they have additionalProperties set. I commented on a related an issue on their side here: bhch/django-jsonform#144.

When combining a custom root type in pydantic and making the schema optional (with null and blank flag in django model) the django admin edit and create views fail.

AttributeError: 'str' object has no attribute 'root'

I assume it's due the prefilled value by django admin being None and it someone gets turned into a "null" string which then fails during validation as it is no valid json.

Here the core model and schema definitions (inspired by the pydantic example):

class Pets(RootModel):
    root: List[str]

    def __iter__(self):
        return iter(self.root)

    def __getitem__(self, item):
        return self.root[item]


class PetsModel(models.Model):
    pets: Optional[Pets] = SchemaField(blank=True, null=True)

I prepared an example to reproduce the issue: https://github.com/noxan/sample-django-pydantic-field/tree/main

since last version(s) the field lookup from JSONField doesnt work any more for me.

i.e with.

class MySchema(BaseModel):
  required: bool = True

>>> Model.objects.filter(config__required=True)
ValidationError: 1 validation error for FieldSchema[MySchema]
__root__
  value is not a valid dict (type=type_error.dict)

Since Pydantic v2, it looks feasible to walk through the what-is-called CoreSchema, which is used by pydantic_core to perform the data validation/serialization.

This opens up the door to get rid of DjangoJSONEncoder, which the only responsibility left in 0.3.* is to serialize lookup parameters. It's still not possible to perform a partial model serialzation right out of the box (especially if we're talking about nested schemas), but at least now we could traverse inner structure through the model's core schema, extract particular schema definition for the lookup and perform a serialization with it.

Although by_alias is accounted for and passed to the encoder, it is not extracted from kwargs so model.Field and form.Field's constructors will error out as they have no **kwargs in their definition.
I haven't checked what happens for DRF.

Along with DRF's schema generators, it would be handy to have an adapter for def-spectacular, as it's becoming de-facto standard tool for OpenAPI schema generation in DRF.

Hey, first of all, thank you so much for this awesome package.

The package is mostly completely typed, but the absence of a py.typed marker raises errors with mypy. Would it be possible to add a marker to make it PEP 561 compliant?

Hi dear maintainer, I am creating this ticket just to notify about possible extension of your repo by cooperating with the following repo: https://github.com/bhch/django-jsonform/

It has the ability to create very functional widgets for JSON schema which is very convenient for forms and django-admins.

Having great FE UI for django-admin and ordinary forms is essential to build functional project. Therefore mentioning or adding HOWTO with example of collaboration of these 2 projects would be beneficial, imho.

This is more likely relevant to serializer fields, though may still apply to parsers/renderers, that needs to be figured out.

While rendering serializer schema, we can come up in the situation when multiple fields may refer to the same pydantic model. Currently, each field independently renders its schema into a nested definition. Probably we can extract all common definitions into an upper level and point to them with references.

First of all, great idea -- this will give me less to worry about when working with JSON fields.

I just ran a quick test with one of my models, and am I right that schema validation only occurs when loading the model data? So I should always use the schema to alter the model JSON data and not assign properties directly? (It's not that I mind, I just want to know for sure).

class SettingsSchema(pydantic.BaseModel):
    private: bool = False

class Item(models.Model):
    settings: SettingsSchema = SchemaField(default=SettingsSchema)

def test_settings_validation():
    item = Item()
    item.settings.private= 'not a bool'
    item.save()
    item.refresh_from_db()

Test output:

def test_item_settings_validation():
        item = Item()
        item.settings.private = 'not a bool'
        item.save()
>     item.refresh_from_db()
...
validation_error = ValidationError(model='FieldSchema[SettingsSchema]', errors=[{'loc': ('__root__', 'private'), 'msg': 'value could not be parsed to a boolean', 'type': 'type_error.bool'}])

The error happens when you pass an invalid JSON inside the SchemaField

Python: 3.10.12
Django: 4.1.10
Pydantic: 1.10.2
django-pydantic-field: 0.2.8

Steps to reproduce:

from pydantic import BaseModel
from django.forms import Form
from django_pydantic_field.forms import SchemaField

class MySchema(BaseModel):
    a: int

class MyForm(Form):
    my_field = SchemaField(schema=MySchema)


f = MyForm(data={'my_field': "invalid json}}}"})
f.is_valid()  # <- here you'll get AttributeError

Traceback:

Traceback (most recent call last):
  File "pydantic/main.py", line 539, in pydantic.main.BaseModel.parse_raw
  File "pydantic/parse.py", line 37, in pydantic.parse.load_str_bytes
  File "/usr/local/lib/python3.10/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
  File "/usr/local/lib/python3.10/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/local/lib/python3.10/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/django_pydantic_field/forms.py", line 40, in to_python
    return super().to_python(value)
  File "/usr/local/lib/python3.10/site-packages/django/forms/fields.py", line 1359, in to_python
    converted = json.loads(value, cls=self.decoder)
  File "/usr/local/lib/python3.10/json/__init__.py", line 359, in loads
    return cls(**kw).decode(s)
  File "/usr/local/lib/python3.10/site-packages/django_pydantic_field/base.py", line 75, in decode
    value = self.schema.parse_raw(obj).__root__  # type: ignore
  File "pydantic/main.py", line 548, in pydantic.main.BaseModel.parse_raw
pydantic.error_wrappers.ValidationError: 1 validation error for FieldSchema[MySchema]
__root__
  Expecting value: line 1 column 1 (char 0) (type=value_error.jsondecode; msg=Expecting value; doc=invalid json}}}; pos=0; lineno=1; colno=1)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/usr/local/lib/python3.10/site-packages/django/forms/forms.py", line 205, in is_valid
    return self.is_bound and not self.errors
  File "/usr/local/lib/python3.10/site-packages/django/forms/forms.py", line 200, in errors
    self.full_clean()
  File "/usr/local/lib/python3.10/site-packages/django/forms/forms.py", line 437, in full_clean
    self._clean_fields()
  File "/usr/local/lib/python3.10/site-packages/django/forms/forms.py", line 449, in _clean_fields
    value = field.clean(value)
  File "/usr/local/lib/python3.10/site-packages/django/forms/fields.py", line 198, in clean
    value = self.to_python(value)
  File "/usr/local/lib/python3.10/site-packages/django_pydantic_field/forms.py", line 42, in to_python
    raise ValidationError(e.errors(), code="invalid")
  File "/usr/local/lib/python3.10/site-packages/django/core/exceptions.py", line 167, in __init__
    message = ValidationError(message)
  File "/usr/local/lib/python3.10/site-packages/django/core/exceptions.py", line 160, in __init__
    self.error_dict[field] = messages.error_list
AttributeError: 'ValidationError' object has no attribute 'error_list'

I have the following pydantic models:

class EUTaxonomyValue(BaseModel):
    type: Literal["eu_taxonomy"] = "eu_taxonomy"
    nace_code: str
    economic_activity: str
    classification: str | None


class EconomicActivity(BaseModel):
    type: Literal["economic_activity"] = "economic_activity"
    activity_code: str
    activity_name: str


class DateRange(BaseModel):
    type: Literal["daterange"] = "daterange"
    start_date: date
    end_date: date

ComplexValue = Annotated[
    None | EUTaxonomyValue | EconomicActivity | DateRange,
    Field(discriminator="type", title="Complex Value"),
]

Now I define a new field value_complex in one of my serializers:

class FieldValueSerializer(serializers.ModelSerializer):
    value_complex = SchemaField(schema=ComplexValue, required=False)

I'm not sure if this issue is related to django-pydantic-field or drf-spectacular.

The generated Open API 3 Schema is invalid.

value_complex:
          title: value_complex
          discriminator:
            propertyName: type
            mapping:
              eu_taxonomy: '#/definitions/EUTaxonomyValue'
              economic_activity: '#/definitions/EconomicActivity'
              daterange: '#/definitions/DateRange'
          oneOf:
          - title: EUTaxonomyValue
            type: object
            properties:
              type:
                title: Type
                default: eu_taxonomy
                enum:
                - eu_taxonomy
                type: string
              nace_code:
                title: Nace Code
                type: string
              economic_activity:
                title: Economic Activity
                type: string
              classification:
                title: Classification
                type: string
            required:
            - nace_code
            - economic_activity
          - title: EconomicActivity
            type: object
            properties:
              type:
                title: Type
                default: economic_activity
                enum:
                - economic_activity
                type: string
              activity_code:
                title: Activity Code
                type: string
              activity_name:
                title: Activity Name
                type: string
            required:
            - activity_code
            - activity_name
          - title: DateRange
            type: object
            properties:
              type:
                title: Type
                default: daterange
                enum:
                - daterange
                type: string
              start_date:
                title: Start Date
                type: string
                format: date
              end_date:
                title: End Date
                type: string
                format: date
            required:
            - start_date
            - end_date
          definitions:
            EUTaxonomyValue:
              title: EUTaxonomyValue
              type: object
              properties:
                type:
                  title: Type
                  default: eu_taxonomy
                  enum:
                  - eu_taxonomy
                  type: string
                nace_code:
                  title: Nace Code
                  type: string
                economic_activity:
                  title: Economic Activity
                  type: string
                classification:
                  title: Classification
                  type: string
              required:
              - nace_code
              - economic_activity
            EconomicActivity:
              title: EconomicActivity
              type: object
              properties:
                type:
                  title: Type
                  default: economic_activity
                  enum:
                  - economic_activity
                  type: string
                activity_code:
                  title: Activity Code
                  type: string
                activity_name:
                  title: Activity Name
                  type: string
              required:
              - activity_code
              - activity_name
            DateRange:
              title: DateRange
              type: object
              properties:
                type:
                  title: Type
                  default: daterange
                  enum:
                  - daterange
                  type: string
                start_date:
                  title: Start Date
                  type: string
                  format: date
                end_date:
                  title: End Date
                  type: string
                  format: date
              required:
              - start_date
              - end_date

Regarding https://swagger.io/docs/specification/data-models/inheritance-and-polymorphism/ oneOf should have a list of $ref and those 3 pydantic models should be defined in #/components/schemas/