adcombo / flask-combo-jsonapi Goto Github PK

This issue is blocked by #10

• start with version 1.0
• Setup Github Action for this task.
• Update installation manual https://github.com/AdCombo/flask-combo-jsonapi/blob/develop/docs/installation.rst/

Hello all,

I am trying to filter results by searching for a word within an array. For instance this simple model:

class Person(db.Model):
    id = db.Column(db.Integer, primary_key=True)
    words = db.Column(db.ARRAY(db.TEXT))

class PersonSchema(Schema):
    class Meta:
        type_ = 'person'
        self_view = 'person_detail'
        self_view_kwargs = {'id': '<id>'}
        self_view_many = 'person_list'

    id = fields.Integer(as_string=True, dump_only=True)
    words = fields.List(fields.String())

# skipping some DB code here

session.add(Person(id=1, words=["foo", "nobar", "arbitraryWord"]))
session.add(Person(id=2, words=["bar", "nofoo"]))

In this example I try to find all persons with the word "arbitraryWord" in their list of words. I hoped using the operator "in_" would work like this:

       {"name":"arbitraryWord",
        "op":"in_",
        "field":"words"}]

The result is a "Bad filter value". I tried several variations with no luck. Hope someone can give me a hint. Find full example code here: https://github.com/vi0815/exampleArraySearch

dump_only=True on id field as given in examples results in error

curl -X PATCH -d '{ "data":{ "type": "email", "id": "100","attributes":{"status": "S"} } }' -H 'Content-Type: application/vnd.api+json' 'localhost:5000/v1/emails/100' | jq .

{
  "errors": [
    {
      "detail": "Unknown field.",
      "source": {
        "pointer": "/data/id"
      },
      "status": "422",
      "title": "Validation error"
    }
  ],
  "jsonapi": {
    "version": "1.0"
  }
}

It can be reproduced by using your example (in /examples/api.py) and performing the patch call:

PATCH /computers/1 HTTP/1.1
Content-Type: application/vnd.api+json
Accept: application/vnd.api+json

{
  "data": {
    "type": "computer",
    "id": "1",
    "attributes": {
      "serial": "Amstrad 2"
    }
  }
}

It seems that the only way to avoid this is to remove the dump_only=True on the id field from the computer schema, but in that case, we can update the id with a patch call... which we obviously don't want. ;-)

Original issue: miLibris/flask-rest-jsonapi#193

If you have a model with JSONB field, and there's a dumped date / datetime, marshmallow won't be able to serialize it due to it being serialized already (date and datetime strings will cause an error when tried to serialize)
smth like TypeError: descriptor 'isoformat' requires a 'datetime.date' object but received a 'str'

possible solution: load all JSONB fields when loading object, or just before serializing

from marshmallow import Schema, fields
from sqlalchemy.dialects.postgresql import JSONB


def load_jsonb_columns(instance, schema: Schema) -> None:
    """
    :param instance:
    :param schema:
    :return:
    """
    for attr in type(instance)._sa_class_manager.attributes:
        if not isinstance(attr.expression.type, JSONB):
            continue

        if hasattr(instance, attr.name) and attr.name in schema.declared_fields:
            fld: fields.Nested = schema.declared_fields[attr.name]
            if not isinstance(fld, fields.Nested):
                continue
            deserialized = fld.schema.load(getattr(instance, attr.name))
            setattr(instance, attr.name, deserialized)

also there can be other types which require to be converted

Hi is any special reason not to use sqlalchemy 1.4.x ?

There's already a sphinx config. It requires minor refactoring and then we can deploy to read-the-docs

There's an issue with relationship fields in marshmallow-jsonapi. While PR is waiting to be reviewed
marshmallow-code/marshmallow-jsonapi#302, we need to fix dependencies here.

Description
When attempting to use sparse fieldsets for an invalid type, a 500 is raised. Is this the correct functionality? I would think this is more of a 400 level error. To demonstrate, if one were to make a request for GET /cars?fields[dealership]=name using the code below, they would get the documented 500 response. I would be happy to open a PR to change this functionality if there is agreement that there is a better way to handle these errors (i.e. raise 4XX error).

Problematic Code

Example

# Example
class Car(Base):
    id = Column(Int, primary_key=True)
    make = Column(Str)
    model = Column(Str)

class CarSchema
    class Meta:
        type_ = "car"

    make = fields.Str()
    model = fields.Str()

class CarListResource(ResourceList):
    methods = ["GET"]
    schema = CarSchema
    data_layer = {"session": db.session, "model": Car}

app = Flask(__name__)
api = Api(app)
api.route(
    CarListResource, "car_resource", "/cars",
)

Example Response

// GET /cars?fields[dealership]=name`
// status code: 500

{
  "errors": [
    {
      "status": "500",
      "title": "Unknown error",
      "detail": "Couldn't find schema for type: dealership"
    }
  ],
  "jsonapi": {
    "version": "1.0"
  }
}

I have question regarding branches can we delete branches which are not used or not have any significance.

Sometimes in projects we use models with __init__ method. something like this:

class Cube(Model):
    x = Column()
    y = Column()
    z = Column()

    def __init__():
        self.x = 10
        self.y = 10

However, a resource with a post method will not work with this model. The create_object method throws a TypeError. This is due to such initialization of the model in this method:

obj = self.model(**{key: value
                            for (key, value) in data.items() if key not in join_fields})

That is, there is no validation of the arguments of the __init__ method.

I suggest using inspect.getfullargspec to determine that a model can only accept a limited list of arguments.

Currently the jsonapi_exception_formatter decorator normally handles JsonApiException exceptions only. Source:

        try:
            return func(*args, **kwargs)
        except JsonApiException as e:
            return make_response(jsonify(jsonapi_errors([e.to_dict()])),
                                 e.status,
                                 headers)
        except Exception as e:
             ...

However, plugins can implement their own errors, which can be correlated with JsonApiException errors. This is necessary to avoid handling errors such as critical.
moreover, this library integrates with flask and should support its standard http-errors.
I suggest in the block except Exception to check the attribute code (or status_code) of the error and try to correlate it with the JSONAPI error. If a match is found, then do not log this error, but return the corresponding JSONAPI error.
Example:

except Exception as e:
    http_ex = format_http_exception(e)
    if http_ex:
        return make_response(jsonify(jsonapi_errors([http_ex.to_dict()])),
                                 http_ex.status,
                                 headers)

Installation using the source package downloaded from pypi (https://pypi.org/project/Flask-COMBO-JSONAPI/#files) would fail because "requirements.txt" used by setup.py wasn't included in the archive.

Steps to reproduce:

$ pip install Flask-COMBO-JSONAPI-1.1.0.tar.gz
Processing /tmp/Flask-COMBO-JSONAPI-1.1.0.tar.gz
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/private/var/folders/t7/t04m89ss3cj2x1bqv4gmqj5r000t9r/T/pip-req-build-6neyzad5/setup.py", line 8, in <module>
        with open(requirements_filepath) as fp:
    FileNotFoundError: [Errno 2] No such file or directory: 'requirements.txt'

    ----------------------------------------

It can be resolved if it is declared to be included in MANIFEST.in

There's an option to set custom url_field, but no way to do patch, because id is hardcoded

I know that another pk field is violation of JSON:API spec, but this feature would be neat

Sometimes an error with invalid value can be passed to the `format_http_exception. And it fails here:

  File "/.../flask-combo-jsonapi/flask_combo_jsonapi/errors.py", line 32, in format_http_exception
    status = int(code)
ValueError: invalid literal for int() with base 10: 'f405'

Maybe changing to except (TypeError, ValueError): will help

Instead of raising 404 if an object is not found, the library tries to serialize it using the schema, hence causing marshmallow-code/marshmallow-jsonapi#302

What is the use case of this behaviour? Why not raise ObjectNotFound if object returned after get from data layer is None?

I am creating a plugin which will dasherize the calls from API in filter and sort .

Example: starts-at will be converted to starts_at

But how do I initalize the filter and sort in query string manager to the new dict which I created(which has dasherized parameters).

def _update_query(
        self,
        *args,
        query: Query = None,
        qs: QueryStringManager = None,
        view_kwargs=None,
        self_json_api=None,
        **kwargs,
    ) -> Query:

A clickhouse may have entries with the same primary keys. And the alchemy in this line eliminates duplicates:

For such a case, I think you need to override the all method and use something like session.execute(query).fetchall()

Rename repo from flask-rest-jsonapi so that it's easier to search and/or allows it to be added to the JSON:API site as a separate project? In addition it makes it clearer to users that it's not the exact same project, especially in case this API diverges from upstream.

UPD: new name will be flask-combo-jsonapi

Flow:

• create a separate branch for renaming all imports and other names

• do the same in combojsonapi

• refactor our production to use new name

• run tests

• merge that branch:

  1. merge to master, tag 1.0.0
  2. rename repo
  3. update requirement in combojsonapi
  4. release combojsonapi @1.0.0

• deploy to pypi #11

Right now marshmallow 3.2.1 (2019-09-30) is used
Latest version is 3.12.1 (2021-05-10)

The only change I could find is this:

Passing the string "self" to fields.Nested is deprecated. Use a callable instead.
3.3.0 (2019-12-05)
https://marshmallow.readthedocs.io/en/stable/changelog.html#id17

Otherwise only additions / fixes / improvements which don't affect our usage.

Can't find anything breaking, but better test it before releasing (probably adding coverage would help)

Planning for v1.1.0

This way to get schema works only if using apispec plugin - it resolves schema on start

but __schema MAY and IS not resolved yet

It's resolved in Relationship.schema
https://github.com/AdCombo/combojsonapi/blob/98162841423acf20414b52c96ec2dffdda9f7131/combojsonapi/utils/marshmallow_fields.py#L22

so this way it works:

def get_related_schema(schema, field):
    return schema._declared_fields[field].schema

I cannot seem to get around this issue. The documentation isn't helpful to me in this case, but it suggests that my post-data isn't being received.

(forgive me if this isnt the proper location for this kind of report, but I'm having a difficult time finding support)

I've followed the docs on defining my models, schemas and resources - yet I cannot successfully POST.

class MyModelSchema(BaseSchema):
    """MyModel model"""
    class Meta:
        type_ = "my_model"
        self_view = "my_model_detail"
        self_view_kwargs = {"id": "<id>"}
        self_view_many = "my_models_list"
        model = MyModel
        include_relationships = False
        load_instance = True
        sqla_session = Session

    id = fields.Integer(as_string=True, dump_only=True)
    name = fields.String()
    display_name = fields.String()

    

class MyModelList(ResourceList):
    schema = MyModelSchema
    data_layer = {"session": session,
                 "model": MyModel,
                  }



class MyModelDetail(ResourceDetail):
    schema = MyModelSchema
    data_layer = {"session": session,
                 "model": MyModel,
                 }

Sometimes we just don't need the collection count for the model/resource or the query and/or the table is so huge, that it would take a lot of time to fetch the collection count.
It happens inside the get_collection here: https://github.com/AdCombo/flask-rest-jsonapi/blob/725cf11286a1ea06aa2ca62182a212d03fceb1c8/flask_rest_jsonapi/data_layers/alchemy.py#L160

So, the decision is:

• create a separate method in the SqlalchemyDataLayer for getting the collection count. For example get_collection_count. So it can be easily overridden and being replaced with anything else
• create disable_collection_count and default_collection_count, get it from the resource

Hi

When using SQLAlchemy, the class of the created object is a child of flask_sqlalchemy().Model.
Ex

db = flask_sqlalchemy(app)
class Person(db.Model): pass
class PersonDetail(ResourceDetail):
    data_layer = { 'model': Person }

But what if we don't use SQLAlchemy? What are the required parent class or interface that our custom class must implement?

Can you clarify this in the doc?

Thank you!
Camille

The purpose of this validation is to "fail early" if developer has made a mistake with fields.
I reverted commit with an attempt to implement such validation, because it didn't consider that validation is needed only for "POST" method (because schema fields will be used as args for model.__init__) and that some fields like serial primary keys should be generated automatically.

Original question on Flapison: multimeric/flapison#22

Has this library been tested with ToastedMarshmallow?
We have been using this library to great effect recently in a project (for about 8 months), but we're now seeing some performance hits during the Marshmallow serialization (sometimes 100's of ms) when dealing with many objects (100's)

I've seen people talk about ToastedMarshmallow and notice that the compute_schema recreates schema instances everytime, according to the ToastedMarshmallow documentation this will reduce the gain heavily.

Is there a reason we recreate schema objects every time? If not, what would be the best way to change the code to allow one schema object per resource?

I tried changing the code and using an in-memory object dict as an object cache, but toastedmarshmallow still made no performance impact.
Anybody tried this before?

Also posted here: miLibris#192

Search by JSON field is limited to nesting up to 2. Search with nesting of 3 or more is often required, for example extra.lvl1.lvl2

At least create hook for running black for formatting (#13)

Hi guys

Would you know distutils enough to make sqlalchemy an optional dependency?

Since we have only custom DataLayers, we don't use SQLAlchemy and it pulls too many dependencies (like greenlet which does not work on Alpine without a recompilation... so the full pip install is failing). It's a mess. Anyway SQLAlchemy is just a default implementation and should not be mandatory to use it.

I you can supply a simple and nice way to remove sqlalchemy from the packages pulled by pip, it would be awesome.

Best regards
Camille

Hi,

The following test fails:

==================================== ERRORS ====================================
_ ERROR at setup of test_get_list_with_simple_filter_relationship_custom_qs_api _

client = <FlaskClient <Flask 'tests.conftest'>>, app = <Flask 'tests.conftest'>
api_blueprint = <Blueprint 'api'>
custom_query_string_manager = <class 'tests.test_sqlalchemy_data_layer.custom_query_string_manager.<locals>.QS'>
person_list_2 = <class 'tests.test_sqlalchemy_data_layer.person_list_2.<locals>.PersonList'>

    @pytest.fixture(scope="module")
    def register_routes_custom_qs(
            client,
            app,
            api_blueprint,
            custom_query_string_manager,
            person_list_2,
    ):
        api = Api(blueprint=api_blueprint, qs_manager_class=custom_query_string_manager)
        api.route(person_list_2, "person_list_qs", "/qs/persons")
>       api.init_app(app)

tests/test_sqlalchemy_data_layer.py:655: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
flask_combo_jsonapi/api.py:59: in init_app
    self.app.register_blueprint(self.blueprint)
/gnu/store/4rlh2lfp2p6hdpsdsvcd8b05hm97vw86-python-flask-2.1.1/lib/python3.9/site-packages/flask/scaffold.py:56: in wrapper_func
    return f(self, *args, **kwargs)
/gnu/store/4rlh2lfp2p6hdpsdsvcd8b05hm97vw86-python-flask-2.1.1/lib/python3.9/site-packages/flask/app.py:1028: in register_blueprint
    blueprint.register(self, options)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <Blueprint 'api'>, app = <Flask 'tests.conftest'>, options = {}

    def register(self, app: "Flask", options: dict) -> None:
        """Called by :meth:`Flask.register_blueprint` to register all
        views and callbacks registered on the blueprint with the
        application. Creates a :class:`.BlueprintSetupState` and calls
        each :meth:`record` callback with it.
    
        :param app: The application this blueprint is being registered
            with.
        :param options: Keyword arguments forwarded from
            :meth:`~Flask.register_blueprint`.
    
        .. versionchanged:: 2.0.1
            Nested blueprints are registered with their dotted name.
            This allows different blueprints with the same name to be
            nested at different locations.
    
        .. versionchanged:: 2.0.1
            The ``name`` option can be used to change the (pre-dotted)
            name the blueprint is registered with. This allows the same
            blueprint to be registered multiple times with unique names
            for ``url_for``.
    
        .. versionchanged:: 2.0.1
            Registering the same blueprint with the same name multiple
            times is deprecated and will become an error in Flask 2.1.
        """
        name_prefix = options.get("name_prefix", "")
        self_name = options.get("name", self.name)
        name = f"{name_prefix}.{self_name}".lstrip(".")
    
        if name in app.blueprints:
            bp_desc = "this" if app.blueprints[name] is self else "a different"
            existing_at = f" '{name}'" if self_name != name else ""
    
>           raise ValueError(
                f"The name '{self_name}' is already registered for"
                f" {bp_desc} blueprint{existing_at}. Use 'name=' to"
                f" provide a unique name."
            )
E           ValueError: The name 'api' is already registered for this blueprint. Use 'name=' to provide a unique name.

/gnu/store/4rlh2lfp2p6hdpsdsvcd8b05hm97vw86-python-flask-2.1.1/lib/python3.9/site-packages/flask/blueprints.py:305: ValueError
=============================== warnings summary ===============================
tests/conftest.py:12
  /tmp/guix-build-python-flask-combo-jsonapi-1.1.0.drv-0/source/tests/conftest.py:12: PytestDeprecationWarning: @pytest.yield_fixture is deprecated.
  Use @pytest.fixture instead; they are the same.
    @pytest.yield_fixture(scope="session")

tests/test_sqlalchemy_data_layer.py::test_get_list
  /tmp/guix-build-python-flask-combo-jsonapi-1.1.0.drv-0/source/tests/test_sqlalchemy_data_layer.py:181: SAWarning: relationship 'Person.computers_owned' will copy column person.person_id to column computer.person_id, which conflicts with relationship(s): 'Computer.person' (copies person.person_id to computer.person_id), 'Person.computers' (copies person.person_id to computer.person_id). If this is not the intention, consider if these relationships should be linked with back_populates, or if viewonly=True should be applied to one or more if they are read-only. For the less common case that foreign key constraints are partially overlapping, the orm.foreign() annotation can be used to isolate the columns that should be written towards.   To silence this warning, add the parameter 'overlaps="computers,person"' to the 'Person.computers_owned' relationship. (Background on this error at: https://sqlalche.me/e/14/qzyx)
    person_ = person_model(name="test")

tests/test_sqlalchemy_data_layer.py::test_delete_detail
  /tmp/guix-build-python-flask-combo-jsonapi-1.1.0.drv-0/source/tests/test_sqlalchemy_data_layer.py:187: SAWarning: DELETE statement on table 'person' expected to delete 1 row(s); 0 were matched.  Please set confirm_deleted_rows=False within the mapper configuration to prevent this warning.
    session_.commit()

-- Docs: https://docs.pytest.org/en/stable/warnings.html
=========================== short test summary info ============================
ERROR tests/test_sqlalchemy_data_layer.py::test_get_list_with_simple_filter_relationship_custom_qs_api
=================== 118 passed, 3 warnings, 1 error in 1.29s ===================

Tested on GNU Guix with Python 3.9.9 and the following dependencies:

[email protected] [email protected] [email protected] [email protected]
+ [email protected] [email protected] [email protected] [email protected]
+ [email protected] [email protected] [email protected]

For example, let's say, we have this resource heirarchy event -> session -> track, location

And I want to fetch them all, my query would be

/v1/events/123?include=session.track,session.location

This should be possible, as evident here http://jsonapiplayground.reyesoft.com/v2/stores/1?include=books.author,books.chapters

But the library implementation overrides the previous include by latest include, so it will only include session.location, and not session.track

There we can configure:

• setup (probably using python-poetry: https://python-poetry.org/docs/pyproject/)
• black formater: https://black.readthedocs.io/en/stable/pyproject_toml.html