vianneymi / monggregate Goto Github PK

Cf #68

Test Matrices

• OS
• python versions
• pydantic versions
• optional dependecies or not
• pymongo/motor versions

• Create alias
• Add support for inner and right joins
• Retrieve reference collection from pipeline name

This should only be required when db is set

General issue to address

• #7
• #8
• #10 and
• #11

Takes into account that it's not an apple to apple comparison as

• some mirror methods accept additional keyword arguments representing keys in the documents to query
• the aliases are not taken into account yet in the mirror methods
• some mirror methods accept positional arguments

• #48
• Add tests

• Allow hybrid (SQL/NoSQL) joins by passing name in join (i.e allow storing matches in an array as in regular Mongo Lookup)
• In SQL tabular mode, prevent key names collision

SortByCount only takes str (field path) as of now when it could take expressions
Ex :

{ $sortByCount: { lname: "$employee.last", fname: "$employee.first" } }

This breaks

pipeline = Pipeline(collection="movies")
pipeline.match(
    query={
        "title":"A Star Is Born"
    }
).sort(
    by="year",
    ascending=True
)

while it shouldn't.

It is because a branch is missing validates_boolens:

elif descending is None and isinstance(ascending, bool):
    descending = not ascending

If left at None, the generated query will fail

When passing a field name with by and a negative direction with descending=True, the sort is still performed in ascending order

• Move away from using a root_validator, instead use individual or compound validator on field or set of fields, leverage full direct capabilities of pydantic this will allow to make statement private again
• Make statement private again

• alias generator #40
• remove automatic transformation of objects to statement
• improve transformation of objects to statement (deeply nested cases)
• improve expressions
• Make collection really optional in pipeline
• Simplify double mode EXPORT vs RUN
• pipeline list #25
• Improve package structure

I'll use this issue to adress multiple issues that wil be referenced below

pymongo should be required only for pipeline to directly executable.
Currently instantiating a pipeline in a environment that does not contain pymongo crashes

This will allow to build more complex aggregation by referencing other interface objects that will automatically be translated into statements.
E.g. passing a pipeline instance to a lookup statement, or a an accumulator operator to a group statement

Make sure that the requirements really capture the package requirements and that the package can be built in newly environments with packages respecting the ranges or set versions described in requirements

• Define alias generator in BaseClass
• Deprecate aliases defined manually using pydantic.Field
• Check that nothing break
• Whenever possible use .dict() or future serialize() function to generate statements

If some users need the package for older versions of python, I might consider create a backport. I just need to know if it is worth "the effort"

References

• https://www.mongodb.com/developer/languages/python/python-quickstart-aggregation/
• https://stackoverflow.com/questions/12702080/mongodb-explain-for-aggregation-framework
• https://choosealicense.com/licenses/mit/
• https://epydoc.sourceforge.net/manual-fields.html#module-metadata-variables
• https://twitter.com/__mharrison__/status/1481406568222453761

Implement subset of operators to at least cover what is used in the below tutorial:
https://www.mongodb.com/developer/languages/python/python-quickstart-aggregation/#your-first-aggregation-pipeline

That is :

• $size
• $gt
• $sum
• $push
• $type

• by should not be mandatory in group stage and when provided should be added to the final query

• To be distinct from current expression type
• This expression class should overload python operators ...
• ... but this overloading should be optional and controlled through the config class

Currently, only the stage classes are tested but not their mirror methods in the pipeline class

For attributes that are typed as being expressions, we should provide an alternative type for simple cases and to enforce validation.
Ex:

class Limit(BaseModel):
   value : Expression = 0

cannot have a constraint on value because of the Expression type
however, this should work

class Limit(BaseModel):
   value : int | Expression = Field(gt=0) 

The check of combination of arguments is wrong

        # Check combination of arguments
        if right and left_on and right_on and \
            not(let or pipeline is None):
            # in a subquery to select all on the foreign collection
            # pipeline can be an empty list which is falsy
            type_ = "simple"

        elif let and pipeline is not None and not(left_on or right_on):
            type_ =  "uncorrelated"

        elif let and pipeline is None and left_on and right_on:
            type_ = "correlated"

        else:
            raise TypeError("Incompatible combination of arguments")

        return type_

The stage classes leveraging pydantic use field aliases for attribute names. We should also allow this in their mirror pipeline functions

This is for testing pull requests on GitHub

• Rename current operators function in Search class so that its clear that they are constructors
• Create alternative/mirror functions pipelinizable functions inside Search class. That is functions that at the end return self so that they can be chained. Those functions should insert clauses in a compound operator
• #85

• Search for contributors
• #79
• Create changelog
• Create GitHub releases
• #80
• Perform thorough code reviews
• Create a dedicated doc website

pydantic v2 broke autocompletion in the package because of the way BaseModel is imported now

• UnionWith
• Unset
• Search
• SearchMeta

• #98

• Interface a couple more operators
• Clarify how expressions can and should be used
• Create a class hosting all operators (sort of equivalent of pipeline with stages but for operators)
• default types for expressions #27
• create a type alias for statements #29

Setup package so that mypy understands that it is typed.

Exemple:

.filter should be recognized in the above screenshot

Add tests to at least ensure that the statements are those expected

• Copy what was done later on when interfacing Atlas Search. That is adding a description of the operator/stage
• Add link to official mongo doc