Having just decided to move forward with this repository and moved it into aiidateam, I'm now going to be fickle and ask "why are we using REST instead of GraphQL?"

From the (naturally biased) opinion of graphql site, this is why it is better than REST: https://www.howtographql.com/basics/1-graphql-is-the-better-rest/

Particularly the new /querybuilder endpoint feels like essentially a bespoke implementation of graphql, but lacking things like pagination, typing etc.

Note there has already been a trial version of this in https://github.com/dev-zero/aiida-graphql, using https://pypi.org/project/strawberry-graphql. At the time a drawback was that in only supported python 3.7+, but this is no longer an issue since aiida-core only supports this.

This is certainly not to say I am insisting we do move to graphql, but I think it's good to have some "due diligence" here to say we have considered it. Particularly now since we are entirely replacing the existing REST framework.

Here's some youtube vids (the best source of research 😆) with a more balanced look:

• https://www.youtube.com/watch?v=S1wQ0WvJK64 (Why I Don’t Use GraphQL Anymore)
• https://www.youtube.com/watch?v=AYZOHt6kz6Y (GraphQL vs REST for Side Projects)

Essentially I think it comes down to: GraphQL can be more complex to implement on the backend, but can be more powerful on the frontend (for clients).
So, do we except the outcome being a less powerful API, but with the benefit of (probably) easier development/maintenance? (quite possibly)

As well as modal authorization levels (for getting, posting, etc), we should consider how rate limiting can be implemented, e.g. so someone can't DDoS your REST server.

From https://stackoverflow.com/a/65491442/5033292, https://github.com/laurentS/slowapi looks like a possible solution (although it is only in alpha)

To keep things simple, the aiida-restapi package started without explicit handling of AiiDA profiles (it simply uses the default profile of the AiiDA installation where it is running).

This is fine for the moment, but will likely need to be generalized once this API starts being used in production.
Note that the decision of how to implement profile handling may depend on how we integrate authentication with the AiiDA side #5 .

It seems that, in general, the exact structure of the query string is not standardized.
For example, RFC3986 only mentions that

query components are often used to carry identifying information in the form of "key=value" pairs

Similar the latest RFC8820.

However, there is a W3C recommendation for encoding form data (application/x-www-form-urlencoded):

The name is separated from the value by = and name/value pairs are separated from each other by &

While the OpenAPI standard is not explicit on the name-value separator, all of the examples in the spec use =.

As pointed out by @chrisjsewell, fastapi relies on starlette which internally relies on urllib's parse_qsl function for parsing the query parameters. parse_qsl (part of the python standard library) also hardcodes the = separator.

This clashes with the filter operators from the aiida-core rest api, which allow e.g. for queries like

http://localhost:5000/api/v4/nodes?mtime>=2019-04-23

The aiida-core rest api has extremely liberal cross-origin resource sharing policies (just allowing any origin), see
https://github.com/aiidateam/aiida-core/blob/38080c44272a8a5ae24beee7109df9964450a464/aiida/restapi/run_api.py#L91-L93

We can mirror that
https://fastapi.tiangolo.com/tutorial/cors/?h=%20cors#use-corsmiddleware

Basically the fields you can return from an ORM entity (via QueryBuilder) are not the same as the fields available to create an ORM entity, e.g. here it suggests you can specify the id and user_id, which is not true:

equally the way the pydantic models are set up in models.py, IMO, feel a bit "off":

class Group(AiidaModel):
    """AiiDA Group model."""

    _orm_entity = orm.Group

    id: Optional[int] = Field(description="Id of the object")
    label: str = Field(description="Used to access the group. Must be unique.")
    type_string: Optional[str] = Field(description="Type of the group")
    user_id: Optional[str] = Field(description="Id of the user that created the node.")
    description: Optional[str] = Field(description="Short description of the group.")

Here, everything except for label is set as Optional, since this is the only required field to create a group. But this does not accurately describe what is stored in the database, and thus what you would return from a GET method, where things like id are not optional (i.e. it should never be None).

Finally, fields like user_id will be returned from e.g. QueryBuilder().append(Group, project=["**"]), because they are a field on the database table, but they will always be None when using from_orm, because they are not exposed as an attribute of the AiiDA ORM class.

FYI this is what I was talking about in #17 (comment) @ltalirz

Also note the /groups POST method is currently called create_user 😜

Currently, no PUT method is available to the processes endpoint.

User story

In AiiDAlab, we want to update the extra parameters of a ProcessNode. This can be done using AiiDA directly.

process.base.extras.set("builder_parameters", parameters)

For REST API, the PUT method is needed to do this.

@CasperWA has implemented https://github.com/aiidateam/aiida-optimade, which is essentially a specialised aiida server for optimade (mainly focused on querying for StructureData).
We should think if there is anything we can "share", be it actual code or just general ideas.

FYI @CasperWA you may be interested to see I took Optimade's filter grammar as an initial starting point for a querybuilder one: https://github.com/aiidateam/aiida-restapi/blob/master/aiida_restapi/static/filter_grammar.lark

@chrisjsewell writes

Some possible levels of access:

• Data retrieval (could have this allowed by default)
• Sensitive data retrieval ??
• Element (node, group, computer, etc) additions
• Element deletions
• Submitting processes

I personally think we can start just with keeping the routes that are open in the aiida-core REST API open, and protecting any functionality that modifies the DB.

Currently using the API requires a lot of boiler plate for the client. It would be great to at least have a streamlined library in Python.

Idea would be to have all mutating endpoints (PUT, POST and DELETE) return a 405 Method Not Allowed by default, unless a configuration option is explicitly set to allow mutations.

This is something that should happen gradually once we've agreed on the general architecture.

It's important not to do this too late in order to make sure our design is capable of replicating the old REST API's behavior.

after #17

While testing manually we realized that errors like forgetting to load the AiiDA profile are reported as an internal server errors

We may want to include this info in the response (although this may be a security concern)

Now that I've snuck in the initial GraphQL implementation (#17) 😄, it's a bit of a misnomer to call this just restapi.
Possible names: aiida-webserver, aiida-webapi

@ltalirz @NinadBhat ?

https://twitter.com/fastapi/status/1633141880157356038?s=46&t=P4qqIUgyzKn1k2ZMRrNaJQ

hint this is where you setup your database connection pools

Probably something like

# create adhoc ssl cert
openssl req -x509 -newkey rsa:4096 -nodes -out cert.pem -keyout key.pem -days 365 
uvicorn --ssl-keyfile key.pem  --ssl-certfile cert.pem aiida_restapi:app

Currently when submitting a process and the inputs are incorrect (for example label is missing, which shouldn't really even be required), the error message is not very useful.
We simply get a

422 Unprocessable Entity
{'detail': [{'loc': ['body', 'label'], 'msg': 'field required', 'type': 'value_error.missing'}]}

It should ideally tell what data is missing.

Integrate running of the REST API into the verdi cli, essentially replacing the verdi restapi command.

Since we're using uvicorn here it's not quite clear to me whether this is straightforward; also this currently means removing access to the old restapi, i.e. we can only do this once we're at feature parity.

This is not blocking for any use cases and also does not affect architectural considerations; thus low priority.

Reimplementing the entity models (node, user, ...) from scratch is quite straightforward with pydantic, and probably the quickest way to start.

Ideally, at some point however we should try to infer the models, either from AiiDA's public ORM interface or from the underlying django/sqlalchemy DB models.

Note: An intermediate solution may be to embed the pydantic models in the ORM in some way in order to at least keep these two close together.

and also the tests (see #4)

from a call to the current API:

{
  "data": {
    "available_endpoints": [
      "calcjobs        GET,HEAD,OPTIONS     /api/v4/calcjobs/<id>/input_files/",
      "calcjobs        GET,HEAD,OPTIONS     /api/v4/calcjobs/<id>/output_files/",
      "computers       GET,HEAD,OPTIONS     /api/v4/computers/",
      "computers       GET,HEAD,OPTIONS     /api/v4/computers/<id>/",
      "computers       GET,HEAD,OPTIONS     /api/v4/computers/page/",
      "computers       GET,HEAD,OPTIONS     /api/v4/computers/page/<int:page>/",
      "computers       GET,HEAD,OPTIONS     /api/v4/computers/projectable_properties/",
      "groups          GET,HEAD,OPTIONS     /api/v4/groups/",
      "groups          GET,HEAD,OPTIONS     /api/v4/groups/<id>/",
      "groups          GET,HEAD,OPTIONS     /api/v4/groups/page/",
      "groups          GET,HEAD,OPTIONS     /api/v4/groups/page/<int:page>/",
      "groups          GET,HEAD,OPTIONS     /api/v4/groups/projectable_properties/",
      "nodes           GET,HEAD,OPTIONS     /api/v4/nodes/",
      "nodes           GET,HEAD,OPTIONS     /api/v4/nodes/<id>/",
      "nodes           GET,HEAD,OPTIONS     /api/v4/nodes/<id>/contents/attributes/",
      "nodes           GET,HEAD,OPTIONS     /api/v4/nodes/<id>/contents/comments/",
      "nodes           GET,HEAD,OPTIONS     /api/v4/nodes/<id>/contents/derived_properties/",
      "nodes           GET,HEAD,OPTIONS     /api/v4/nodes/<id>/contents/extras/",
      "nodes           GET,HEAD,OPTIONS     /api/v4/nodes/<id>/download/",
      "nodes           GET,HEAD,OPTIONS     /api/v4/nodes/<id>/links/incoming/",
      "nodes           GET,HEAD,OPTIONS     /api/v4/nodes/<id>/links/incoming/page/",
      "nodes           GET,HEAD,OPTIONS     /api/v4/nodes/<id>/links/incoming/page/<int:page>/",
      "nodes           GET,HEAD,OPTIONS     /api/v4/nodes/<id>/links/outgoing/",
      "nodes           GET,HEAD,OPTIONS     /api/v4/nodes/<id>/links/outgoing/page/",
      "nodes           GET,HEAD,OPTIONS     /api/v4/nodes/<id>/links/outgoing/page/<int:page>/",
      "nodes           GET,HEAD,OPTIONS     /api/v4/nodes/<id>/links/tree/",
      "nodes           GET,HEAD,OPTIONS     /api/v4/nodes/<id>/repo/contents/",
      "nodes           GET,HEAD,OPTIONS     /api/v4/nodes/<id>/repo/list/",
      "nodes           GET,HEAD,OPTIONS     /api/v4/nodes/download_formats/",
      "nodes           GET,HEAD,OPTIONS     /api/v4/nodes/full_types/",
      "nodes           GET,HEAD,OPTIONS     /api/v4/nodes/full_types_count/",
      "nodes           GET,HEAD,OPTIONS     /api/v4/nodes/page/",
      "nodes           GET,HEAD,OPTIONS     /api/v4/nodes/page/<int:page>/",
      "nodes           GET,HEAD,OPTIONS     /api/v4/nodes/projectable_properties/",
      "nodes           GET,HEAD,OPTIONS     /api/v4/nodes/statistics/",
      "processes       GET,HEAD,OPTIONS     /api/v4/processes/<id>/report/",
      "processes       GET,HEAD,OPTIONS     /api/v4/processes/projectable_properties/",
      "querybuilder    GET,HEAD,POST,OPTIONS /api/v4/querybuilder/",
      "server          GET,HEAD,OPTIONS     /api/v4/",
      "server          GET,HEAD,OPTIONS     /api/v4/server/",
      "server          GET,HEAD,OPTIONS     /api/v4/server/endpoints/",
      "users           GET,HEAD,OPTIONS     /api/v4/users/",
      "users           GET,HEAD,OPTIONS     /api/v4/users/<id>/",
      "users           GET,HEAD,OPTIONS     /api/v4/users/page/",
      "users           GET,HEAD,OPTIONS     /api/v4/users/page/<int:page>/",
      "users           GET,HEAD,OPTIONS     /api/v4/users/projectable_properties/"
    ]
  },
  "method": "GET",
  "path": "/api/v4",
  "query_string": "",
  "resource_type": "Info",
  "url": "https://15.188.110.176:5000/api/v4",
  "url_root": "https://15.188.110.176:5000/"
}

It might be ok to store the hashed passwords of AiiDA users in the AiiDA DB - but perhaps in a different table (a bit like what is done for the Computers an the Authinfo).

To discuss

I installed the aiida-restapi package in a new python virtual environment. Then, I tried to start the REST API with uvicorn aiida_restapi:app command. I got the following error:

ModuleNotFoundError: No module named 'jose'

I installed manually this module with pip install python-jose.
The next error was:

ModuleNotFoundError: No module named 'passlib'

which I solved with pip install passlib.

The final error was:

RuntimeError: Form data requires "python-multipart" to be installed.

solved with pip install python-multipart. Then finally it worked.

The input dictionary is handled by the substitute_node function to convert node UUIDs into the corresponding nodes by loading them. However, this does not support input namespaces as it will only serialize top-level keys and not recurse into nested dictionaries.

Obviously the time to e.g. retrieve a Node etc is very dependent on aiida-core, plus postgres setup, plus hardware etc.
But... it would be nice if we had some basic feedback on the magnitude of the times related to different endpoints, and flag any that are particularly problematic.

For example, in aiida-core I set up: https://aiidateam.github.io/aiida-core/dev/bench/ubuntu-18.04/django/

Perhaps, there is some pydantic specific tool to achieve this?

I believe this is relatively straightforward to implement.

Could decide to do this during GSOC (but not mandatory)