geneontology / api-gorest Goto Github PK

I am not so sure if there are only 3500 models after https://s3.amazonaws.com/geneontology-public/gocam/gocam-models.json
The live api seems to return correct number https://api.geneontology.cloud/models/

Maybe the cache not updating

The live Api getModels

Recently, with our migration to EC2 and move to a smaller machine, we've come to understand that some queries coming through the GO-CAM API to the SPARQL endpoint and both voluminous (possibly crawling bots) and are no longer able to meet the query timeout.

An example of the main problematic query is https://gist.github.com/kltm/19e2d12cde2aa2e1f3f92cebdf16d937
Note that the code for this is https://github.com/lpalbou/api-gorest-2020/blob/aee0b9bd1e8b6c7ea1c815cfd70e2f1972deb0d7/queries/sparql-gp.js#L67; we'll probably want to

After some discussion on the software call, steps here may be:

• determine optimizations for the query
  • direct optimization
  • splitting, with coordination from the UI
• otherwise, explore caching in the API
• Test that fixes are producing same results as current, or that we want to change current

Secondarily, we'll want to make sure the devops processes are clear to roll this out into production without hiccups. (May want to try just redeployment first?) TBD. (Caveat here that we may run into other such queries. Good to get our practices down and document how we do this.)

@balhoff Already has some very (very) promising ideas here.

Tagging @tmushayahama @dustine32 @sierra-moxon @cmungall @vanaukenk @balhoff .

Sometimes the api doesn't give back (23% of the time). But there is CORS so I cannot tell you the actual error. This is an example of https://api.geneontology.cloud/models/pmid/ . Not only pmid but others too.

Also this might be a good exercise to check how this api works while Laurent is here
tagging @dustine32 @sierra-moxon @lpalbou

Noting that there is a lot of similarity with #3, but that this is a separate issue.

Recently, with our migration to EC2 and move to a smaller machine, we've come to understand that some queries coming through the GO-CAM API to the SPARQL endpoint are no longer able to meet the query timeout.

An example of the main problematic query is https://github.com/lpalbou/api-gorest-2020/blob/aee0b9bd1e8b6c7ea1c815cfd70e2f1972deb0d7/queries/sparql-models.js#L219
Note that the code for this is https://github.com/lpalbou/api-gorest-2020/blob/aee0b9bd1e8b6c7ea1c815cfd70e2f1972deb0d7/queries/sparql-gp.js#L67; we'll probably want to

We're still looking at how to proceed, but the steps here may be:

• determine if optimizations for the query are possible
  • direct optimization
  • splitting, with coordination from the UI
• otherwise, explore caching in the API
• test that fixes are producing same results as current, or that we want to change current

Secondarily, we'll want to make sure the devops processes are clear to roll this out into production without hiccups. (May want to try just redeployment first?) TBD. (Caveat here that we may run into other such queries. Good to get our practices down and document how we do this.)

@balhoff Would it be possible for you to look at the SPARQL for this one as well?

Tagging @tmushayahama @dustine32 @sierra-moxon @cmungall @vanaukenk @balhoff .