itisfoundation / osparc-simcore-clients Goto Github PK

View Code? Open in Web Editor NEW

1.0 9.0 4.0 536 KB

python client for osparc-simcore public API

Home Page: https://itisfoundation.github.io/osparc-simcore-clients/

License: MIT License

Python 56.91% Makefile 8.06% Dockerfile 1.10% Shell 5.33% Jupyter Notebook 27.33% HTML 1.27%

api-wrapper osparc python

osparc-simcore-clients's Introduction

oSPARC-Simcore-Clients

To get started using the tools developed in this repository have a look at the Documentation.

For more in depth knowledge concerning the development of the repository, take a look at

For reporting an issue, use our issue tracker

Development notes

These notes are intended for people who want to contribute to the development of this repository.

The different clients for the oSPARC public API are generated using the openapi-generator-tool. The generated files should not be changed manually! Instead proposed changes should be implemented directly on the openapi-generator-repository. See also templates and customization.

Workflow

See the different clients/<language>/README.md for the workflows for generating the different clients. Here is the workflow which all clients have in common:

To generate a client one needs two "ingredients":
1. The openapi specification which is a json file located in api/openapi.json. This generated in osparc-simcore and then moved here.
2. The openapi-generator-tool. The exact docker image of this tool to use is specifies in scripts/common.Makefile.

Documentation

Each client (corresponding to a directory clients/<language>) should have a make recipy called install-doc which installs packages needed to generate the documentation for that client, as well as a make recipe docs which generates the documentation. All of these recipes are called from the root's docs recipe, which is what is called when the website is published.

Repos which depend on this one

Here is an inexhaustive list which have this repo as a dependency. So changing stuff here might break stuff there:

osparc-simcore-clients's People

Contributors

Stargazers

Watchers

Forkers

helen-research githk bisgaard-itis wvangeit

osparc-simcore-clients's Issues

Fix abort multipart upload

What version of `osparc` are you using?

Long story short

Expected behaviour

Actual behaviour

Steps to reproduce

Your environment

python client wrapper based on generator

When API exposes job pagination, I would like to have a compatible client function JobsAPI.list_jobs that includes pagination parameters (defaulted to compatible values) and an extra function that JobsAPi.iter_jobs to seamlessly iterate all jobs as with https://github.com/ITISFoundation/osparc-simcore/blob/master/packages/pytest-simcore/src/pytest_simcore/helpers/utils_docker_registry.py#LL48C5-L66C74

Configuration defaults captures `OSPARC_*`

User Story

The Configuration object should by default capture specific environment variables. such that

from osparc import ApiClient, UsersApi, Confirmation
import os

assert Configuration() = Configuration(
    host=os.environ["OSPARC_API_HOST"],
    username=os.environ["OSPARC_API_KEY"],
    password=os.environ["OSPARC_API_SECRET"],
)

Definition of Done

Open a jupyter-math service in osparc (see ITISFoundation/jupyter-math#29, ITISFoundation/osparc-simcore#5695)

from osparc import ApiClient, UsersApi, Confirmation
import os

cfg = Configuration()
print(cfg.host)

with ApiClient(cfg) as api_client:
    users_api = UsersApi(api_client)
    profile = users_api.get_my_profile()
    print(profile)

Fixe release workflow with github-actions

Define workflow that includes

release to TestPypi repo
release to Pypi repo
versioning with bump2version, when to tag?
how to sync SDK and API versions?

🚚 Reorganize client to improve workflows

Description

In order to have a great workflow in place, where maintainability is as low as possible and where we can easily support new types of clients, we need to restructure the repo.

Tasks

Beta Give feedback

Rename repo
Reorganize repo so the current python client lives in a subdir with fully functional tools
Move tooling for generation of the client from #ITISFoundation/osparc-simcore to this repo
Implement a proper workflow (to ease maintenance) for documentation (automatically generated + manually written)
Setup github actions
Options

Expose osparc python package within the sparc.client

Here's a list of tasks which should be achieved:

Tasks

Beta Give feedback

docstring of jobs is wrong

What version of `osparc` are you using?

0.6.3

Long story short

the docstring of jobs says there are optional parameters, but the method does not allow for more than solver_key and solver_version

Expected behaviour

correct docstring or modified method

Actual behaviour

wrong doctring

Steps to reproduce

pip install osparc

with osparc.ApiClient(cfg) as api_client:
    solvers_api = osparc.SolversApi(api_client)
    solvers_api.jobs(id, version)

Your environment

Introduce proper logging

Currently there are print statements spread out in the client. These need to be converted into proper a proper logging system

Ensure jobs are deleted after e2e client

Long story short

The following should be added to the e2e client tests:

Tasks

Beta Give feedback

Make sure to stop remaining jobs after tests are done
Delete jobs after tests are done (in after_script)
Use only clients uploaded to pypi and testpypi
make sure to pip-freeze to artifacts dir
Options

list_jobs entrypoint redirects to internal function for paged function

What version of `osparc` are you using?

0.6.3

Long story short

pip install osparc
then doing

with osparc.ApiClient(cfg) as api_client:
    solvers_api = osparc.SolversApi(api_client)
    solvers_api.list_jobs(...)

the doc string of list_jobs tells me to look at get_jobs_page for the paginated version. then looking at get_jobs_page tells me it is internal.
The proper paginated method is called jobs

Expected behaviour

the docstring should bring me to the correct paginated method.

Actual behaviour

docstring is wrong

Steps to reproduce

Your environment

Clean up after introducing e2e testing.

Here's the list of tasks for this ticket:

Tasks

Beta Give feedback

I would make the json config more self-explanatory. It adds affordable complexity and it is easily extendable. See blow the tasklist for an example.
convert to using a pytest.ini file format instead of a pyproject.toml file. Reason: the toml file is usually used for configuring an entire python project and hence is usually checked into the repo. In our case the file is only supposed to configure a single pytest session.
add all command line options to pytest.ini.
make sure to call python -m pytest in bash script running the tests
Parametrize e2e tests so a different branch/repo can be checked out.
Check if toml/ini file exists in install_osparc_python_client.py. Also adapt make recipe to directly install client.
Improve logging. Add logging config to pytest.ini, both for cli logging and for file logging.
make sure the post process job fails if any of the individual jobs fail
Options

[
  {
    "name": "master": {
      "url": "api.osparc-master.speag.com",
      "is_compatible": true,
      "url": "api.osparc-staging.speag.com",
      "is_compatible": false,
      "url": "api.osparc.speag.com",
      "is_compatible": false,
      "url": "api.staging.osparc.io",
      "is_compatible": false,
      "url": "api.osparc.io",
      "is_compatible": false
    }
  },
  {
    "name": "production": {
      "url": "api.osparc-master.speag.com",
      "is_compatible": true,
      "url": "api.osparc-staging.speag.com",
      "is_compatible": true,
      "url": "api.osparc.speag.com",
      "is_compatible": true,
      "url": "api.staging.osparc.io",
      "is_compatible": true,
      "url": "api.osparc.io",
      "is_compatible": true
    }
  }
]

[
  {
    "name": "master": {
      "url": "api.osparc-master.speag.com",
      "is_compatible": true,
      "url": "api.osparc-staging.speag.com",
      "is_compatible": false,
      "url": "api.osparc.speag.com",
      "is_compatible": false,
      "url": "api.staging.osparc.io",
      "is_compatible": false,
      "url": "api.osparc.io",
      "is_compatible": false
    }
  },
  {
    "name": "production": {
      "url": "api.osparc-master.speag.com",
      "is_compatible": true,
      "url": "api.osparc-staging.speag.com",
      "is_compatible": true,
      "url": "api.osparc.speag.com",
      "is_compatible": true,
      "url": "api.staging.osparc.io",
      "is_compatible": true,
      "url": "api.osparc.io",
      "is_compatible": true
    }
  }
]

Multi-version documentation

Currently only master documentation is published in pages

released package does not sync with published doc
do not keep doc of previous package releases

Add test for logstreaming

User Story

We need to have a test running against sim4life.io which tests the log streaming which will warn us in case i fails.

Example

Definition of Done

Ensure python client e2e logstreaming test only runs with client version >= 0.6.5

What version of `osparc` are you using?

Long story short

It seems the logstreaming e2e test uses the delete_file endpoint. That one is only supported by the client v >= 0.6.5.

Expected behaviour

Actual behaviour

Steps to reproduce

Your environment

Upload limit

User Story

Add an upload limit, so that users can only upload a given amount of data to the osparc platform.

release of osparc python client

Add a tutorial which shows the functionality developed during the Watermelon sprint

Tasks

Beta Give feedback

Public basic tutorials (2), add to osparc-simcore-clients/
tests backwards compatibility?
release1: files+solver resources (not studies)
release2: studies
Options

Add appropriate retry strategy to improve stability

User Story

Add retry strategy when server returns one of the following status codes: 429,503,504
Due to recent changes these are now returned by the api server when the backend is overloaded and the user needs to retry

Example

Definition of Done

Documentation

Minor. This entry appears twice

Introduce longer timeout threshold for multipart upload

What version of `osparc` are you using?

Long story short

When running a lot of multipart uploads concurrently, the completion request to S3 times out.

Expected behaviour

Actual behaviour

Steps to reproduce

Your environment

Add test to run jupyter notebooks

Jupyter notebooks are currently auto-generated from markdown doc. Verify that every code cell runs "as is" (e.g. no variables missing, etc). Ideally this should be an automatic job in the CI

Broken link in the readme page

In the readme section, when trying to open the "Basic tutorial option", it loads the following link:

https://itisfoundation.github.io/osparc-simcore-clients/#/clients/python/artifacts/docs/BasicTutorial

The webpage states an error message of 404 so the web is not being correctly load/generated.

DeprecationWarning with HTTPResponse.getheade on osparc/rest.py:47

site-packages/osparc/rest.py:47: DeprecationWarning: HTTPResponse.getheader() is deprecated and will be removed in urllib3 v2.1.0. Instead use HTTResponse.headers.get(name, default).
    return self.urllib3_response.getheader(name, default)

Create sparc.client tutorial which demonstrates cross-core functionalities

The user story is here: #53

That user story is probably not completely relevant anymore. We'll update this issue once all the Cores have agreed on the tutorial.

Move logstreaming from `dev` mode to `production` mode

User Story

Example

Definition of Done

Basic tutorial 2: Pseudo-code for user story involving CWL @elisabettai

As a user, I would like to do this using the osparc Python package (called osparc later on for simplicity):

User story

I found some excel files on the SPARC Portal here: https://sparc.science/datasets/273?type=dataset&datasetDetailsTab=files&path=files%2Fderivative
Download one/all of them using osparc (I am assuming the file entrypoints can do that)
Then I want to make some analysis/plots out of those data. So I check if there is some solver that accept as input xlsx file as input (This I guess can be achieved by querying all the solvers ports. think we don't have any service of this type.
But I see that there's a solver called osparc python runner (see here how to use it.).
I then develop my python code locally to plot/analyse the data and output a figure in the output ports, I package it according to how the osparc python runner requires.
I give my code as the input to the osparc python runner
I run then the job with the osparc python runner and I get the job status and logs (or render them on the screen while running, but I doubt this is possible already). I want to see if the job ran successfully
Once I get a successful run, I upload the output figure to the system (is this the purpose of /v0/files/content?

cannot download binary files

@newton1985 I'm having some issues downloading the results though- I'll attach what I'm seeing here

Improve release workflow

Possible workflows provided changes in OAS of osparc-simcore public API

new osparc-simcore public API merges to master
- osparc-simcore/tests/public-api already tests against current master
- manual run tests/public-api/examples/sleeper.py?
new osparc-simcore public API pre-released to staging
- set version as beta with commit-id?
- test repo version against staging?
- publish to pypi
- test published version against staging? and all previous versions as well
- manual test run tests/public-api/examples/sleeper.py?
- announce release?
new osparc-simcore public API released to production
- dump version as major/minor/patch ...
- test repo version against production?
- publish to pypi
- test published version against production? and all previous versions as well
- manual test run tests/public-api/examples/sleeper.py?
- announce release?
Where to announce pre/release?:
- osparc-simcore CHANGELOG
- support-channel (see NOTES)

Baklava issue

Add tutorial specific for files - also to test corner cases of the system.

Nice to have

Beta Give feedback

Hash/FileId which can be computed in client
Options

Write basic tutorial in "sparc.client-language" in the [tutorial](https://github.com/nih-sparc/sparc.client/blob/main/docs/tutorial.ipynb)

This was pointed out at the Tech call (July 13) by @mguidon.

The current plan is to have the documentation on Readme. We can get help in moving/copying our documentation there. See this wrike ticket here (https://www.wrike.com/open.htm?id=920664113)
They asked for ideas of a cross-core tutorial (https://docs.google.com/spreadsheets/d/1WZIjBVo2wyQyxiPP7KzEmchBf_9UX48hvaielX1kST4/edit#gid=944422487) until Wednesday, July 19th such that it can be discussed in the PI meeting

This task is about Point 1.
fyi @pcrespov

Tasks

Beta Give feedback

add osparc-tutorial nih-sparc/sparc.client#21

documentation
add above tutorial to Readme in Markdown format
Options

itisfoundation / osparc-simcore-clients Goto Github PK

osparc-simcore-clients's Introduction

oSPARC-Simcore-Clients

Development notes

Workflow

Documentation

Repos which depend on this one

osparc-simcore-clients's People

Contributors

Stargazers

Watchers

Forkers

osparc-simcore-clients's Issues

What version of osparc are you using?

Long story short

Expected behaviour

Actual behaviour

Steps to reproduce

Your environment

User Story

Definition of Done

Description

Tasks

Tasks

What version of osparc are you using?

Long story short

Expected behaviour

Actual behaviour

Steps to reproduce

Your environment

Long story short

Tasks

What version of osparc are you using?

Long story short

Expected behaviour

Actual behaviour

Steps to reproduce

Your environment

Tasks

User Story

Example

Definition of Done

What version of osparc are you using?

Long story short

Expected behaviour

Actual behaviour

Steps to reproduce

Your environment

User Story

Tasks

User Story

Example

Definition of Done

What version of osparc are you using?

Long story short

Expected behaviour

Actual behaviour

Steps to reproduce

Your environment

User Story

Example

Definition of Done

User story

Nice to have

Tasks

Recommend Projects

Recommend Topics

Recommend Org

What version of `osparc` are you using?

What version of `osparc` are you using?

What version of `osparc` are you using?

What version of `osparc` are you using?

What version of `osparc` are you using?