Git Product home page Git Product logo

blueos-extensions-repository's Introduction

BlueOS-Extensions-Repository

Warning This is still very experimental and subject to changes!

This is a repository for metadata of BlueOS Extensions.

For publishing a new extension, open a pull request to this repository with the following structure:

Data in this repository

/repos/yourcompany/yourextension/metadata.json

{
    "name": "The Name of Your Extension",
    "website": "https://your.extension.website.com/",
    "docker": "your-dockerhub-user/your-extension-docker",
    "description": "A brief description of your extension. This will be shown in the store card."
}

/repos/yourcompany/yourextension/company_logo.png Your company logo

/repos/yourcompany/yourextension/extension_logo.png Your extension logo

Data in dockerhub

Additionally, we have versioned data. This data should be in each of your dockerhub tags, and use the following format:

LABEL version="1.0.0"
LABEL permissions='{\
  "ExposedPorts": {\
    "80/tcp": {}\          // we have a server at port 80
  },\
  "HostConfig": {\
    "PortBindings": {\
      "80/tcp": [\         // our server at port 80 is automatically bound to a free port in the host
        {\
          "HostPort": ""\
        }\
      ]\
    }\
  }\
}'
LABEL authors='[\
    {\
        "name": "John Doe",\
        "email": "[email protected]"\
    }\
]'
LABEL company='{\
  "about": "brief description",\
  "name": "Company/Person Name",\
  "email": "[email protected]"\
}'
LABEL readme="https://raw.githubusercontent.com/username/repo/{tag}/README.md"
LABEL links='{\
  "website": "https://...",\
  "support": "mailto:[email protected]",\
  "documentation": "https://docs.company.com/cool-extension/",\
}'
LABEL type="example"
LABEL tags='[\
  "positioning",\
  "navigation"\
]'
  • version is the name of the current tag, which we expect to be a valid semver.
  • permissionsis a json file that follows the Docker API payload for creating containers.
  • authors is a json list of authors of your extension
  • company is a json with information about the maintainer responsible for providing new versions
  • readme is a URL to a markdown-based README file for the extension
  • links is a collection of additional useful/relevant links
  • type is a primary categorisation of the extension, and should be one of:
    • "device-integration"
    • "example"
    • "theme"
    • "other"
  • tags is a collection of relevant tags for filtering, which should be lowercase alpha-numeric with dashes
    • limit of 10 per extension

Some additional information and examples are available in the metadata documentation.

How this repo works

Every time this repo changes, a Github Action runs and goes through all the .json files in here. For each of them, it reaches out to dockerhub and fetches all the available tags, extracting the metadata in LABELS and crafting a complete manifest.json, which is stored in this repo's gh-pages branch.

There is also a website that gets generated, to show which extensions are currently available in the store.

Testing the website locally

# clone and enter the repo
git clone ...
cd BlueOS-Extensions-Repository
# install yarn if necessary (`brew install yarn` on mac)
sudo apt install -y yarn
# install Python dependencies
python -m pip install --upgrade poetry
python -m poetry install
# generate the manifest (takes some time)
python -m poetry run blueos_repository/consolidate.py
# make the manifest accessible to the local website
mv manifest.json website/public/
# install website dependencies
cd website
yarn install --frozen-lockfile
# generate and serve the website locally, in development mode
yarn dev
... # go to http://localhost:3000/ in your browser (may need to refresh if it's not working)

blueos-extensions-repository's People

Contributors

clydemcqueen avatar es-alexander avatar hasanthesyrian avatar joaomario109 avatar maphstr avatar martinbj-nortek avatar nicknothom avatar patrickelectric avatar raultrombin avatar rjehangir avatar tdubsfo avatar williangalvani avatar wlseb avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

blueos-extensions-repository's Issues

manifest: support hardware/integration metadata

Device integrations would benefit from having consistent ways of linking to information about the product being integrated, integration guides (perhaps we should prefer markdown but also allow PDFs), and things like CAD file links for mounting hardware, which we could then present nicely, and allow locally downloading and storing with the extension (see bluerobotics/BlueOS#1640) as appropriate (like we're planning to do for READMEs).

I'm thinking something like

LABEL hardware='{\
    "integration": "https://docs.company.com/dvls/integration_guide.md",\
    "components": [\
        {\
            "product": "My DVL 100",\
            "manufacturer": "My Company",\
            "details": "https://store.com/my-dvl-100",\
            "image": "https://store.com/images/my-dvl-100.png",\
            "purpose": "sensor for velocity measurement/positioning",\
            "groups": [1]\
        },\
        {\
            "product": "My DVL 250",\
            "manufacturer": "My Company",\
            "details": "https://store.com/my-dvl-250",\
            "image": "https://store.com/images/my-dvl-250.png",\
            "purpose": "sensor for velocity measurement/positioning",\
            "groups": [2]\
        },\
        {\
            "product": "Megamount3000",\
            "manufacturer": "3D printed",\
            "details": "https://printables.com/my-dvl-mount",\
            "image": "https://...",\
            "purpose": "attaching DVL to a vehicle",\
            "groups": [1, 2]\
        }\
    ]\
}'

consolidate: allow using existing manifest as a cache

Consolidation already takes quite a while, and it's generally unlikely that existing tags will have changed labels or a changed structure. The default behaviour should be to use the existing manifest file as a cache, and only fetch data for new tags, with the option to instead do a clean/full refresh and generate the entire file from scratch.

Something like a --cache command-line argument could be useful, where

  • if unspecified it uses the latest online manifest
  • if "none" it does a full refresh
  • if given a URL it tries to fetch it and use it, erroring out on failure to fetch
  • if given a local filename it tries to read it and use it, erroring out on failure to read

This should save a bunch of consolidation time, and also reduce unnecessary API usage.

Extension: Navigator Webassistant - Not showing on BlueOS bazaar

Issue:
The extensions-manager not show the new available extension;
Route http://blueos-wifi.local/kraken/v1.0/extensions_manifest also does not show the new extension;

from local test:
python3 -m poetry run blueos_repository/consolidate.py
Result:

fetching tags in patrickelectric/blueos-node-red
['1.0.2', '1.0.1', '1.0.0']
fetching labels for patrickelectric/blueos-node-red:1.0.2
fetching labels for patrickelectric/blueos-node-red:1.0.1
fetching labels for patrickelectric/blueos-node-red:1.0.0
fetching tags in patrickelectric/blueos-bb
['master']
master is not valid SemVer, ignoring it...
fetching tags in raulelektron/blueos_navigator_webassistant
['0.0.0', 'latest']
fetching labels for raulelektron/blueos_navigator_webassistant:0.0.0
Error status 404
latest is not valid SemVer, ignoring it...
fetching tags in williangalvani/blueos-extension-proxy
['v1.0.3', 'master', 'iframes', 'v1.0.2', 'nocache', 'websockets', 'v1.0.1', 'v1.0.0', 'v0.0.1']
fetching labels for williangalvani/blueos-extension-proxy:v1.0.3
master is not valid SemVer, ignoring it...
iframes is not valid SemVer, ignoring it...
fetching labels for williangalvani/blueos-extension-proxy:v1.0.2
nocache is not valid SemVer, ignoring it...
websockets is not valid SemVer, ignoring it...
fetching labels for williangalvani/blueos-extension-proxy:v1.0.1
fetching labels for williangalvani/blueos-extension-proxy:v1.0.0
fetching labels for williangalvani/blueos-extension-proxy:v0.0.1
Error status 404

Extension Idea: Rust Playground integration

Like jupyter lab for python, check viability of an extension for rust (Important to check if it can access the devices).
Rust-Playground

image

Rust-Playground - Source Code

Usefull content:
https://users.rust-lang.org/t/create-rustup-play-for-a-self-hosted-playground/13714/3

The official playground is [integer32llc/rust-playground 55](https://github.com/integer32llc/rust-playground). I have been able to get it running locally.

It requires [yarn 8](https://yarnpkg.com/lang/en/docs/install/) and [docker 1](https://docs.docker.com/engine/installation/) already installed.

git clone https://github.com/integer32llc/rust-playground
Build and run the frontend
cd rust-playground/ui/frontend
yarn
yarn run watch
Build and run the backend
cd rust-playground/ui
RUST_LOG=ui=debug PLAYGROUND_UI_ROOT=$PWD/frontend/build/ cargo run
Build the compiler containers and tools
Fair warning: this uses something like 16G of /var/lib/docker.

cd rust-playground/compiler
./build.sh
Once everything is running, visit [http://localhost:5000/ 5](http://localhost:5000/).

Improve CI checks

We should have automated CI checks for a pull request of a new extension, to confirm

  • that a valid image version is available at the specified location
  • that it includes the required dockerfile metadata fields
  • the validity of the URLs in the metadata
  • that the extension runs (on a test system, I suppose?)
  • the validity of the register_service endpoint (if there is one)
  • old/deprecated metadata variables and structures aren't being used

Extension Idea: Jupyter integration

Would be useful for people wanting to test out Python programs on their system, with a nice interface (e.g. for testing Navigator library functionalities).
-@RaulTrombin

Maybe try to integrate jupyter lab (rather than just notebook), on top of a volume/bind for persistent storage?

manifest: support time-based patch history

This kind of thing could significantly reduce download requirements for users accessing the store, especially as our number of extensions scales up. Much easier to scale fast if the appropriate infrastructure has been designed in from early on :-)

Patch history is already possible to some extent with the git history of the gh-pages branch, especially if we enforce an ordering for the extensions and versions, but a content-aware JSON patch history could likely be a fair amount more efficient.

consolidate: improve robustness

Currently it's possible to break the consolidation cycle by adding a typo or problem to the Dockerfile of a new version of an accepted extension. That's problematic and relies excessive helicopter parenting of the repo.

The consolidation script should gracefully handle extension fetch failures, either by skipping the malformed data, and/or entirely skipping the version with the malformed Dockerfile. An error message should be printed out/logged whenever a failure/skip occurs.

Refactor blueos-repository for clarity and efficiency

Revise blueos-repository to remove manager like code in favor of a modular class system for extension fetching and detail extraction. This change allows for parallel retrieval of all extension details, obviating the need for the current run function that centrally manages manifest data collection and has intrinsically a high complexity.

manifest: update to match metadata documentation

  • We have new metadata recommendations that have not yet been implemented (e.g. the new type and tags fields, and some of the existing link fields being consolidated into the new links field).
  • We're also not including the website field in each RepositoryEntry.
  • We're planning to rename the company fields to maintainer fields instead, which is a breaking change

We should update consolidate.py appropriately.

website: allow linking to a specific extension

It would be nice to be able to link to a given extension, for the purpose of sharing that it exists.

If extensions will have their own pages ideally this would just open up the extension's page, but in the interim it would be nice if it could jump to the section of the listing page where that extension's card is, and display the card as selected / highlighted / active.

Chrome's "link to highlight" tool is an existing potential solution, but it's brittle (will fail if the extension name or docker changes), and the results aren't super visually pleasing. As an example:
https://docs.bluerobotics.com/BlueOS-Extensions-Repository/#:~:text=Cerulean%20DVL,-Maintainer

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.