Cross-platform build scripts about robot HOT 17 OPEN

As someone who has reinvented make several times, the idea thrills me...

But how many Windows users need to make releases? Wouldn't it just be simpler to set up a bamboo/jenkins job for releasing the ontology?

OTOH I think there is a need for a wide spectrum of users to be able to run certain sequences of commands to get their local setup correct or synchronized. At the moment, we typically check in import modules plus a catalog, but this won't scale in the long run. It would be useful to run a robot command to get all this taken care of. Also refreshing the import module. But this perhaps points to a slightly different set of requirements, perhaps tending towards the onto-maven/ivy/npm idea.

from robot.

I agree that it makes more sense to build official releases on a build/integration server.

The use cases I have in mind are local development and testing. If an ontology project uses ROBOT to do imports and convert from tables, then developers will want to be able to update the imports and tables then make sure that their changes work before committing. I also want ROBOT to cover local application ontology development use cases, like the ones in the https://github.com/jamesaoverton/obo-tutorial. Developing local application ontologies and contributing to community reference ontologies should, ideally, use the same tools and include as wide a group of potential contributors as possible.

Many (most?) of the core OBI developers use Windows, for example. So if I want to use ROBOT for OBI, then I need a cross-platform solution. I'm just not sure that this proposal is the right solution.

from robot.

Common Workflow Language might cover 80% of what we need.

Rabix Bunny is an open source (Apache License 2.0) Java implementation that optionally uses Docker.

@rctauber and I will look into this a bit more.

from robot.

We're looking into CWL for other projects. cc @kltm @realmarcin @simonjupp

from robot.

Rabix bunny looks interesting, thanks for posting. Though in my limited knowledge CWL already supports executing tasks in docker container, so I'm assuming Rabix docker integration is different/more.

And what do you think is the 20% not covered by CWL?

from robot.

I mentioned Rabix Bunny because it's a Java implementation of CWL, and ROBOT is written in Java. I meant to emphasize that Docker is optional. I was thinking (but I didn't say) that ROBOT could include Bunny in the JAR we distribute, requiring only Java and not Docker, and keeping our deployment story the same.

Of course, ROBOT can already be used as part of a CWL workflow with any given CWL implementation, with or without Docker, and ROBOT could be distributed in a Docker image. There's lots of options and I'm still digesting these ideas.

Make is built around string substitution and manipulation. I'm just learning about CWL, and I thought it deliberately avoided that can of worms. But now I see that workflow definitions can include JavaScript, although they warn you not to: http://www.commonwl.org/user_guide/13-expressions/. So 20% is too high an estimate.

I haven't use CWL yet. I'd like to hear about real-world experience with it.

from robot.

from robot.

Revisiting this

Now I know more about CWL I think it's not such a good fit. At least I don't think the workflow should be directly specified in CWL YAML (though there may be a use case in compiling down to it from another high level representation).

I am coming round to the custom robot/ODK yaml idea. One nice thing here is that it doesn't immediately commit us to writing a workflow engine. We could experiment and do compilation to Makefile in the interim.

I have in mind something slightly more abstracted than James original proposal here. For example, for many repos I would like to specify something like

main:
  source: foo-edit.owl
  reasoner:
    type: elk
    materialize: part_of
  filters: ...
  checks:
  formats:
   - obo
   - owl
   - json
subsets:
  products:
   - goslim_foo
   - goslim_bar
   - ...
  formats:
   - obo
   - owl
   - json
   - tsv
imports:
  sources:
     - uberon:
          versionIRI: ">=2018-10-01"
          product: uberon-base
          filters: <robot filter here>
    - go:
         versionIRI: "*"
         ...
extra:
  my_custom_report:
    desc: some report that's useful
    sparql: <relpath>
  ...

from robot.

I like this more abstract approach. A few thoughts:

1. If this is (also) project configuration, then it would be good to coordinate with Protege for a subset of properties that we could share, such as ID patterns, ID ranges, etc. That stuff could be in a separate file, but then we have to draw a distinction somewhere.
2. I strongly prefer convention over configuration, and we're trying to push OBO projects to converge on a strong set of conventions. We should have really good defaults, so that most of these configuration files are tiny.
3. I expect this config format will evolve. I think we should build in a versioning system from the start, and be able to support legacy formats.
4. It would be nice to be able to convert this config data to JSON-LD (RDF).

from robot.

from robot.

If this is (also) project configuration, then it would be good to coordinate with Protege

Would love to coordinate on all of this

from robot.

Great! You can see the branch we're working on in the ODK (formerly starter kit): INCATools/ontology-development-kit#138

You can get a sense of the project schema:

https://github.com/INCATools/ontology-development-kit/blob/yaml-driven-config2/odk/odk.py#L44-L325

Need to add more examples here:
https://github.com/INCATools/ontology-development-kit/tree/yaml-driven-config2/examples

from robot.

Thanks for the pointers @cmungall. I'm with @jamesaoverton about convention over config too. I'd like Protege to support a standard project directory format.

from robot.

... and happy for the format to be YAML. I could imagine a project settings editor in Protege.

from robot.

I've had a quick look at the schema... actually the python code was easier to read :) Definitely want to support this in Protege in some way e.g. editor for project settings, using the imports mechanism instead of the xmlcatalog etc.

I've just got some observations/suggestions (some are probably banal) and questions. These come from me as someone who isn't (yet) familiar with the specifics of ODK. I realise that it might be too late to change some of this stuff, be here we go...

• Could the patterns directory be within the source directory? Aren't patterns source code for the ontology or does "source" have a more specific meaning? It seems strange to go a level up. There's probably a good reason for the way it is, but I don't know what that is.
• There's a github_org property at the moment. This can be a user id or an organisation id. Consider changing this property name to github_id
• There's a repo property. As far as I can tell this is expected to be a GitHub repo name. I thought it was any URL at first. Would you consider renaming it to something like github_repo_name (or repo_name).
• Values for the edit_format property are in lower case and these must be either OWL or OBO. Consider using upper case strings.
• travis_emails seems overly specific. Consider continuous_integration_emails
• The naming for the uribase property is inconsistent with other property names. Consider uri_base instead, or iri_base.
• Have you considered having a separate target directory so that builds products are in one easily delectable place? I guess I'm used to Maven, but I'd prefer this.
• Similar to the point above, have you considered a generated_sources directory (for intermediate (and mirrored) imports etc.? This would indicate that these things should not be hand edited. Could still be committed to a repo.

from robot.

Is this more in the realm of ODK or do we still want to consider something like this for ROBOT?

from robot.

Not sure, but I'd like to leave this ROBOT issue open for now.

from robot.