On OSX to install gems you need to have admin/root access

As a result when I tried to run metanorma new test --type iso --doctype standard (in the scope of metanorma/mn-templates-iso#1 task) I got the error:

metanorma new --type iso --doctype standard iso-rice/document.adoc
You've an existing document with the /Users/camobap/Developers/Projects/riboseinc/mn-templates-iso/iso-rice/document.adoc
Still want to continue, and overwrite the existing one? (yes/no): yes
/Library/Ruby/Gems/2.3.0/gems/git-1.5.0/lib/git/lib.rb:967:in `command': git  clone '--' 'https://github.com/metanorma/mn-templates-iso' '/Library/Ruby/Gems/2.3.0/gems/metanorma-cli-1.1.7/templates/iso'  2>&1:fatal: could not create work tree dir '/Library/Ruby/Gems/2.3.0/gems/metanorma-cli-1.1.7/templates/iso': Permission denied (Git::GitExecuteError)
	from /Library/Ruby/Gems/2.3.0/gems/git-1.5.0/lib/git/lib.rb:77:in `clone'
	from /Library/Ruby/Gems/2.3.0/gems/git-1.5.0/lib/git/base.rb:29:in `clone'
	from /Library/Ruby/Gems/2.3.0/gems/git-1.5.0/lib/git.rb:97:in `clone'
	from /Library/Ruby/Gems/2.3.0/gems/metanorma-cli-1.1.7/lib/metanorma/cli/git_template.rb:78:in `clone_git_template'
	from /Library/Ruby/Gems/2.3.0/gems/metanorma-cli-1.1.7/lib/metanorma/cli/git_template.rb:67:in `download_template'
	from /Library/Ruby/Gems/2.3.0/gems/metanorma-cli-1.1.7/lib/metanorma/cli/git_template.rb:26:in `find_or_download'
	from /Library/Ruby/Gems/2.3.0/gems/metanorma-cli-1.1.7/lib/metanorma/cli/git_template.rb:36:in `find_or_download_by'
	from /Library/Ruby/Gems/2.3.0/gems/metanorma-cli-1.1.7/lib/metanorma/cli/generator.rb:61:in `find_template'
	from /Library/Ruby/Gems/2.3.0/gems/metanorma-cli-1.1.7/lib/metanorma/cli/generator.rb:74:in `type_specific_template'
	from /Library/Ruby/Gems/2.3.0/gems/metanorma-cli-1.1.7/lib/metanorma/cli/generator.rb:47:in `create_metanorma_document'
	from /Library/Ruby/Gems/2.3.0/gems/metanorma-cli-1.1.7/lib/metanorma/cli/generator.rb:21:in `run'
	from /Library/Ruby/Gems/2.3.0/gems/metanorma-cli-1.1.7/lib/metanorma/cli/generator.rb:35:in `run'
	from /Library/Ruby/Gems/2.3.0/gems/metanorma-cli-1.1.7/lib/metanorma/cli/command.rb:65:in `create_new_document'
	from /Library/Ruby/Gems/2.3.0/gems/metanorma-cli-1.1.7/lib/metanorma/cli/command.rb:16:in `new'
	from /Library/Ruby/Gems/2.3.0/gems/thor-0.20.3/lib/thor/command.rb:27:in `run'
	from /Library/Ruby/Gems/2.3.0/gems/thor-0.20.3/lib/thor/invocation.rb:126:in `invoke_command'
	from /Library/Ruby/Gems/2.3.0/gems/thor-0.20.3/lib/thor.rb:387:in `dispatch'
	from /Library/Ruby/Gems/2.3.0/gems/thor-0.20.3/lib/thor/base.rb:466:in `start'
	from /Library/Ruby/Gems/2.3.0/gems/metanorma-cli-1.1.7/lib/metanorma/cli.rb:56:in `start'
	from /Library/Ruby/Gems/2.3.0/gems/metanorma-cli-1.1.7/exe/metanorma:14:in `<top (required)>'
	from /usr/local/bin/metanorma:22:in `load'
	from /usr/local/bin/metanorma:22:in `<main>'

Should we handle this somehow? I hope we should use $HOME/.metanorma or something like this to keep templates

cc @ronaldtse @opoudjis

Use this travis config:

• https://github.com/metanorma/csa-caiq/blob/master/.travis.yml

Use this Makefile:

• https://github.com/metanorma/csa-caiq/blob/master/Makefile

Use this gitignore:

• https://github.com/metanorma/csa-caiq/blob/master/.gitignore

Remove deploy.sh (the Travis script uses https://github.com/metanorma/metanorma-build-scripts/blob/master/deploy-to-gh-pages.sh)

I faced with this during https://travis-ci.com/metanorma/mn-templates-iso/jobs/199659544 https://ci.appveyor.com/project/ribose/mn-templates-iso/builds/24506334

metanorma command failed but CI pipeline is green because zero code was returned

@ronaldtse @abunashir please triage this, I can help with this if need

Currently there is functionality to extract sourcecode fragments. Extend this to images and requirements, and label it generically as extract.

The metanorma_spec.rb seems to be repeating lots of the steps, and it's quite hard to get a high-level idea by looking in it real quick, so the idea with this issue to clean up that spec file and refactor it a bit.

See https://travis-ci.org/riboseinc/metanorma-cli/jobs/475140199

Apparently Travis' RUBY HEAD image doesn't have bundler by default anymore.

Include metanorma-unece, metanorma-mpfd as official supported types (i.e. in the gemspec so they get installed).

Comment from @ronaldtse in #43

While functionality wise this PR is good to go, we still need to get AppVeyor passing because Windows is crucial here.

Can you also help add/update the README to indicate how to use the functionality? Thanks!

How to use a default template repository
How to use a custom template repository (i.e. NIST at GitLab)
How to see what templates are supported by a repository

To build a Metanorma package of artifacts.

Recently we were asked to deploy Metanorma to a web location so that people can directly upload an adoc file (or a zipped directory) to generate Metanorma output, rather than have to install Ruby and the toolchain via metanorma-macos-setup.

We wish to use AWS Lambda via Terraform to achieve this. For example, we can provide a https://app.metanorma.com endpoint that links to this Lambda Metanorma function. This Lambda function will have to run Ruby and run the Metanorma-CLI.

Since AWS Lambda does not natively support Ruby, we will need to use a precompiled Ruby that runs on Lambda. AWS's page on Ruby demonstrates how it works using Traveling Ruby.

However, Phusion's Traveling Ruby is locked to Ruby 2.2 and seems like abandonware. But Homebrew has a Portable Ruby (https://homebrew.bintray.com/bottles-portable/) that gives Ruby 2.3. I suspect it is possible to run the full Metanorma toolchain with 2.3 (with some work, of course).

The Lambda code size (zipped) limit is 50MB, but we can always use S3 to store the package if it's too large (the gems and all), and make the Lambda function pull the package on demand (each Lambda invocation gets 500MB in /tmp).

@strogonoff would you have time to take on this challenge?

@CAMOBAP795 could you please help update Makefile.win to reflect the current Makefile?

1. There is no more metanorma.env file. The output formats are read via:
   https://github.com/metanorma/csa-caiq/blob/c17534c929ff250cca6ed8dd127c60b078443a75/Makefile#L4-L5

2. The targets to decrypt deploy key are now removed (functionality moved to metanorma-build-scripts/deploy-to-gh-pages.sh)

https://github.com/metanorma/csa-caiq/blob/c17534c929ff250cca6ed8dd127c60b078443a75/Makefile#L119-L126

And please make sure it is compilable using Makefile.win.

Thanks!

I really don't think including 10 metanorma-* gems will cause loading to be slow.

There must be something we're doing wrong in that require tree or in gem enumeration?

# To list all extensions for all flavors on this computer.
metanorma list-extensions

Originally posted by @ronaldtse in #77 (comment)

We need a command to generate a new skeleton for a new Metanorma document.

This is similar to the bundle gem and rails new commands. We want to generate a new Metanorma document skeleton, perhaps we can use git to download the skeleton (or include it in the gem).

The files that are necessary are:

• Gemfile
• index.adoc
• .travis.yml
• .github-ci.yml
• metanorma.env (the contents now go inside the metanorma file)
• Makefile (contents should go into metanorma-cli gem)
• sections/{nn}-{title}.adoc
• deploy.sh (contents should go into metanorma-cli gem)

and?

Hey Guys, whenever we use any command from metanorma / relaton cli, then it always kept outputting the processor information in the terminal, is this necessary or is there any specific reason we are doing that?

Eg:

//cc @ronaldtse, @opoudjis

We want to use the same command framework (Thor) as Relaton CLI.

Think about it. Installation of metanorma is quite complex and requires development rights on the machine.

• If the user already has metanorma installed, there's no need to run compilation on the remote end.

• For example, without metanorma installed, the user can still run metanorma new -t iso -d standard my-standard, then metanorma compile my-standard. I think we need some YAML config file in the my-standard directory for this.

Intro / Problem

For v1.1.7 our metanorma help looks like this

Commands:
  metanorma compile FILENAME                                # Compile to a metanorma document
  metanorma help [COMMAND]                                  # Describe available commands or one specific command
  metanorma new NAME -d, --doctype=DOCTYPE -t, --type=TYPE  # Create new Metanorma document
  metanorma version -t, --type=TYPE                         # Version of the code

I see a list of problems here:

• No general description
• No usage pattern
• Not standardized action description (actions are compile, help, new, version)
• No list of valid/accepted values especially for --type & --doctype
• Poor error handling, just type wrong command name & it will print stack trace which doesn't clarify the problem

Proposed solution

Let's use https://ruby-doc.org/stdlib-2.4.1/libdoc/optparse/rdoc/OptionParser.html standard module which will fix most of the issues, we will be able even to implement guess like in the git command

git sats
git: 'sats' is not a git command. See 'git --help'.

The most similar command is
	stash

cc @ronaldtse @abunashir

I wonder if we should package the HTML (and other output) in a separate directory all together because it makes life easier with distribution (right now I have to make a new directory, copy the deliverables in it, and then zip it).

The YAML input in https://github.com/riboseinc/relaton-server-csd/blob/master/data/repo/index.yaml should be able to drive the manifest generator as well, but with Relaton XML links.

Specifically, the Manifest HTML (i.e. the "HTML index" at /index.html), should contain for each entry:

• a relaton XML link
• a metanorma XML output
• a metanorma HTML output
• a metanorma PDF output
• a title description and doc identifier

List out all supported templates - as mentioned in #45 (comment)

cc: @ronaldtse

metanorma --help
/Library/Ruby/Gems/2.3.0/gems/metanorma-0.3.11/lib/metanorma/compile.rb:32:in `read': No such file or directory @ rb_sysopen - --help (Errno::ENOENT)
	from /Library/Ruby/Gems/2.3.0/gems/metanorma-0.3.11/lib/metanorma/compile.rb:32:in `options_extract'
	from /Library/Ruby/Gems/2.3.0/gems/metanorma-0.3.11/lib/metanorma/compile.rb:12:in `compile'
	from /Library/Ruby/Gems/2.3.0/gems/metanorma-cli-1.1.7/lib/metanorma/cli/compiler.rb:24:in `compile_file'
	from /Library/Ruby/Gems/2.3.0/gems/metanorma-cli-1.1.7/lib/metanorma/cli/compiler.rb:12:in `compile'
	from /Library/Ruby/Gems/2.3.0/gems/metanorma-cli-1.1.7/lib/metanorma/cli/compiler.rb:16:in `compile'
	from /Library/Ruby/Gems/2.3.0/gems/metanorma-cli-1.1.7/lib/metanorma/cli/command.rb:33:in `compile'
	from /Library/Ruby/Gems/2.3.0/gems/thor-0.20.3/lib/thor/command.rb:27:in `run'
	from /Library/Ruby/Gems/2.3.0/gems/thor-0.20.3/lib/thor/invocation.rb:126:in `invoke_command'
	from /Library/Ruby/Gems/2.3.0/gems/thor-0.20.3/lib/thor.rb:387:in `dispatch'
	from /Library/Ruby/Gems/2.3.0/gems/thor-0.20.3/lib/thor/base.rb:466:in `start'
	from /Library/Ruby/Gems/2.3.0/gems/metanorma-cli-1.1.7/lib/metanorma/cli.rb:56:in `start'
	from /Library/Ruby/Gems/2.3.0/gems/metanorma-cli-1.1.7/exe/metanorma:14:in `<top (required)>'
	from /usr/local/bin/metanorma:22:in `load'
	from /usr/local/bin/metanorma:22:in `<main>'

cc @ronaldtse @opoudjis

When running the CLI, this always shows up regardless of type specified.

[metanorma] Warning: ogc is not a default standard type.
[metanorma] Info: Attempting to load `metanorma-ogc` gem for standard type `ogc`.

This is no longer accurate because all standard gems are now loaded on demand. Previously this message indicated the loading of a custom gem.

How about we just remove this line?

[metanorma] Warning: ogc is not a default standard type.

Change

[metanorma] Info: Attempting to load `metanorma-ogc` gem for standard type `ogc`.

to

[metanorma] Info: Loading `metanorma-ogc` gem for standard type `ogc`.

metanorma -t ogc -r metanorma-ogc test.adoc does not work. Please address.

I have not put the setenv on for the new interface. Since metanorma -t ogc test.adoc does work, that's not the issue.

Currently, when one wants to do make clean, one must rely on some sort of Makefile magic to guess what files are being generated by metanorma.

I see two ways going about this:

1. Make metanorma output the list of files it would generate, to STDOUT, so other utilities like make could make use of it. Some sort of dry run.
2. Make a metanorma clean sub-command / option flag that cleans for you.

Asciidoctor populates the filename of the current document to the backend as a document attribute, node.attr("docfile"). When Asciidoctor is called via Metanorma-Cli, the filename is not available to gems that expect to find it there (e.g. in naming the local Relaton cache.)

For pushing a document to a Metanorma data store, e.g. the CalConnect Standards Registry.

Add a new command line interface, that will list out the supported doctypes, details here: #56 (comment)

cc: @ronaldtse

In the mn-templates-csd template repository, the common/ directory provides a number of dot-files, including:

• .gitignore
• .gitlab-ci.yml
• .travis.yml

However, today the metanorma new command does not copy them from common.

Example:

bundle exec metanorma new my-new-document -t ogc -d standard
Creating my-new-document/Makefile
Creating my-new-document/appveyor.yml
Creating my-new-document/Gemfile
Creating my-new-document/deploy.sh
Creating my-new-document/Makefile.win
Creating my-new-document/sections/clause_7_normative_text.adoc
Creating my-new-document/sections/clause_3_references.adoc
Creating my-new-document/sections/annex-a.adoc
Creating my-new-document/sections/annex-bibliography.adoc
Creating my-new-document/sections/clause_2_conformance.adoc
Creating my-new-document/sections/clause_0_front_material.adoc
Creating my-new-document/sections/clause_8_media_types.adoc
Creating my-new-document/sections/annex-history.adoc
Creating my-new-document/sections/annex-n.adoc
Creating my-new-document/sections/clause_5_conventions.adoc
Creating my-new-document/sections/clause_6_informative_text.adoc
Creating my-new-document/sections/clause_1_scope.adoc
Creating my-new-document/sections/clause_4_terms_and_definitions.adoc
Creating my-new-document/UML/README.adoc
Creating my-new-document/images/README.adoc
Creating my-new-document/abstract_tests/README.adoc
Creating my-new-document/code/README.adoc
Creating my-new-document/requirements/requirement.adoc
Creating my-new-document/requirements/README.adoc
Creating my-new-document/requirements/requirements_class.adoc
Creating my-new-document/README.adoc
Creating my-new-document/document.adoc
Creating my-new-document/figures/README.adoc

@abunashir could you fix this ASAP? Thanks.

From #45.

"How to see what templates are supported by a repository"

e.g.

metanorma new -t ogc

Should list out what document templates are available for this flavor.

Or:

metanorma template list # => All flavors
metanorma template list -t ogc # => One flavor

Instead of this:

$ metanorma -t mpfd mpfd-bpn.adoc
[metanorma] Error: mpfd is not a supported standard type.
$ metanorma -t mpfd -r metanorma-mpfd mpfd-bpn.adoc

We could also automatically do the require on behalf of the user like this:

$ metanorma -t mpfd mpfd-bpn.adoc
[metanorma] Warning: 'mpfd' is not a supported standard type, loading 'metanorma-mpfd' automatically.
[metanorma] Error: loading 'metanorma-mpfd' failed. You may need to run 'gem install metanorma-mpfd'.
$ gem install metanorma-mpfd
$ metanorma -t mpfd mpfd-bpn.adoc

@CAMOBAP795 the latest PR was merged but master spec now failed:

  1) Metanorma::Cli::Generator.run with local template success for existing template
     Failure/Error: expect(file_exits?(document, template)).to be_truthy, lambda { template }
       Gemfile
     # ./spec/metanorma/cli/generator_spec.rb:123:in `block in expect_document_to_include_base_templates'
     # ./spec/metanorma/cli/generator_spec.rb:122:in `each'
     # ./spec/metanorma/cli/generator_spec.rb:122:in `expect_document_to_include_base_templates'
     # ./spec/metanorma/cli/generator_spec.rb:79:in `block (4 levels) in <top (required)>'
     # ./vendor/bundle/ruby/2.5.0/gems/rspec-command-1.0.3/lib/rspec_command.rb:47:in `block (2 levels) in <module:RSpecCommand>'
     # ./vendor/bundle/ruby/2.5.0/gems/rspec-command-1.0.3/lib/rspec_command.rb:45:in `block in <module:RSpecCommand>'
Finished in 19.54 seconds (files took 0.28555 seconds to load)
30 examples, 1 failure
Failed examples:
rspec ./spec/metanorma/cli/generator_spec.rb:65 # Metanorma::Cli::Generator.run with local template success for existing template

Does the guide at https://www.metanorma.com/blog/12-25-2018/metanorma-on-windows-via-chocolatey/ supersede the current installation guide for all Windows users of Metanorma CLI? I’ll update the docs & the Getting Started section if so. (Please reassign to me in that case)

As title. For @opoudjis

For @opoudjis trying to fix for:

• homebrew formula install
• child gem loading problem

Sample attached.
csd-tofix.zip

Comments from review

• #35 (comment)
• #35 (comment)
• #35 (comment)
• #35 (comment)

Is this fixed?

• #35 (comment)

Can we change these calls to FileUtils? Thanks.

system "rm -f test.xml test.html test.alt.html test.doc"
system "metanorma -t iso test.adoc"
expect(File.exist?("test.xml")).to be true

outfilename = outfilename.sub(/\.html$/, "")
system "mkdir -p #{outfilename}"
system "mv #{outfilename}.html #{outfilename}"
system "mv #{outfilename}_images #{outfilename}"

Relaton RXL files should allow generated through specifying the -x rxl option. This allows us to simplify the Makefiles by enabling an extra file extension in the document attributes.

Alternatively, we can expose the document attribute :mn-relaton-output-file: <blah> to set the option for -R.

And respect those filenames in extraction. The extraction has scope over images, sourcecode, and requirements. There need to be sensible defaults if no filename has been provided (which is currently the case: consecutive numerals). But images will always have filenames.

system mkdir ...
system cp ...
system rm ...

=>
https://ruby-doc.org/stdlib-2.4.1/libdoc/fileutils/rdoc/FileUtils.html

require 'fileutils'
FileUtils.cd(dir, options)
FileUtils.cd(dir, options) {|dir| block }
FileUtils.pwd()
FileUtils.mkdir(dir, options)
FileUtils.mkdir(list, options)
FileUtils.mkdir_p(dir, options)
FileUtils.mkdir_p(list, options)
FileUtils.rmdir(dir, options)
FileUtils.rmdir(list, options)
FileUtils.ln(target, link, options)
FileUtils.ln(targets, dir, options)
FileUtils.ln_s(target, link, options)
FileUtils.ln_s(targets, dir, options)
FileUtils.ln_sf(target, link, options)
FileUtils.cp(src, dest, options)
FileUtils.cp(list, dir, options)
FileUtils.cp_r(src, dest, options)
FileUtils.cp_r(list, dir, options)
FileUtils.mv(src, dest, options)
FileUtils.mv(list, dir, options)
FileUtils.rm(list, options)
FileUtils.rm_r(list, options)
FileUtils.rm_rf(list, options)
FileUtils.install(src, dest, options)
FileUtils.chmod(mode, list, options)
FileUtils.chmod_R(mode, list, options)
FileUtils.chown(user, group, list, options)
FileUtils.chown_R(user, group, list, options)
FileUtils.touch(list, options)

A Metanorma Registry is a directory structure defined as:

registry/
|--- documents/ (Metanorma XML documents)
        |--- {document-id}.xml
        |--- {document-id}.xml
        |--- ...
|--- bibdata/
        |--- {*}.yaml (Relaton BibYAML files, they are in the registry but only given metadata)
        |--- {*}.rxl (Relaton XML files, they are in the registry but only given metadata)
|--- metanorma-registry.yaml

The metanorma-registry.yaml file contains default options that apply across the documents, such as:

mn-document-class: csd
mn-output-extensions:
  - xml
  - html
  - pdf
mn-output-name-pattern: "DID.EXT" # "{document-id}.{extension}"

When you run metanorma registry build, it will generate:

registry_site/
|--- documents/ (Metanorma XML documents)
        |--- {document-id}/
              |--- index.html (and the HTML should link to the other files)
              |--- {document-id}.xml
              |--- {document-id}.html
              |--- {document-id}.pdf
              |--- {document-id}.doc (if applicable)
              |--- {document-id}.rxl
              | ...
        |--- ...
|--- bibdata/
        |--- {bib-id}/
              |--- index.html (and the HTML should link to the other files)
              |--- {bib-id}.html
              |--- {bib-id}.rxl
              |--- {bib-id}.yaml
              | ...
        |--- ...
|--- manifest.xml

Basically, we want to "Ruby-ise" this Makefile: https://github.com/CalConnect/standards.calconnect.org/blob/master/Makefile
(which I wrote)

As title.

Report from metanorma/homebrew-metanorma#15 :

A Homebrew default install shows a dependency error:

/Users/ts/.rubies/ruby-2.5.3/lib/ruby/site_ruby/2.5.0/rubygems/resolver.rb:231:in `search_for': Unable to resolve dependency: 'metanorma-mpfd (~> 0.1.0)' requires 'metanorma-standoc (~> 1.1.0)' (Gem::UnsatisfiableDependencyError)

@opoudjis We need to ensure all dependencies are satisfied. The enterprise gems, mpfd and ribose flavors should not be included in the default installation and therefore should be removed from metanorma-cli.gemspec.

With the recent update, we are pushing the new version of the CLI to live, but we still have an option to switch back to the old version by setting some specific configuration.

We are keeping this one to ensure backward compatibility for the moment and to make sure none of our tools breaks for these changes, so let's run it for a while and once confirmed then let's deprecate the old version and remove all of those related codes and dependencies.

We don't this right away, once we are comfortable with the current version then we can do work on this issue. cc: @ronaldtse, @opoudjis!

metanorma --type ietf -x xmlrfc cc-51006.adoc
What's this?
[metanorma] Error: xmlrfc format is not supported for this standard.

We need to return the list of possible extensions to generate. It was previously in the CLI but now taken out to skip loading of gems.

[metanorma] Error: xmlrfc format is not supported for this standard.

We need to show a list of valid extensions.

This is urgent. @abunashir can you please help? Thanks!

Enterprise users need to use private template repositories, and it is cumbersome to enter the full template source address every time they generate a new document.

Therefore it is necessary to store the repository configuration on the local computer. This takes the same approach as Git, where repository remotes are stored in .git/config.

Local CLI configuration

~/.metanorma/config.yml

e.g.

---
:template_sources:
- :name: my-repo-name
  :address: [email protected]/myaccount/mn-templates

Template repo add

Add template-repo source to local metanorma configuration.

metanorma template source add <repo_name> <git_repo_address>

e.g.

metanorma template source add my-repo-name https://github.com/blah/blah-metanorma-templates” 

Template repo list

List template-sources.

metanorma template source list

Remove template source

metanorma template source remove <repo_name>