Git Product home page Git Product logo

extractly's Introduction

Extractly

CI Coverage Status Hex.pm Hex.pm Hex.pm

Extractly

Provide easy access to information inside the templates rendered by mix xtra

Extractly.do_not_edit_warning/1

Emits a comment including a message not to edit the created file, as it will be recreated from this template.

It is a convenience to include this into your templates as follows

      <%= xtra.do_not_edit_warning %>

or I18n'ed

      <%= xtra.do_not_edit_warning, lang: :fr %>

If you are not generating html or markdown the comment can be parametrized

      <%= xtra.do_not_edit_warning, comment_start: "-- ", comment_end: "" %>

If you want to include the name of the source template use template: template option, so a call may be as complex as:

      <%= xtra.do_not_edit_warning, comment_start: "-- ", comment_end: "", template: template, lang: :it %>

Extractly.functiondoc/2

Returns docstring of a function Ex:

      iex(1)> {:ok, lines} = Extractly.functiondoc("Extractly.moduledoc/2") |> hd()
      ...(1)> lines |> String.split("\n") |> Enum.take(3)
      ["  Returns docstring of a module", "", "  E.g. verbatim"]

We can also pass a list of functions to get their docs concatenated

      iex(2)> [{:ok, moduledoc}, {:error, message}] = Extractly.functiondoc(["Extractly.moduledoc/2", "Extactly.functiondoc/2"])
      ...(2)> moduledoc |> String.split("\n") |> Enum.take(4)
      [ "  Returns docstring of a module",
        "  E.g. verbatim",
        "",
        "      Extractly.moduledoc(\"Extractly\")"]
      ...(2)> message
      "Function doc for function Extactly.functiondoc/2 not found"

If all the functions are in the same module the following form can be used

      iex(3)> [{:ok, out}, _] = Extractly.functiondoc(["moduledoc/2", "functiondoc/2"], module: "Extractly")
      ...(3)> String.split(out, "\n") |> hd()
      "  Returns docstring of a module"

However it is convenient to add a markdown headline before each functiondoc, especially in these cases, it can be done by indicating the headline: level option

      iex(4)> [{:ok, moduledoc}, {:ok, functiondoc}] = Extractly.functiondoc(["moduledoc/2", "functiondoc/2"], module: "Extractly", headline: 2)
      ...(4)> moduledoc |> String.split("\n") |> Enum.take(3)
      [ "## Extractly.moduledoc/2",
        "",
        "  Returns docstring of a module"]
      ...(4)> functiondoc |> String.split("\n") |> Enum.take(3)
      [ "## Extractly.functiondoc/2",
        "",
        "  Returns docstring of a function"]

Often times we are interested by all public functiondocs...

      iex(5)> [{:ok, out}|_] = Extractly.functiondoc(:all, module: "Extractly", headline: 2)
      ...(5)> String.split(out, "\n") |> Enum.take(3)
      [ "## Extractly.do_not_edit_warning/1",
        "",
        "  Emits a comment including a message not to edit the created file, as it will be recreated from this template."]

We can specify a language to wrap indented code blocks into ```elixir\n...\n```

Here is an example

      iex(6)> [ok: doc] = Extractly.functiondoc("Extractly.functiondoc/2", wrap_code_blocks: "elixir")
      ...(6)> doc |> String.split("\n") |> Enum.take(10)
      [ "  Returns docstring of a function",
        "  Ex:",
        "",
        "```elixir",
        "      iex(1)> {:ok, lines} = Extractly.functiondoc(\"Extractly.moduledoc/2\") |> hd()",
        "      ...(1)> lines |> String.split(\"\\n\") |> Enum.take(3)",
        "      [\"  Returns docstring of a module\", \"\", \"  E.g. verbatim\"]",
        "```",
        "",
        "  We can also pass a list of functions to get their docs concatenated"]

Extractly.macrodoc/2

Returns docstring of a macro

Extractly.moduledoc/2

Returns docstring of a module

E.g. verbatim

      iex(7)> {:ok, doc} = Extractly.moduledoc("Extractly")
      ...(7)> doc
      "  Provide easy access to information inside the templates rendered by `mix xtra`\n"

We can use the same options as with functiondoc

      iex(8)> {:ok, doc} = Extractly.moduledoc("Extractly", headline: 2)
      ...(8)> doc |> String.split("\n") |> Enum.take(3)
      [
        "## Extractly", "", "  Provide easy access to information inside the templates rendered by `mix xtra`"
      ]

If we also want to use functiondoc :all, module: "Extractly" after the call of moduledoc we can include :all in the call of moduledoc, which will include function and macro docstrings as well

      iex(9)> [{:ok, moduledoc} | _] =
      ...(9)>   moduledoc("Extractly", headline: 3, include: :all)
      ...(9)> moduledoc
      "### Extractly\n\n  Provide easy access to information inside the templates rendered by `mix xtra`\n"
      iex(10)> [_, {:ok, first_functiondoc} | _] =
      ...(10)>   moduledoc("Extractly", headline: 3, include: :all)
      ...(10)> first_functiondoc |> String.split("\n") |> Enum.take(5)
      [
        "### Extractly.do_not_edit_warning/1",
        "",
        "  Emits a comment including a message not to edit the created file, as it will be recreated from this template.",
        "",
        "  It is a convenience to include this into your templates as follows"
      ]

Extractly.task/2

Returns the output of a mix task Ex:

    iex(14)> Extractly.task("cmd", ~W[echo 42])
    "42\n"
    iex(15)> try do
    ...(15)>   Extractly.task("xxx")
    ...(15)> rescue
    ...(15)>   e in RuntimeError -> e.message |> String.split("\n") |> hd()
    ...(15)> end
    "The following output was produced wih error code 1"

Extractly.toc/2

Extract Table Of Contents from a markdown document

The files used for the following doctest can be found here

    iex(11)> lines = [
    ...(11)>         "## Usage",
    ...(11)>         "### API",
    ...(11)>         "#### EarmarkParser.as_ast/2",
    ...(11)>         "### Support",
    ...(11)> ]
    ...(11)> toc(lines, gh_links: true)
    {:ok, [
      "- [Usage](#usage)",
      "  - [API](#api)",
      "    - [EarmarkParser.as_ast/2](#earmarkparseras_ast2)",
      "  - [Support](#support)",
    ]}
But if you do not want links
    iex(12)> lines = [
    ...(12)>         "## Usage",
    ...(12)>         "### API",
    ...(12)>         "#### EarmarkParser.as_ast/2",
    ...(12)>         "### Support",
    ...(12)> ]
    ...(12)> toc(lines)
    {:ok, [
      "- Usage",
      "  - API",
      "    - EarmarkParser.as_ast/2",
      "  - Support",
    ]}

In case of bad options an error tuple is returned (no utf8 encoded input should ever result in an error_tuple

    iex(13)> lines = [] # options are checked even if input is empty
    ...(13)> toc(lines, no_such_option: "x")
    {:error, "Unsupported option no_such_option"}

A more detailed description can be found in Extractly.Toc's docstrings

Extractly.version/0

A convenience method to access this libraries version

Extractly.Toc

Extract Table Of Contents from a list of lines representing a Markdown document

Extractly.Toc.render/2

Depending on the options the Table Of Contents extracted from the lines can be rendered in different formats, the default being Markdown

Markdown

    iex(1)> render(["# Hello", "## World"])
    ["- Hello",  "  - World"]

Numbered lists can be created too

    iex(2)> render(["# Hello", "## World"], type: :ol)
    ["1. Hello",  "   1. World"]

Oftentimes the level of headlines is adapted for output, e.g. ### for the top and ##### for the second level.

render accounts for that

    iex(3)> render(["### Alpha", "ignored", "##### Alpha.1", "", "### Beta"])
    ["- Alpha", "  - Alpha.1", "- Beta"]
Remove Gaps

Sometimes there will be gaps in the levels of headlines and these holes might not reflect semantic but rather stylistic concerns, if this is the case the option remove_gaps can be set to true

    iex(4)> render(["# First", "### Third (but will go to second level)", "## Second"], remove_gaps: true)
    ["- First", "  - Third (but will go to second level)", "  - Second"]
Github README Links

This is all nice, however a TOC is most useful if links are provided.

render can render Github like links to within the page, here is a real world example from a Github README.md file

    iex(5)> lines = [
    ...(5)>         "## Usage",
    ...(5)>         "### API",
    ...(5)>         "#### EarmarkParser.as_ast/2",
    ...(5)>         "### Support",
    ...(5)> ]
    ...(5)> render(lines, gh_links: true)
    [
      "- [Usage](#usage)",
      "  - [API](#api)",
      "    - [EarmarkParser.as_ast/2](#earmarkparseras_ast2)",
      "  - [Support](#support)",
    ]

HTML

Sometimes it might be appropriate to generate HTML directly

    iex(6)> render(["## One", "### Two"], format: :html)
    ["<ul>", "<li>One<ul>", "<li>Two</li>", "</ul></li>", "</ul>"]
Exlcuding levels and changing list styles

Let us examine these two options with HTML output, they work too for Markdown of course, but are meaningless with the more raw output formats

So we do not want to include levels greater than, say 3, and we also want to ignore top level headlines, probably because only one top level part has sublevels

    iex(7)> document = [
    ...(7)> "# Ignore",
    ...(7)> "# Too, but not what's below",
    ...(7)> "## Synopsis",
    ...(7)> "## Description",
    ...(7)> "### API",
    ...(7)> "#### too detailed",
    ...(7)> "### Tips & Tricks",
    ...(7)> "# Ignored again"
    ...(7)> ]
    ...(7)> render(document, format: :html, min_level: 2, max_level: 3, start: 5, type: :ol)
    [
      ~S{<ol start="5">},
      ~S{<li>Synopsis</li>},
      ~S{<li>Description<ol>},
      ~S{<li>API</li>},
      ~S{<li>Tips &amp; Tricks</li>},
      ~S{</ol></li>},
      ~S{</ol>},
    ]

PushList

Either a linear PushList

    iex(8)> render(["# I", "## I.1", "## I.2", "### I.2.(i)", "# II", "### II.1.(ii)"], format: :push_list)
    ["I", :open, "I.1", "I.2", :open, "I.2.(i)", :close, :close, "II", :open, :open, "II.1.(ii)", :close, :close]

AST tree

    iex(9)> render(["# I", "## I.1", "## I.2", "### I.2.(i)", "# II", "### II.1.(ii)"], format: :ast)
    ["I", ["I.1", "I.2", ["I.2.(i)"]], "II", [["II.1.(ii)"]]]

Unsupported Formats

    iex(9)> render(["# Does not really matter"], format: :unknown)
    {:error, "Unsupported format: unknown in render"}

Mix.Tasks.Xtra

Mix task to Transform EEx templates in the context of the Extractly module.

This tool serves two purposes.

  1. A simple CLI to basicly EEx.eval_file/2

  2. Access to the Extractly module (available as binding xtra too)

  3. Access to the name of the rendered template with the template binding

The Extractly module gives easy access to Elixir metainformation of the application using the extractly package, notably, module and function documentation.

This is BTW the raison d'être of this package, simple creation of a README.md file with very simple access to the projects hex documentation.

Thusly hexdoc and Github will always be synchronized.

To see that in action just look at the README.md.eex file of this package and compare with what you are reading here.

Example Template:

  Some text
  <%= xtra.functiondoc("M.shiny_function/2") %>
  <%= xtra.moduledoc("String") %>

  <%= xtra.moduledoc("MyModule", include: :all) %>
  <%= xtra.toc "SomeFile.md" %>

  More text

A special case is the occurrence of <%= xtra.toc :self, ... %> which just inserts a placeholder which than is replaced by the TOC of the generated output in a second pass

Mix.Tasks.Xtra.Help

Usage:

mix xtra [options]... [template]

Options:

--help | -h     Prints short help information to stdout and exits.
--quiet | -q    No output to stdout or stderr
--version | -v  Prints the current version to stdout and exits.
--verbose | -V  Prints additional output to stderr

--output filename
          The name of the file the rendered template is written to, defaults to the templates'
          name, without the suffix `.eex`

Argument:

template, filename of the `EEx` template to use, defaults to `"README.md.eex"`

Documentation can be generated with ExDoc and published on HexDocs. Once published, the docs can be found at https://hexdocs.pm/extractly.

Author

Copyright © 20[18-21] Robert Dober, [email protected],

LICENSE

Same as Elixir, which is Apache License v2.0. Please refer to LICENSE for details.

extractly's People

Contributors

robertdober avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar

Forkers

crankwheel

extractly's Issues

Add a --quiet|-q param to the mix task

Right now succesful completion of mix xtra is silent, which is not very reassuring.

Add some helpful output which can be silenced with the new -q|--quiet param

Extract a Table of Contents from a Markdown document

Synopsis

As Extractly is used by Earmark and potentially by EarmarkParser parsing the markdown is not an option (and probably total overkill anyway).

Therefore scanning for lines matching ~r{\A \s{0,3} (#{1,7}) \s+ (.*)}x shall deliver the necessary information.

Sketch

With that in mind such a list shall be obtained by the following pseudo code

     lines
     |> Enum.map(&Regex.run(~r{\A \s{0,3} (#{1,7}) \s+ (.*)}x, &1)
     |> Enum.filter(& &1)
     |> Enum.map(fn [_, header, title] -> {String.length(header), title} end) 

yielding, e.g. ...

   [{1, "Main"}, {2, "Sub 1"}, {2, "Sub 2"}, {3, "SubSub"}, {1, "Epilogue"}]

which than, depending on some options shall yield markdown, e.g.

1. Main
   1. Sub 1
   1. Sub 2
      1. SubSub
1. Epilogue

Options

  • numbered: 1 value for starting position, nil for unnumbered lists
  • levels: 7 maximum level of headers taken into account
  • output: :markdown alternatives like :ast, :html or :json might be a nice enhancement

Limitations

Setext headers are not taken into account, as I do not have a use case for now

add output of mix tasks to input

Given a mix task bellum_gallicum that produces the following output

     Ominia Gallia divisa est in partes tres

and a template some_file.md.eex with

   <%= xtra.taks("bellum_gallicum") %>

and running the mix task mix xtra some_file.md

then the file some_file.md shall contain

     Ominia Gallia divisa est in partes tres

Leave a comment in the output for where the source comes from

When asking contributers to regenerate the README.md correctly it is ciumbersome to follow the inclusion path to find the sources

Add an option to include a comment to document the source of the generated text (defaulting to true hence the minor version upgrade`)

Add an option to wrap indented code parts into language specific code blocks

What

Now and without the option the markdown from docstrings is copied verbosly e.g.

              iex(42)> some_doctest
             42

but when issuing e.g. `<%= xtra.moduledoc "MyModule", code_block_language: "elixir" the docstrings will be processed to yiedl

             ```elixir
              iex(42)> some_doctest
              42
              ```

Rationale:

This will generate the same syntax highlighted output in README.md on Github that the docstrings yield on hexdoc.pm

gh_links: true bad name conversion

The headline Structure Conserving Transformers is translated into structureconservingtransformers but needs to be translated into #structure-conserving-transformers

Include all docstrings of a module

This usecase comes from the following pattern

<%= xtra.moduledoc "MyModule", ... %>
<%= xtra.functiondoc :all, module: "MyModule", ... %>

it would be nice to condense this repeating pattern

e.g.

<%= xtra.moduledoc "MyModule", include: ["functiondocs", "macrodocs"], ... %> 

Add directives into source files

A typical use case would be to add the following to a docstring

<!-- %extractly%ignore_line%
  Some markdown useful inside the extracted markdown but not in hexdocs where there is more context
 %extractly%ignore_line% -->

However sometimes it might also be a good idea not to extract all information from a docstring and therefore three more directives shall be implemented

<!-- %extractly%stop_processing% this directive ends all extraction -->
<!-- %extractly%suspend_processing% name says it all -->
<!-- %extractly%resume_processing% idem -->

Add messaging to stderr

Depending on two options

  • No option meaning default behavior: Log errors to sdterr
  • verbose option set: Log all functioncalls to stderr
  • quiet option set: Log nothing

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.