vinci1it2000 / schedula Goto Github PK

View Code? Open in Web Editor NEW

48.0 48.0 15.0 37.32 MB

A smart function scheduler for dynamic flow-based programming.

Home Page: http://schedula.readthedocs.io/

License: European Union Public License 1.1

Python 59.79% HTML 3.75% Smarty 0.07% Shell 0.56% CSS 1.21% JavaScript 34.62%

schedula's People

Contributors

Stargazers

Watchers

Forkers

gfon ankostis brad-eki bollwyvl stefanocorsi ashenafimenza andreslaverdemarin naushad-rahman eklump mfkiwl hyperc-ai scottgifford karang27 jeffamaxey rmrf2020

schedula's Issues

Site fails with sphinx 1.6.2

In Build #5601384, co2mpas's site has failed because sphinx is upgraded to 1.6.2 - when pinned to 1.5.5 it worked.

Traceback (most recent call last):
  File "/home/docs/checkouts/readthedocs.org/user_builds/co2mpas-ta/envs/latest/lib/python3.5/site-packages/sphinx/cmdline.py", line 305, in main
    opts.warningiserror, opts.tags, opts.verbosity, opts.jobs)
  File "/home/docs/checkouts/readthedocs.org/user_builds/co2mpas-ta/envs/latest/lib/python3.5/site-packages/sphinx/application.py", line 196, in __init__
    self.setup_extension(extension)
  File "/home/docs/checkouts/readthedocs.org/user_builds/co2mpas-ta/envs/latest/lib/python3.5/site-packages/sphinx/application.py", line 456, in setup_extension
    self.registry.load_extension(self, extname)
  File "/home/docs/checkouts/readthedocs.org/user_builds/co2mpas-ta/envs/latest/lib/python3.5/site-packages/sphinx/registry.py", line 207, in load_extension
    metadata = mod.setup(app)
  File "/home/docs/checkouts/readthedocs.org/user_builds/co2mpas-ta/envs/latest/lib/python3.5/site-packages/schedula/ext/autosummary.py", line 196, in setup
    listeners = app._listeners[event]
AttributeError: 'Sphinx' object has no attribute '_listeners'

Exception occurred:
  File "/home/docs/checkouts/readthedocs.org/user_builds/co2mpas-ta/envs/latest/lib/python3.5/site-packages/schedula/ext/autosummary.py", line 196, in setup
    listeners = app._listeners[event]
AttributeError: 'Sphinx' object has no attribute '_listeners'

No wheel released in PyPi repo

Currently no wheel has been published in PyPi repo - only sources.

The way to reolve this is to use these 2 commands on the next release:

python setup.py sdist bdist_wheel ## Build sources and whl into ./dist/ dir
twine upload dist/*                ## Release built wheel

Repeatitive use of identical try...except blocks in Sol.py

A minor comment on this lib's central piece of functionality, where user functiins are executed:

schedula/schedula/utils/sol.py

Lines 570 to 630 in 86ebce3

 value = self._evaluate_function( 

 (kwargs,), node_id, node_attr, attr 

 ) 

 except KeyboardInterrupt as ex: 

 raise ex 

 except Exception as ex: 

 if 'started' in attr: 

 dt = datetime.today() - attr['started'] 

 attr['duration'] = dt 

 # Some error occurs. 

 msg = "Failed DISPATCHING '%s' due to:\n %r" 

 self._warning(msg, node_id, ex) 

 return False 

 else: 

 # Data node that has just one estimation value. 

 value = list(est.values())[0]['value'] 

 else: # Use the estimation function of node. 

 try: 

 # Dict of all data node estimations. 

 kwargs = {k: v['value'] for k, v in est.items()} 

 # Evaluate output. 

 value = self._evaluate_function( 

 (kwargs,), node_id, node_attr, attr 

 ) 

 except KeyboardInterrupt as ex: 

 raise ex 

 except Exception as ex: 

 if 'started' in attr: 

 attr['duration'] = datetime.today() - attr['started'] 

 # Is missing estimation function of data node or some error. 

 msg = "Failed DISPATCHING '%s' due to:\n %r" 

 self._warning(msg, node_id, ex) 

 return False 

 try: 

 # Apply filters to output. 

 value = self._apply_filters(value, node_id, node_attr, attr) 

 except KeyboardInterrupt as ex: 

 raise ex 

 except Exception as ex: 

 if 'started' in attr: 

 attr['duration'] = datetime.today() - attr['started'] 

 # Some error occurs. 

 msg = "Failed DISPATCHING '%s' due to:\n %r" 

 self._warning(msg, node_id, ex) 

 return False 

 if value is not NONE: # Set data output. 

 self[node_id] = value 

 if 'started' in attr: 

 attr['duration'] = datetime.today() - attr['started'] 

 if 'callback' in node_attr: # Invoke callback func of data node. 

 try: 

 # noinspection PyCallingNonCallable 

 node_attr['callback'](value) 

 except KeyboardInterrupt as ex: 

 raise ex 

 except Exception as ex:

There are 3 try..except blocks in function _set_data_node_output() that do the same thing. They can be substituted with a single one with nested a little "outer" - is there any rewson not to do that?

Pre releases do not work with readthedocs

Still co2mpas builds fail on readthedocs because it cannot find --pre released of schedula, and picks the old one.

In general when the project is a library, it makes much sense to release more often full release.
And more so when it is at the early stages of development (as it is now).

Remove nose from `setup_requires`

The setup-requieres keyword ni setup.py is rarely needed, if ever now that project.toml PEP518 is out.
Additionally, any package added there can have consequences for downstream projects & tools (e.g. it breaks conda packaging with recent conda-4.10).
Besides, nose is a testing library and should definetely not be there.

Jinja templates should use autoescape=True to reduce risk of template injection

Is your feature request related to a problem? Please describe.
The Jinja templates used for autosummary and drw currently leave the autoescape setting at its default value, False. As a result, template variables can inject HTML into generated pages, creating a security risk if those variables are under the control of an attacker.

Describe the solution you'd like
Set autoescape=True when using Jinja

Describe alternatives you've considered
Update the documentation to make it clear that template values must not be under the control of an attacker

Additional context
This issue was found by an automated security scanner

Advice on reducing memory usage of schedula for large Excel sheet from formulas

Hello,

After some early successes using formulas and schedula to calculate Excel LCAs, we have run into memory usage troubles when processing larger sheets. Some analysis shows that the memory is mostly in the formulas dsp property, which is a shcedula object, and I was hoping for some advice on what I could do to reduce memory usage there to support processing larger spreadsheets.

In a successful run on a host with 512GB of RAM, from a 5M-cell Excel file, I am able to load the 2.2M cells needed for my LCA calculation, with 165GB of total memory used. With logging I added, I can see that about 2GB is required to load the sheet into OpenPyXL, an additional 125GB to create the cells in formulas, an additional 10GB to .finish() the formulas object, and an additional 24GB to .compile() this into a function. For more details see excelsior-successful-run.txt.

In an unsuccessful run on a host with 512GB of RAM, from a 9.4M-cell Excel file, my program is killed by the Linux OOM killer after creating 8.8M cells and consuming 498GB of RAM. Logging indicates 5GB is used to load the Excel with OpenPyXL, then memory usage increases as formulas adds cells, until it is exhausted. For more details see excelsior-failed-run.txt.

My code to load the data and compile the function looks like this:

# out_ranges = ["'[/path/to/big-spreadsheet.xlsx]LOOKUPS'!E2"]
xl = formulas.ExcelModel()
xl.from_ranges(
    *out_ranges
)
xl.finish()

xl_func = xl.compile(
    inputs=formulas_input_mappings,
    outputs=formulas_output_mappings,
)

Any advice is appreciated!

Schedula broke with latest networx 2.4 due to dropped graph.node attribute

Recently released networkx-2.4 (16 Oct 2019) broke schedula due to removal of deprecated methods: https://networkx.github.io/documentation/stable/release/release_2.4.html#deprecations:

  File "/home/user/Work/co2mpas.git/co2mpas/cli/__init__.py", line 133, in plot
    ['plot', 'done']
  File "/usr/lib/python3.7/site-packages/schedula/utils/blue.py", line 127, in __call__
    return self.register(memo={})(*args, **kwargs)
  File "/usr/lib/python3.7/site-packages/schedula/dispatcher.py", line 1682, in __call__
    return self.dispatch(*args, **kwargs)
  File "/usr/lib/python3.7/site-packages/schedula/dispatcher.py", line 1673, in dispatch
    sol._run(stopper=stopper, executor=executor)
  File "/usr/lib/python3.7/site-packages/schedula/utils/sol.py", line 291, in _run
    if not sol._visit_nodes(v, d, fringe, check_cutoff, **ctx):
  File "/usr/lib/python3.7/site-packages/schedula/utils/sol.py", line 950, in _visit_nodes
    if not self._set_node_output(node_id, no_call, **kw):  # Set output.
  File "/usr/lib/python3.7/site-packages/schedula/utils/sol.py", line 577, in _set_node_output
    next_nds, **kw)
  File "/usr/lib/python3.7/site-packages/schedula/utils/sol.py", line 678, in _set_data_node_output
    value = async_thread(self, args, node_attr, node_id, sf, **kw)
  File "/usr/lib/python3.7/site-packages/schedula/utils/asy.py", line 240, in async_thread
    return sol._evaluate_node(args, node_attr, node_id, *a, **kw)
  File "/usr/lib/python3.7/site-packages/schedula/utils/sol.py", line 613, in _evaluate_node
    attr = self.workflow.node[node_id]
AttributeError: 'DiGraph' object has no attribute 'node'

Get one liner formula from graph node

It would be useful to get a one liner string representing a DAG from the dispatcher.

so basically having one call like "dsp.get_one_liner(origin = 'A2')" which basically returns all subnodes of the graph starting from A2. Example:

A2: "=SUM(A1,A3)"
A3: "=SUM(1,3)"
A1: "=A4"
A4: "=SUM(1,1,1,1,1)"

dsp.get_one_liner(origin = 'A2') --> returns: "=SUM(SUM(1,1,1,1,1),SUM(1,3)"
dsp.get_one_liner(origin = 'A1') --> returns: "=SUM(1,1,1,1,1)"

and so on...

doc: dispatch directive fails in ReadTheDocws

Schedula's directive for workflow diagrams breaks site generation, ass seen in this co2mpas build, and produces a WARNING as shown at the bottom.

reading sources... [ 98%] changes
reading sources... [ 98%] co2mpas_RelNotes
reading sources... [ 99%] developers
reading sources... [ 99%] explanation
failed at node 11294[1]
dot: maze.c:315: chkSgraph: Assertion `np->cells[1]' failed.
/usr/lib/python3/dist-packages/matplotlib/font_manager.py:273: UserWarning: Matplotlib is building the font cache using fc-list. This may take a moment.
  warnings.warn('Matplotlib is building the font cache using fc-list. This may take a moment.')
/usr/lib/python3/dist-packages/matplotlib/font_manager.py:273: UserWarning: Matplotlib is building the font cache using fc-list. This may take a moment.
  warnings.warn('Matplotlib is building the font cache using fc-list. This may take a moment.')

Traceback (most recent call last):
  File "/home/docs/checkouts/readthedocs.org/user_builds/co2mpas-ta/envs/latest/lib/python3.5/site-packages/sphinx/cmdline.py", line 244, in main
    app.build(opts.force_all, filenames)
  File "/home/docs/checkouts/readthedocs.org/user_builds/co2mpas-ta/envs/latest/lib/python3.5/site-packages/sphinx/application.py", line 297, in build
    self.builder.build_update()
  File "/home/docs/checkouts/readthedocs.org/user_builds/co2mpas-ta/envs/latest/lib/python3.5/site-packages/sphinx/builders/__init__.py", line 251, in build_update
    'out of date' % len(to_build))
  File "/home/docs/checkouts/readthedocs.org/user_builds/co2mpas-ta/envs/latest/lib/python3.5/site-packages/sphinx/builders/__init__.py", line 265, in build
    self.doctreedir, self.app))
  File "/home/docs/checkouts/readthedocs.org/user_builds/co2mpas-ta/envs/latest/lib/python3.5/site-packages/sphinx/environment.py", line 569, in update
    self._read_serial(docnames, app)
  File "/home/docs/checkouts/readthedocs.org/user_builds/co2mpas-ta/envs/latest/lib/python3.5/site-packages/sphinx/environment.py", line 589, in _read_serial
    self.read_doc(docname, app)
  File "/home/docs/checkouts/readthedocs.org/user_builds/co2mpas-ta/envs/latest/lib/python3.5/site-packages/sphinx/environment.py", line 742, in read_doc
    pub.publish()
  File "/home/docs/checkouts/readthedocs.org/user_builds/co2mpas-ta/envs/latest/lib/python3.5/site-packages/docutils/core.py", line 217, in publish
    self.settings)
  File "/home/docs/checkouts/readthedocs.org/user_builds/co2mpas-ta/envs/latest/lib/python3.5/site-packages/sphinx/io.py", line 49, in read
    self.parse()
  File "/home/docs/checkouts/readthedocs.org/user_builds/co2mpas-ta/envs/latest/lib/python3.5/site-packages/docutils/readers/__init__.py", line 78, in parse
    self.parser.parse(self.input, document)
  File "/home/docs/checkouts/readthedocs.org/user_builds/co2mpas-ta/envs/latest/lib/python3.5/site-packages/docutils/parsers/rst/__init__.py", line 185, in parse
    self.statemachine.run(inputlines, document, inliner=self.inliner)
  File "/home/docs/checkouts/readthedocs.org/user_builds/co2mpas-ta/envs/latest/lib/python3.5/site-packages/docutils/parsers/rst/states.py", line 170, in run
    input_source=document['source'])
  File "/home/docs/checkouts/readthedocs.org/user_builds/co2mpas-ta/envs/latest/lib/python3.5/site-packages/docutils/statemachine.py", line 239, in run
    context, state, transitions)
  File "/home/docs/checkouts/readthedocs.org/user_builds/co2mpas-ta/envs/latest/lib/python3.5/site-packages/docutils/statemachine.py", line 460, in check_line
    return method(match, context, next_state)
  File "/home/docs/checkouts/readthedocs.org/user_builds/co2mpas-ta/envs/latest/lib/python3.5/site-packages/docutils/parsers/rst/states.py", line 2745, in underline
    self.section(title, source, style, lineno - 1, messages)
  File "/home/docs/checkouts/readthedocs.org/user_builds/co2mpas-ta/envs/latest/lib/python3.5/site-packages/docutils/parsers/rst/states.py", line 326, in section
    self.new_subsection(title, lineno, messages)
  File "/home/docs/checkouts/readthedocs.org/user_builds/co2mpas-ta/envs/latest/lib/python3.5/site-packages/docutils/parsers/rst/states.py", line 394, in new_subsection
    node=section_node, match_titles=True)
  File "/home/docs/checkouts/readthedocs.org/user_builds/co2mpas-ta/envs/latest/lib/python3.5/site-packages/docutils/parsers/rst/states.py", line 281, in nested_parse
    node=node, match_titles=match_titles)
  File "/home/docs/checkouts/readthedocs.org/user_builds/co2mpas-ta/envs/latest/lib/python3.5/site-packages/docutils/parsers/rst/states.py", line 195, in run
    results = StateMachineWS.run(self, input_lines, input_offset)
  File "/home/docs/checkouts/readthedocs.org/user_builds/co2mpas-ta/envs/latest/lib/python3.5/site-packages/docutils/statemachine.py", line 239, in run
    context, state, transitions)
  File "/home/docs/checkouts/readthedocs.org/user_builds/co2mpas-ta/envs/latest/lib/python3.5/site-packages/docutils/statemachine.py", line 460, in check_line
    return method(match, context, next_state)
  File "/home/docs/checkouts/readthedocs.org/user_builds/co2mpas-ta/envs/latest/lib/python3.5/site-packages/docutils/parsers/rst/states.py", line 2745, in underline
    self.section(title, source, style, lineno - 1, messages)
  File "/home/docs/checkouts/readthedocs.org/user_builds/co2mpas-ta/envs/latest/lib/python3.5/site-packages/docutils/parsers/rst/states.py", line 326, in section
    self.new_subsection(title, lineno, messages)
  File "/home/docs/checkouts/readthedocs.org/user_builds/co2mpas-ta/envs/latest/lib/python3.5/site-packages/docutils/parsers/rst/states.py", line 394, in new_subsection
    node=section_node, match_titles=True)
  File "/home/docs/checkouts/readthedocs.org/user_builds/co2mpas-ta/envs/latest/lib/python3.5/site-packages/docutils/parsers/rst/states.py", line 281, in nested_parse
    node=node, match_titles=match_titles)
  File "/home/docs/checkouts/readthedocs.org/user_builds/co2mpas-ta/envs/latest/lib/python3.5/site-packages/docutils/parsers/rst/states.py", line 195, in run
    results = StateMachineWS.run(self, input_lines, input_offset)
  File "/home/docs/checkouts/readthedocs.org/user_builds/co2mpas-ta/envs/latest/lib/python3.5/site-packages/docutils/statemachine.py", line 239, in run
    context, state, transitions)
  File "/home/docs/checkouts/readthedocs.org/user_builds/co2mpas-ta/envs/latest/lib/python3.5/site-packages/docutils/statemachine.py", line 460, in check_line
    return method(match, context, next_state)
  File "/home/docs/checkouts/readthedocs.org/user_builds/co2mpas-ta/envs/latest/lib/python3.5/site-packages/docutils/parsers/rst/states.py", line 2320, in explicit_markup
    self.explicit_list(blank_finish)
  File "/home/docs/checkouts/readthedocs.org/user_builds/co2mpas-ta/envs/latest/lib/python3.5/site-packages/docutils/parsers/rst/states.py", line 2350, in explicit_list
    match_titles=self.state_machine.match_titles)
  File "/home/docs/checkouts/readthedocs.org/user_builds/co2mpas-ta/envs/latest/lib/python3.5/site-packages/docutils/parsers/rst/states.py", line 318, in nested_list_parse
    node=node, match_titles=match_titles)
  File "/home/docs/checkouts/readthedocs.org/user_builds/co2mpas-ta/envs/latest/lib/python3.5/site-packages/docutils/parsers/rst/states.py", line 195, in run
    results = StateMachineWS.run(self, input_lines, input_offset)
  File "/home/docs/checkouts/readthedocs.org/user_builds/co2mpas-ta/envs/latest/lib/python3.5/site-packages/docutils/statemachine.py", line 239, in run
    context, state, transitions)
  File "/home/docs/checkouts/readthedocs.org/user_builds/co2mpas-ta/envs/latest/lib/python3.5/site-packages/docutils/statemachine.py", line 460, in check_line
    return method(match, context, next_state)
  File "/home/docs/checkouts/readthedocs.org/user_builds/co2mpas-ta/envs/latest/lib/python3.5/site-packages/docutils/parsers/rst/states.py", line 2623, in explicit_markup
    nodelist, blank_finish = self.explicit_construct(match)
  File "/home/docs/checkouts/readthedocs.org/user_builds/co2mpas-ta/envs/latest/lib/python3.5/site-packages/docutils/parsers/rst/states.py", line 2330, in explicit_construct
    return method(self, expmatch)
  File "/home/docs/checkouts/readthedocs.org/user_builds/co2mpas-ta/envs/latest/lib/python3.5/site-packages/docutils/parsers/rst/states.py", line 2073, in directive
    directive_class, match, type_name, option_presets)
  File "/home/docs/checkouts/readthedocs.org/user_builds/co2mpas-ta/envs/latest/lib/python3.5/site-packages/docutils/parsers/rst/states.py", line 2122, in run_directive
    result = directive_instance.run()
  File "/home/docs/checkouts/readthedocs.org/user_builds/co2mpas-ta/envs/latest/lib/python3.5/site-packages/sphinx/ext/autodoc.py", line 1614, in run
    documenter.generate(more_content=self.content)
  File "/home/docs/checkouts/readthedocs.org/user_builds/co2mpas-ta/envs/latest/lib/python3.5/site-packages/sphinx/ext/autodoc.py", line 961, in generate
    self.add_content(more_content)
  File "/home/docs/checkouts/readthedocs.org/user_builds/co2mpas-ta/envs/latest/lib/python3.5/site-packages/schedula/ext/dispatcher.py", line 393, in add_content
    _plot(lines, dsp, dot_view_opt, self)
  File "/home/docs/checkouts/readthedocs.org/user_builds/co2mpas-ta/envs/latest/lib/python3.5/site-packages/schedula/ext/dispatcher.py", line 152, in _plot
    dot.sitemap.render(directory=dspdir, index=False)
  File "/home/docs/checkouts/readthedocs.org/user_builds/co2mpas-ta/envs/latest/lib/python3.5/site-packages/schedula/utils/drw/__init__.py", line 1065, in render
    cached_view(node, directory, context, rendered, header)
  File "/home/docs/checkouts/readthedocs.org/user_builds/co2mpas-ta/envs/latest/lib/python3.5/site-packages/schedula/utils/drw/__init__.py", line 1090, in cached_view
    context=context, header=header
  File "/home/docs/checkouts/readthedocs.org/user_builds/co2mpas-ta/envs/latest/lib/python3.5/site-packages/schedula/utils/drw/__init__.py", line 755, in view
    os.rename(fpath, filepath)
FileNotFoundError: [Errno 2] No such file or directory: '/home/docs/checkouts/readthedocs.org/user_builds/co2mpas-ta/checkouts/latest/doc/_build/_dispatchers/dispatcher-fcc114a407f2bc5d83ebb5e51397ffe5838ff37a/predict_wltp_l.html/tmpvnu7nq5b.svg' -> '/home/docs/checkouts/readthedocs.org/user_builds/co2mpas-ta/checkouts/latest/doc/_build/_dispatchers/dispatcher-fcc114a407f2bc5d83ebb5e51397ffe5838ff37a/predict_wltp_l.html/engine_model-0.html'

Exception occurred:
  File "/home/docs/checkouts/readthedocs.org/user_builds/co2mpas-ta/envs/latest/lib/python3.5/site-packages/schedula/utils/drw/__init__.py", line 755, in view
    os.rename(fpath, filepath)
FileNotFoundError: [Errno 2] No such file or directory: '/home/docs/checkouts/readthedocs.org/user_builds/co2mpas-ta/checkouts/latest/doc/_build/_dispatchers/dispatcher-fcc114a407f2bc5d83ebb5e51397ffe5838ff37a/predict_wltp_l.html/tmpvnu7nq5b.svg' -> '/home/docs/checkouts/readthedocs.org/user_builds/co2mpas-ta/checkouts/latest/doc/_build/_dispatchers/dispatcher-fcc114a407f2bc5d83ebb5e51397ffe5838ff37a/predict_wltp_l.html/engine_model-0.html'
The full traceback has been saved in /tmp/sphinx-err-rcpz42zi.log, if you want to report the issue to the developers.
Please also report this if it was a user error, so that a better error message can be provided next time.
A bug report can be filed in the tracker at <https://github.com/sphinx-doc/sphinx/issues>. Thanks!
Command time: 121s Return: 1

NOTE that this build also reports this WARNING at the top:

...
[autosummary] generating autosummary for: AUTHORS.rst, changes.rst, co2mpas_RelNotes.rst, developers.rst, explanation.rst, glossary.rst, index.rst, install.rst, reference.rst, usage.rst
One of `xlrd`, `...` libraries is needed, will crash later!
Failed LOADING plugin(EntryPoint.parse('pandas_filters = pandalone.xleash._pandas_filters:load_as_xleash_plugin [pandas]')@pandalone 0.2.3) due to: No module named 'pandas.api'
Traceback (most recent call last):
  File "/home/docs/checkouts/readthedocs.org/user_builds/co2mpas-ta/envs/latest/lib/python3.5/site-packages/pandalone/xleash/__init__.py", line 940, in _init_plugins
    plugin_loader = ep.load()
  File "/home/docs/checkouts/readthedocs.org/user_builds/co2mpas-ta/envs/latest/lib/python3.5/site-packages/pkg_resources/__init__.py", line 2291, in load
    return self.resolve()
  File "/home/docs/checkouts/readthedocs.org/user_builds/co2mpas-ta/envs/latest/lib/python3.5/site-packages/pkg_resources/__init__.py", line 2297, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
  File "/home/docs/checkouts/readthedocs.org/user_builds/co2mpas-ta/envs/latest/lib/python3.5/site-packages/pandalone/xleash/_pandas_filters.py", line 21, in <module>
    import pandas.api.types as pdtypes
ImportError: No module named 'pandas.api'
Failed LOADING plugin(EntryPoint.parse('xlrd_be = pandalone.xleash.io._xlrd:load_as_xleash_plugin [xlrd]')@pandalone 0.2.3) due to: No module named 'xlrd'
Traceback (most recent call last):
  File "/home/docs/checkouts/readthedocs.org/user_builds/co2mpas-ta/envs/latest/lib/python3.5/site-packages/pandalone/xleash/__init__.py", line 940, in _init_plugins
    plugin_loader = ep.load()
  File "/home/docs/checkouts/readthedocs.org/user_builds/co2mpas-ta/envs/latest/lib/python3.5/site-packages/pkg_resources/__init__.py", line 2291, in load
    return self.resolve()
  File "/home/docs/checkouts/readthedocs.org/user_builds/co2mpas-ta/envs/latest/lib/python3.5/site-packages/pkg_resources/__init__.py", line 2297, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
  File "/home/docs/checkouts/readthedocs.org/user_builds/co2mpas-ta/envs/latest/lib/python3.5/site-packages/pandalone/xleash/io/_xlrd.py", line 21, in <module>
    from xlrd import (xldate, XL_CELL_DATE, XL_CELL_EMPTY, XL_CELL_TEXT,
ImportError: No module named 'xlrd'
[autosummary] generating autosummary for: 
...

importing `schedula` imports every module there is

The way it is structure this project, when importing schedula.__init__.py, it imports recursively all modules beneath them. In modern computers this is fast, but in general it should be possible to selectively import what is needed.

Dispatch directive pfails with missing search_image_for_language()

Building sites with dispatch directive fails (see co2mpas build)

Traceback (most recent call last):
  File "/home/docs/checkouts/readthedocs.org/user_builds/co2mpas-ta/envs/dev/lib/python3.4/site-packages/sphinx/cmdline.py", line 244, in main
    app.build(opts.force_all, filenames)
  File "/home/docs/checkouts/readthedocs.org/user_builds/co2mpas-ta/envs/dev/lib/python3.4/site-packages/sphinx/application.py", line 266, in build
    self.builder.build_update()
  File "/home/docs/checkouts/readthedocs.org/user_builds/co2mpas-ta/envs/dev/lib/python3.4/site-packages/sphinx/builders/__init__.py", line 251, in build_update
    'out of date' % len(to_build))

[...]

  File "/home/docs/checkouts/readthedocs.org/user_builds/co2mpas-ta/envs/dev/lib/python3.4/site-packages/docutils/parsers/rst/states.py", line 2311, in explicit_construct
    return method(self, expmatch)
  File "/home/docs/checkouts/readthedocs.org/user_builds/co2mpas-ta/envs/dev/lib/python3.4/site-packages/docutils/parsers/rst/states.py", line 2054, in directive
    directive_class, match, type_name, option_presets)
  File "/home/docs/checkouts/readthedocs.org/user_builds/co2mpas-ta/envs/dev/lib/python3.4/site-packages/docutils/parsers/rst/states.py", line 2103, in run_directive
    result = directive_instance.run()
  File "/home/docs/checkouts/readthedocs.org/user_builds/co2mpas-ta/envs/dev/lib/python3.4/site-packages/schedula/ext/dispatcher.py", line 255, in run
    argument = search_image_for_language(self.arguments[0], env)
NameError: name 'search_image_for_language' is not defined

Exception occurred:
  File "/home/docs/checkouts/readthedocs.org/user_builds/co2mpas-ta/envs/dev/lib/python3.4/site-packages/schedula/ext/dispatcher.py", line 255, in run
    argument = search_image_for_language(self.arguments[0], env)
NameError: name 'search_image_for_language' is not defined

It seems that search_image_for_language() was introduced since sphinx-1.4+.
So an appropriate versioned-dependency must be set in this project.
(please remove import * occurrences, it makes things difficult to debug, among others).

chore: filter out commits from irrelevant files in git history

Cloning schedula fetch all co2mpas history that's unrelated to this project.
git filter-branch is your friend.

Build as "universal" wheel

Currently packaged 'wheels' are for the specific platform and python-version,
but actually this project contains pure-python code so it can "universal".

Please the following section in setup.py:

[bdist_wheel]
universal = 1

Getting node ID with % crashes

In alg.py:get_unused_node_id if the initial_guess is a string with the value % this function will crash since node_id_format will equal %<%d> and then when trying to set node_id = node_id_format % n() it will crash with the following:

ValueError: unsupported format character '<' (0x3c) at index 1

This is because the % is not escaped. If it was escaped so it was set to %% then it would work fine.

UNC problem on Windows & conda when launching DOT to render plots

There seems to be problem on Windows with a UNC + CMD shell when co2mpas (also co2wui#146) try to launch the internal flask-server serving the plots:

(base) C:\Users\vagrant\Documents>co2mpas plot
 * Serving Flask app "C:\Users\vagrant\Documents\cache_plot" (lazy loading)
 * Environment: production
   WARNING: This is a development server. Do not use it in a production deployment.
   Use a production WSGI server instead.
 * Debug mode: off
2019-10-12 09:34:12,666: INFO:werkzeug: * Running on http://127.0.0.1:5000/ (Press CTRL+C to quit)
2019-10-12 09:34:13,338: INFO:co2mpas:Done! [17.17 sec]
2019-10-12 09:34:14,978: INFO:werkzeug:127.0.0.1 - - [12/Oct/2019 09:34:14] "GET / HTTP/1.1" 200 -
2019-10-12 09:34:16,510: INFO:werkzeug:127.0.0.1 - - [12/Oct/2019 09:34:16] "GET /favicon.ico HTTP/1.1" 404 -
2019-10-12 09:34:16,556: INFO:werkzeug:127.0.0.1 - - [12/Oct/2019 09:34:16] "GET /favicon.ico HTTP/1.1" 404 -
'\\?\C:\Users\vagrant\Documents\cache_plot\static'
CMD.EXE was started with the above path as the current directory.
UNC paths are not supported.  Defaulting to Windows directory.
Error: dot: can't open tmpdx48cqh0
2019-10-12 09:34:29,556:ERROR:schedula.utils.drw:dot could not render C:\Users\vagrant\Documents\cache_plot\static\core.html due to:
 CalledProcessError(2, ['dot.bat', '-Tsvg', '-O', 'tmpdx48cqh0'])
2019-10-12 09:34:29,556: INFO:werkzeug:127.0.0.1 - - [12/Oct/2019 09:34:29] "GET /core.html HTTP/1.1" 404 -

Loop detection and remedy

Is it possible to have a loop detection sytem?

both on demand, as in: "dsp.get_loops()"

and on compile: "dsp=Dispatcher(throw_exception_on_loops=True)"

DRY when adding sub-dispatch nodes

When adding a subdispatch-node with add_dispatcher() currently you have to specify IO mappings even for names that are identical.

I would add 2 new kwds in that method, inp_names & out_names, that accept lists of data-names that are assumed to have 1-to-1 mappings.

Split `schedula` into 2 or more packages, with the one with the very barebone serial small implementation

Is your feature request related to a problem? Please describe.

Sometimes when building thin abstraction layers for data conversion libraries one needs an impl like schedula that will determine a path in the form of chain of calls (i.e. rst -> HTML -> MarkDown -> ANSI-highlighted MarkDown) between backends. It looks like the thing schedula was made for. But one needs one's library be small. In some cases the integrators of the lib may want to vendor the dependencies, that means not only the lib itself must be as small as possible, but alsomits deps like schedula.

Describe the solution you'd like

schedula is separated into schedula with bells and whistles, schedula-concurrent with everything parallel and async and schedula_core doing the essential functionality, without any concurrency and other bells and whistles. It also may make sense to reuse third-party small and minimal, packages for Dijkstra algorythm.
each package is put into anmown git repo to minimize download size (GitHub doesn't support partial fetches).

Describe alternatives you've considered
Not using schedula.

	value = self._evaluate_function(
	(kwargs,), node_id, node_attr, attr
	)
	except KeyboardInterrupt as ex:
	raise ex
	except Exception as ex:
	if 'started' in attr:
	dt = datetime.today() - attr['started']
	attr['duration'] = dt
	# Some error occurs.
	msg = "Failed DISPATCHING '%s' due to:\n %r"
	self._warning(msg, node_id, ex)
	return False
	else:
	# Data node that has just one estimation value.
	value = list(est.values())[0]['value']

	else: # Use the estimation function of node.
	try:
	# Dict of all data node estimations.
	kwargs = {k: v['value'] for k, v in est.items()}

	# Evaluate output.
	value = self._evaluate_function(
	(kwargs,), node_id, node_attr, attr
	)
	except KeyboardInterrupt as ex:
	raise ex
	except Exception as ex:
	if 'started' in attr:
	attr['duration'] = datetime.today() - attr['started']
	# Is missing estimation function of data node or some error.
	msg = "Failed DISPATCHING '%s' due to:\n %r"
	self._warning(msg, node_id, ex)
	return False
	try:
	# Apply filters to output.
	value = self._apply_filters(value, node_id, node_attr, attr)
	except KeyboardInterrupt as ex:
	raise ex
	except Exception as ex:
	if 'started' in attr:
	attr['duration'] = datetime.today() - attr['started']
	# Some error occurs.
	msg = "Failed DISPATCHING '%s' due to:\n %r"
	self._warning(msg, node_id, ex)
	return False

	if value is not NONE: # Set data output.
	self[node_id] = value

	if 'started' in attr:
	attr['duration'] = datetime.today() - attr['started']

	if 'callback' in node_attr: # Invoke callback func of data node.
	try:
	# noinspection PyCallingNonCallable
	node_attr['callback'](value)
	except KeyboardInterrupt as ex:
	raise ex
	except Exception as ex: