Here's my Mastodon feed.
jckantor / nbpages Goto Github PK
View Code? Open in Web Editor NEWCommand line tool to maintain a repository of Jupyter notebooks.
Home Page: https://jckantor.github.io/nbpages/
License: MIT License
Command line tool to maintain a repository of Jupyter notebooks.
Home Page: https://jckantor.github.io/nbpages/
License: MIT License
Here's my Mastodon feed.
Perhaps #26 will also fix this. I am opening a separate issue so we remember to test it.
This would be the most helpful to a new user.
Steps including:
templates/index.md.tpl
templates/notebook_header.tpl
templates/nbpages.tpl
(advanced users only)nbpages.cfg
(advanced users only)Suggestion: Replace figure paths with urls to GitHub when rewriting the notebook
Can we hard code the name 'nbpages.cfg'? Currently there is a --config option to allow the possibility of an alternate or multiple config file. But would that ever be necessary? For the case of multiple destinations, nbpages could always be run from the destination directory with a unique nbpages.cfg.
Indexing is turning out to be one of the most useful features of nbpages. So far there are tag, python module, figure, and data indexes. Additional indexes that might be useful for notebook development are
Currently some of the indexes are cached, which makes it problematic to do rewriting for 'under-construction' tags. (issue #28).
Propose a new index class created with
tag_index = Index()
tag_index.index(pattern)
tag_index.write_html(title)
@jckantor I am trying to set up https://github.com/ndcbe/cbe-xx258 to be the public version of https://github.com/ndcbe/CBExx258-instructor. After creating the public repo, I tried to run nbpages --setup
, but got the following error message:
Traceback (most recent call last):
File "/anaconda3/bin/nbpages", line 11, in <module>
load_entry_point('nbpaages', 'console_scripts', 'nbpages')()
File "/anaconda3/lib/python3.7/site-packages/pkg_resources/__init__.py", line 489, in load_entry_point
return get_distribution(dist).load_entry_point(group, name)
File "/anaconda3/lib/python3.7/site-packages/pkg_resources/__init__.py", line 2793, in load_entry_point
return ep.load()
File "/anaconda3/lib/python3.7/site-packages/pkg_resources/__init__.py", line 2411, in load
return self.resolve()
File "/anaconda3/lib/python3.7/site-packages/pkg_resources/__init__.py", line 2417, in resolve
module = __import__(self.module_name, fromlist=['__name__'], level=0)
File "/Users/adowling/DowlingLab/Teaching/nbpages/src/nbpages/__init__.py", line 4, in <module>
from .nbcollection import Nb, NbCollection
File "/Users/adowling/DowlingLab/Teaching/nbpages/src/nbpages/nbcollection.py", line 20, in <module>
sys.exit(1)
NameError: name 'sys' is not defined
I then manually made nbpages.cfg
using the example here:
https://github.com/ndcbe/nbpages-testing-public/blob/master/nbpages.cfg
When I run nbpages --setup
, I get the following error message:
Traceback (most recent call last):
File "/anaconda3/bin/nbpages", line 11, in <module>
load_entry_point('nbpaages', 'console_scripts', 'nbpages')()
File "/anaconda3/lib/python3.7/site-packages/pkg_resources/__init__.py", line 489, in load_entry_point
return get_distribution(dist).load_entry_point(group, name)
File "/anaconda3/lib/python3.7/site-packages/pkg_resources/__init__.py", line 2793, in load_entry_point
return ep.load()
File "/anaconda3/lib/python3.7/site-packages/pkg_resources/__init__.py", line 2411, in load
return self.resolve()
File "/anaconda3/lib/python3.7/site-packages/pkg_resources/__init__.py", line 2417, in resolve
module = __import__(self.module_name, fromlist=['__name__'], level=0)
File "/Users/adowling/DowlingLab/Teaching/nbpages/src/nbpages/__init__.py", line 4, in <module>
from .nbcollection import Nb, NbCollection
File "/Users/adowling/DowlingLab/Teaching/nbpages/src/nbpages/nbcollection.py", line 24, in <module>
config = config["nbpages"]
File "/anaconda3/lib/python3.7/configparser.py", line 958, in __getitem__
raise KeyError(key)
KeyError: 'nbpages'
I am using the master branch of nbpages.
(base) esc306856:cbe67701-uncertainty-quantification adowling$ nbpages
- inserting header for 01.00-Introduction-to-Uncertainty-Quantification-and-Predictive-Sciences.ipynb
Traceback (most recent call last):
File "/anaconda3/bin/nbpages", line 11, in <module>
load_entry_point('nbpaages', 'console_scripts', 'nbpages')()
File "/Users/adowling/DowlingLab/Teaching/nbpages/src/nbpages/__main__.py", line 92, in main
notebooks.insert_headers()
File "/Users/adowling/DowlingLab/Teaching/nbpages/src/nbpages/nbcollection.py", line 365, in insert_headers
self.nbheader.insert(nb)
File "/Users/adowling/DowlingLab/Teaching/nbpages/src/nbpages/nbcollection.py", line 307, in insert
print('- inserting header for {0}'.format(nb.filenapatth_srme))
AttributeError: 'Chapter' object has no attribute 'filenapatth_srme'
I will commit the offending notebooks in https://github.com/ndcbe/cbe67701-uncertainty-quantification in a few minutes.
This will be a bit more complicated
This helps find "orphaned" data and figures. It is also a precursor to data and figure TOC.
For example, see the links to notebooks here: https://ndcbe.github.io/cbe67701-uncertainty-quantification/
This may be related to issue #13.
Need to complete integration with readthedocs
I installed nbpages
from the master branch this morning on a second computer (macOS X).
I am seeing the following error messages:
ModuleNotFoundError: No module named 'jupyter_contrib_nbextensions'
UserWarning: IPython3 lexer unavailable, falling back on Python 3
Are these missing dependencies for nbpages
?
I installed nbpages
by cloning the git repo and running pin install -e .
read_config.py was originally used to read the nbpages config. That functionality has been moved into main.py and nbcollection.py. Need to remove file, then test.
https://ndcbe.github.io/cbe67701-uncertainty-quantification/data_index.html
This is because I both give the name of the text file markdown description and I use it in the code. This is a low priority.
and clean out REAME.rst for notes to developers.
Or perhaps print the section and subsection name.
Below is output after adopting the changes in PR #24:
(base) esc306856:cbe67701-uncertainty-quantification adowling$ nbpages
- inserting header for 01.00-Introduction-to-Uncertainty-Quantification-and-Predictive-Sciences.ipynb
- inserting header for 02.00-Probability-and-Statistics-Preliminaries.ipynb
- inserting header for 03.00-Input-Parameter-Distributions.ipynb
- inserting navbar for 01.00-Introduction-to-Uncertainty-Quantification-and-Predictive-Sciences.ipynb
- inserting navbar for 01.00-Introduction-to-Uncertainty-Quantification-and-Predictive-Sciences.ipynb
- inserting navbar for 02.00-Probability-and-Statistics-Preliminaries.ipynb
- inserting navbar for 02.00-Probability-and-Statistics-Preliminaries.ipynb
- inserting navbar for 03.00-Input-Parameter-Distributions.ipynb
- inserting navbar for 03.00-Input-Parameter-Distributions.ipynb
- writing table of contents file
[NbConvertApp] Converting notebook docs/toc.ipynb to html
[NbConvertApp] Writing 274340 bytes to docs/toc.html
- writing tag index file
[NbConvertApp] Converting notebook docs/tag_index.ipynb to html
[NbConvertApp] Writing 273413 bytes to docs/tag_index.html
- writing index.md
Traceback (most recent call last):
File "/anaconda3/bin/nbpages", line 11, in <module>
load_entry_point('nbpaages', 'console_scripts', 'nbpages')()
File "/Users/adowling/DowlingLab/Teaching/nbpages/src/nbpages/__main__.py", line 98, in main
notebooks.write_html(NOTEBOOK_DST_DIR, os.path.join("templates", 'nbpages.tpl'))
File "/Users/adowling/DowlingLab/Teaching/nbpages/src/nbpages/nbcollection.py", line 464, in write_html
(body, resources) = html_exporter.from_notebook_node(nb.content)
File "/anaconda3/lib/python3.7/site-packages/nbconvert/exporters/html.py", line 96, in from_notebook_node
output, resources = super(HTMLExporter, self).from_notebook_node(nb, resources, **kw)
File "/anaconda3/lib/python3.7/site-packages/nbconvert/exporters/templateexporter.py", line 315, in from_notebook_node
output = self.template.render(nb=nb_copy, resources=resources)
File "/anaconda3/lib/python3.7/site-packages/nbconvert/exporters/templateexporter.py", line 113, in template
self._template_cached = self._load_template()
File "/anaconda3/lib/python3.7/site-packages/nbconvert/exporters/templateexporter.py", line 286, in _load_template
return self.environment.get_template(template_file)
File "/anaconda3/lib/python3.7/site-packages/jinja2/environment.py", line 830, in get_template
return self._load_template(name, self.make_globals(globals))
File "/anaconda3/lib/python3.7/site-packages/jinja2/environment.py", line 804, in _load_template
template = self.loader.load(self, name, globals)
File "/anaconda3/lib/python3.7/site-packages/jinja2/loaders.py", line 405, in load
return loader.load(environment, name, globals)
File "/anaconda3/lib/python3.7/site-packages/jinja2/loaders.py", line 125, in load
code = environment.compile(source, name, filename)
File "/anaconda3/lib/python3.7/site-packages/jinja2/environment.py", line 591, in compile
self.handle_exception(exc_info, source_hint=source_hint)
File "/anaconda3/lib/python3.7/site-packages/jinja2/environment.py", line 780, in handle_exception
reraise(exc_type, exc_value, tb)
File "/anaconda3/lib/python3.7/site-packages/jinja2/_compat.py", line 37, in reraise
raise value.with_traceback(tb)
File "/Users/adowling/DowlingLab/Teaching/cbe67701-uncertainty-quantification/templates/nbpages.tpl", line 15, in template
{% endblock any
File "/anaconda3/lib/python3.7/site-packages/jinja2/environment.py", line 497, in _parse
return Parser(self, source, name, encode_filename(filename)).parse()
File "/anaconda3/lib/python3.7/site-packages/jinja2/parser.py", line 901, in parse
result = nodes.Template(self.subparse(), lineno=1)
File "/anaconda3/lib/python3.7/site-packages/jinja2/parser.py", line 888, in subparse
self.stream.expect('block_end')
File "/anaconda3/lib/python3.7/site-packages/jinja2/lexer.py", line 384, in expect
self.name, self.filename)
jinja2.exceptions.TemplateSyntaxError: expected token 'end of statement block', got 'any'
Here are the contents of templates/nbpages.tpl
:
<!-- jinja2 template that extendes `full` to add cell tags in the html rendering of notebooks -->
{% extends 'full.tpl'%}
{% block any_cell %}
{% if cell['metadata'].get('tags', []) %}
<div style="background-color:white; border:thin solid grey; margin-left:95px; margin-right:6px">
{% for tag in cell['metadata'].get('tags', []) %}
<a href="https://{github_user_name}.github.io/{github_repo_name}/tag_index.html#{{ tag }}">{{ tag }}</a>
{% endfor %}
</div>
{{ super() }}
{% else %}
{{ super() }}
{% endif %}
{% endblock any
This will make it a little safer by requiring one to explicitly say publish.
Create three repositories owned by ndcbe:
nbpages-testing
(completely hosted in 1 repo)nbpages-testing-dev
(private, publishes to nbpages-testing-public
)nbpages-testing-public
(public)The repos would contain just a few example notebooks and a script to completely reset /docs
.
[NBPAGES]
github_repo_url = [email protected]:ndcbe/cbe67701-uncertainty-quantification.git
github_user_name = [email protected]:ndcbe
github_repo_name = cbe67701-uncertainty-quantification
github_pages_url = https://[email protected]:ndcbe.github.io/cbe67701-uncertainty-quantification
repository = [email protected]:ndcbe/cbe67701-uncertainty-quantification
page_title = cbe67701-uncertainty-quantification
page_url = https://[email protected]:ndcbe.github.io/cbe67701-uncertainty-quantification
Bug: Extra prefix in github_pages_url
. Perhaps this is because the repo is in an organization.
Installation of nbpages:
(teaching) esc306856:nbpages adowling$ pip install -e .
Obtaining file:///Users/adowling/DowlingLab/Teaching/nbpages
Requirement already satisfied: nbconvert in /anaconda3/lib/python3.7/site-packages (from nbpaages==0.0.1.dev12) (5.4.1)
Requirement already satisfied: nbformat in /anaconda3/lib/python3.7/site-packages (from nbpaages==0.0.1.dev12) (4.4.0)
Collecting notedown (from nbpaages==0.0.1.dev12)
Downloading https://files.pythonhosted.org/packages/d3/3a/d7c1817f3acb2e958b32fe85a35f52d270820fb9cebf1364b36c772cf3d0/notedown-1.5.1-py3-none-any.whl
Requirement already satisfied: mistune>=0.8.1 in /anaconda3/lib/python3.7/site-packages (from nbconvert->nbpaages==0.0.1.dev12) (0.8.4)
Requirement already satisfied: jinja2 in /anaconda3/lib/python3.7/site-packages (from nbconvert->nbpaages==0.0.1.dev12) (2.10)
Requirement already satisfied: pygments in /anaconda3/lib/python3.7/site-packages (from nbconvert->nbpaages==0.0.1.dev12) (2.3.1)
Requirement already satisfied: traitlets>=4.2 in /anaconda3/lib/python3.7/site-packages (from nbconvert->nbpaages==0.0.1.dev12) (4.3.2)
Requirement already satisfied: jupyter_core in /anaconda3/lib/python3.7/site-packages (from nbconvert->nbpaages==0.0.1.dev12) (4.4.0)
Requirement already satisfied: entrypoints>=0.2.2 in /anaconda3/lib/python3.7/site-packages (from nbconvert->nbpaages==0.0.1.dev12) (0.3)
Requirement already satisfied: bleach in /anaconda3/lib/python3.7/site-packages (from nbconvert->nbpaages==0.0.1.dev12) (3.1.0)
Requirement already satisfied: pandocfilters>=1.4.1 in /anaconda3/lib/python3.7/site-packages (from nbconvert->nbpaages==0.0.1.dev12) (1.4.2)
Requirement already satisfied: testpath in /anaconda3/lib/python3.7/site-packages (from nbconvert->nbpaages==0.0.1.dev12) (0.4.2)
Requirement already satisfied: defusedxml in /anaconda3/lib/python3.7/site-packages (from nbconvert->nbpaages==0.0.1.dev12) (0.5.0)
Requirement already satisfied: ipython-genutils in /anaconda3/lib/python3.7/site-packages (from nbformat->nbpaages==0.0.1.dev12) (0.2.0)
Requirement already satisfied: jsonschema!=2.5.0,>=2.4 in /anaconda3/lib/python3.7/site-packages (from nbformat->nbpaages==0.0.1.dev12) (3.0.1)
Requirement already satisfied: six in /anaconda3/lib/python3.7/site-packages (from notedown->nbpaages==0.0.1.dev12) (1.12.0)
Collecting pandoc-attributes (from notedown->nbpaages==0.0.1.dev12)
Downloading https://files.pythonhosted.org/packages/c2/0a/442cc9237dc997cd88155bdcb54bf86e703e699881f4134ecb733ccd670c/pandoc-attributes-0.1.7.tar.gz
Requirement already satisfied: MarkupSafe>=0.23 in /anaconda3/lib/python3.7/site-packages (from jinja2->nbconvert->nbpaages==0.0.1.dev12) (1.1.1)
Requirement already satisfied: decorator in /anaconda3/lib/python3.7/site-packages (from traitlets>=4.2->nbconvert->nbpaages==0.0.1.dev12) (4.4.0)
Requirement already satisfied: webencodings in /anaconda3/lib/python3.7/site-packages (from bleach->nbconvert->nbpaages==0.0.1.dev12) (0.5.1)
Requirement already satisfied: attrs>=17.4.0 in /anaconda3/lib/python3.7/site-packages (from jsonschema!=2.5.0,>=2.4->nbformat->nbpaages==0.0.1.dev12) (19.1.0)
Requirement already satisfied: pyrsistent>=0.14.0 in /anaconda3/lib/python3.7/site-packages (from jsonschema!=2.5.0,>=2.4->nbformat->nbpaages==0.0.1.dev12) (0.14.11)
Requirement already satisfied: setuptools in /anaconda3/lib/python3.7/site-packages (from jsonschema!=2.5.0,>=2.4->nbformat->nbpaages==0.0.1.dev12) (40.8.0)
Building wheels for collected packages: pandoc-attributes
Building wheel for pandoc-attributes (setup.py) ... done
Stored in directory: /Users/adowling/Library/Caches/pip/wheels/0f/43/b0/252730521a965a687748f0be111073e67ba229e640d147e65f
Successfully built pandoc-attributes
Installing collected packages: pandoc-attributes, notedown, nbpaages
Running setup.py develop for nbpaages
Successfully installed nbpaages notedown-1.5.1 pandoc-attributes-0.1.7
Attempting to configure my first repo:
(teaching) esc306856:cbe67701-uncertainty-quantification adowling$ nbpages --setup
Traceback (most recent call last):
File "/anaconda3/bin/nbpages", line 11, in <module>
load_entry_point('nbpaages', 'console_scripts', 'nbpages')()
File "/anaconda3/lib/python3.7/site-packages/pkg_resources/__init__.py", line 489, in load_entry_point
return get_distribution(dist).load_entry_point(group, name)
File "/anaconda3/lib/python3.7/site-packages/pkg_resources/__init__.py", line 2793, in load_entry_point
return ep.load()
File "/anaconda3/lib/python3.7/site-packages/pkg_resources/__init__.py", line 2411, in load
return self.resolve()
File "/anaconda3/lib/python3.7/site-packages/pkg_resources/__init__.py", line 2417, in resolve
module = __import__(self.module_name, fromlist=['__name__'], level=0)
File "/Users/adowling/DowlingLab/Teaching/nbpages/src/nbpages/__main__.py", line 7, in <module>
from .nbcollection import NbCollection
File "/Users/adowling/DowlingLab/Teaching/nbpages/src/nbpages/nbcollection.py", line 30, in <module>
assert os.path.exists(config_file), f"configuration file {config_file} not found"
AssertionError: configuration file nbpages.cfg not found
Need to either find nbpages.cfg
or warn it needs to be run for the main directory.
This means it will work the same for /notebooks and /docs.
How to handle the case with two repos? It would need to copy only the files that are used.
End goal: extend "tags" with chapter and section.
Proposal: With a special command-line option, if nbpages
finds a header cell with the tag under-construction
, it will strip out replace all of the subsequent non-header cells that under it with custom text. We can make the default "This notebook section is under construction". Check again soon."
Example with three markdown cells:
## Some time here
blah A blah A blab A
### Subtle point 1
blah B blah B blah B
### Subtle point 2
blah C blah C blah C
If I tagged the first cell with under-construction
, then the contents "blah B blah B, blah B" and "blah C blah C, blah C" would get replaced with "This notebook section is under construction".
Why is this helpful? I want to start migrating old class notebooks but I do not want to make the contents, besides the header structure, public. Some notebooks require a major text or example rewrite. I'd prefer to get everything online and "under construction" then systematically clean notebooks up one section at a time.
Need to verify support for two repositories:
The docs directory accumulates unused files after repeated publishing runs. Do a cleanup of
/docs
/docs/figures
/docs/data
before publish. But don't touch _config.yml or any file listed inside _config.yml
This helps screen that all of the notebooks use the same version of Python.
Possible enhancements:
Take a look at this notebook. Open it in Colab.
https://ndcbe.github.io/cbe67701-uncertainty-quantification/01.01-Contributed-Example.html
Running this line:
stock_data = pd.read_csv('./data/Stock_Data.csv')
Gave the following error:
---------------------------------------------------------------------------
FileNotFoundError Traceback (most recent call last)
<ipython-input-4-4072e866c75c> in <module>()
----> 1 stock_data = pd.read_csv('./data/Stock_Data.csv')
4 frames
/usr/local/lib/python3.6/dist-packages/pandas/io/parsers.py in __init__(self, src, **kwds)
1889 kwds["usecols"] = self.usecols
1890
-> 1891 self._reader = parsers.TextReader(src, **kwds)
1892 self.unnamed_cols = self._reader.unnamed_cols
1893
pandas/_libs/parsers.pyx in pandas._libs.parsers.TextReader.__cinit__()
pandas/_libs/parsers.pyx in pandas._libs.parsers.TextReader._setup_parser_source()
FileNotFoundError: [Errno 2] File ./data/Stock_Data.csv does not exist: './data/Stock_Data.csv'
Here is the problem: When writing to HTML, we need to point to the text file in GitHub.
@jckantor I would like to share this notebook with the class on Tuesday.
The python index doesn't match and parse comma separated lists of libraries from the import statement.
Mimic the syntax of Vocareum.
All of the testing so far has been with a repo owned by an individual user.
We are using nbpages for this class: https://github.com/ndcbe/cbe67701-uncertainty-quantification/
Students are each going to contribute an example notebook (I gave each one a template) via a pull request. Is there a way I can trigger nbpages to run automatically from GitHub on the master branch after a pull request? This would be convenient but is not critical.
https://help.github.com/en/actions/reference/events-that-trigger-workflows
Tags are sorted alphabetically. But the prefix is header section number which needs to be parsed to correct sort. This is a problem if any of the section number components is becomes more than on digit.
Search and report 'Traceback' in order to detect faulty code cells.
Should there be a destination directory for /data and /figures which, if specified, causes --publish to copy those, too?
In my current CBE 20258 notebooks, I use some HTML code to put a colored box around certain markdown cells. For example:
<div style="background-color: rgba(0,255,0,0.05) ; padding: 10px; border: 1px solid darkgreen;">
<b>Home Activity</b>: In the cell below, assign the value of 5 to the variable x. Then type Shift+Enter to execute the line.
</div>
This box is solid dark green which I use for all "home activities". I used a blue box for "class activities" and a red box for "important notes". Would it be possible to add a border and set the background color accordingly for cells with "home-activity", "class-activity", and "important-note" tags when publishing to the HTML pages? Currently, nbpages strips out this formatting, which I think is fine. I would much rather use tags.
Challenge: no clear format for data
remove excessive status messaging during --publish
Need testing classes.
Index usage of python libraries by search code cells for "from ... import" and "import ... " expressions. Display as "python_library_index" sorted by library, sublibrary with links to each instance of usage. Add link to the index file in index.html.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.