mli / new-docs Goto Github PK

When you click a menu item on the left, it would be a better experience if that section expanded automatically.
So, when a menu item is clicked, like "API", it would load the index page for API. Then it would also expand the menu for API.

Instead of just this:

You would get this:

This will help navigating the API because as you click on items, you don't always realize there are navigation options to use on the left because they're hidden behind a tiny arrow button.

The error/warning is not very helpful.

http://ci.mxnet.io/blue/organizations/jenkins/mxnet-new-docs/detail/aaron_porting_content/7/pipeline

The build seems fine when I run with #70 and completes on my staging server.

Try: https://beta.mxnet.io/search.html?q=LBSGD
No results... it just hangs in "Searching" mode.

There's a js error:

Uncaught TypeError: Cannot read property 'textContent' of undefined
    at Object.htmlToText (searchtools.js:66)
    at Object.makeSearchSummary (searchtools.js:483)
    at Object.complete (searchtools.js:277)
    at i (jquery.js:2)
    at Object.fireWith (jquery.js:2)
    at A (jquery.js:4)
    at XMLHttpRequest.<anonymous> (jquery.js:4)

After doing this setup on different operating systems I'm seeing that Sphinx is using either 1.8.5 on mac or 2.0.0 on Windows. I'm sure this is going to lead to some weird problems.

When you try to update the environment with conda env update -f environment.yml you get a warning.

Warning:

Warning: you have pip-installed dependencies in your environment file, but you do not list pip itself as one of your conda dependencies.  Conda may not use the correct pip to install your packages, and they may end up in the wrong place.  Please add an explicit pip dependency.  I'm adding one for you, but still nagging you.

There are over 100-200k build warnings.

copying static files... done
copying extra files... done
dumping search index in English (code: en) ... done
dumping object inventory... done
build succeeded, 209299 warnings.

Here are some samples:

Failed to import:

WARNING: [autosummary] failed to import 'mxnet.io.DataIter': no module named mxnet.io.DataIter

Pandoc version warning:

  nbconvert.utils.pandoc.check_pandoc_version()
/home/markhama/anaconda3/envs/mxnet-docs/lib/python3.7/site-packages/nbsphinx.py:1174: RuntimeWarning: You are using an unsupported version of pandoc (2.2.3.2).
Your version must be at least (1.12.1) but less than (2.0.0).
Refer to http://pandoc.org/installing.html.
Continuing with doubts...

Failed to import module:

WARNING: autodoc: failed to import module 'ndarray.sparse' from module 'mxnet'; the following exception was raised:
...
OSError: /lib64/libc.so.6: version `GLIBC_2.16' not found (required by /home/markhama/anaconda3/envs/mxnet-docs/lib/python3.7/site-packages/mxnet/libmxnet.so)

Failed to import class:

WARNING: autodoc: failed to import class 'ndarray.sparse.CSRNDArray' from module 'mxnet'; the following exception was raised:
...
OSError: libcudart.so.9.2: cannot open shared object file: No such file or directory

Failed to import module:

WARNING: autodoc: failed to import module 'monitor' from module 'mxnet'; the following exception was raised:

Unknown document:

/home/ubuntu/new-docs/python/build/api/symbol/mxnet.symbol.Symbol.rst:302: WARNING: toctree references unknown document 'api/symbol/_autogen/Symbol.debug_str'

Failed to import:

/home/ubuntu/new-docs/python/build/api/symbol/mxnet.symbol.Symbol.rst:314: WARNING: failed to import Symbol.clip

Toctree contains reference to nonexisting document:

/home/ubuntu/new-docs/python/build/guide/index.rst:132: WARNING: toctree contains reference to nonexisting document 'guide/   image/index'

File not found:

/home/ubuntu/new-docs/python/build/guide/modules/gluon/custom-layer.ipynb:25: WARNING: File not found: 'guide/modules/gluon/model-construction.md'

While trying to solve #73 I ran into this other issue which is captured in a Conda issue. It mention there is a fix in Conda v4.6.13

It looks like an upgrade to conda, or recommending a minimum version might solve the problem with updating nbsphinx to master in environment.yml.

Throws a bunch of these during build:

/home/ubuntu/new-docs/python/build/api/c_predict_api.rst:18: WARNING: doxygenfunction: Cannot find file: /home/ubuntu/new-docs/python/docs/doxygen/xml/index.xml

This error is happening a lot.
Error:

  nbconvert.utils.pandoc.check_pandoc_version()
/home/ubuntu/nbsphinx/src/nbsphinx.py:1174: RuntimeWarning: You are using an unsupported version of pandoc (2.2.3.2).
Your version must be at least (1.12.1) but less than (2.0.0).
Refer to http://pandoc.org/installing.html.
Continuing with doubts...

I found that it is a known issue that has been patched in nbsphinx master. However, I have been unable to get the environment.yml file to use the following command to install master.

  - git+https://github.com/spatialaudio/nbsphinx.git@master

I tried a wide range of approaches to installing master, but I'm stuck with 0.4.2 or other errors that prevent a forced upgrade.

Compare:
https://beta.mxnet.io/api/ndarray/index.html
to
https://github.com/mli/new-docs/pull/122/files#diff-cc7e543ab494e686223548eaa26c0cf2

Is CI not running? I'm seeing other strange behavior too. The navigation seems broken.

When trying to write code inline with a link to API docs the rendering seems to be messed up.

[`code`](example.com)

Should render as code but instead it is shown as...

`nn.Sequential <http://beta.mxnet.io/api/gluon/_autogen/mxnet.gluon.nn.Sequential.html#mxnet.gluon.nn.Sequential>`__ 

See http://beta.mxnet.io/guide/packages/gluon/parameters.html as an example.

Can't seem to host images on the website. Solution might be to add a step to the Makefile to copy across images. @astonzhang how are you handling this on the D2L site?

Issue

When searching for a method, I might prefer a particular language or API but I can't tell what is what.

Expected Behavior

I should see some section or API info in the search result.

I've had to disable the disqus plugin to get builds to run.

I put this in the config in a PR so that anyone wanting to work on the site can build without issue. I can split it out if there are updates in the PR that are still in debate.

The following error was witnessed after adding content.

/home/ubuntu/new-docs/python/build/guide/modules/gluon/index.rst:18: WARNING: Content block expected for the "container" directive; none found.
Exception occurred:
  File "/home/ubuntu/anaconda3/envs/mxnet-docs/lib/python3.7/site-packages/mxtheme/card.py", line 25, in run
    cid = nodes.make_id("card-{}".format(options['title']))
KeyError: 'title'

Actual issue: a tab (or three spaces) is missing in front of a .. card::. Error handling/messaging could be improved.

https://beta.mxnet.io/guide/performance/float16.html#using-the-gluon-api

Within https://beta.mxnet.io/api/symbol/_autogen/mxnet.symbol.Symbol.softmax.html#mxnet.symbol.Symbol.softmax

if I click on the hyperlink for "softmax()" within the comment

The arguments are the same as for softmax(), with this array as data.

it just takes me to the page I'm on

Sphinx 2.0 will break some things... since the Windows setup uses 2.0 maybe things are already breaking when running there...

/home/ubuntu/anaconda3/envs/mxnet-cuda10-docs/lib/python3.7/site-packages/sphinx/builders/html.py:1171: RemovedInSphinx20Warning: Now html_sidebars only allows list of sidebar templates as a value. Support for a string value will be removed at Sphinx-2.0.
  self.add_sidebars(pagename, ctx)
/home/ubuntu/anaconda3/envs/mxnet-cuda10-docs/lib/python3.7/site-packages/sphinx/builders/html.py:138: RemovedInSphinx20Warning: builder.css_files is deprecated. Please use app.add_stylesheet() instead.
  ret += other

Typo - Sagemaer https://beta.mxnet.io/index.html
Typo - Overivew https://beta.mxnet.io/api/index.html

Alignment... indent needed (or removed) in the tiles:

Label inconsistent for "Python users":

The order of title/description seems backwards:

Module link from Guide is broken:

Needs descriptions for each section:

• https://beta.mxnet.io/guide/performance/index.html
• https://beta.mxnet.io/guide/deploy/index.html

There are a lot of fancy card options out there that I didn't get a chance to implement. Now that we're relying on cards more and more in our UX design, it would be great to expand the card functionality further for more diverse design needs. Potential extensions:

• cards with images/video clips
• cards with expansion panel that appear on hover.

This is more like 7 different errors together:

/home/ubuntu/anaconda3/envs/mxnet-cuda10-docs/lib/python3.7/site-packages/mxnet/gluon/parameter.py:docstring of mxnet.gluon.Parameter:44: WARNING: duplicate object description of mxnet.gluon.Parameter.grad_req, other instance in /home/ubuntu/new-docs/python/build/api/gluon/_autogen/mxnet.gluon.Parameter.grad_req.rst, use :noindex: for one of them

/home/ubuntu/anaconda3/envs/mxnet-cuda10-docs/lib/python3.7/site-packages/mxnet/gluon/block.py:docstring of mxnet.gluon.nn.HybridBlock:1: WARNING: duplicate object description of mxnet.gluon.nn.HybridBlock, other instance in /home/ubuntu/new-docs/python/build/api/gluon/mxnet.gluon.HybridBlock.rst, use :noindex: for one of them

/home/ubuntu/anaconda3/envs/mxnet-cuda10-docs/lib/python3.7/site-packages/mxnet/gluon/block.py:docstring of mxnet.gluon.nn.SymbolBlock:1: WARNING: duplicate object description of mxnet.gluon.nn.SymbolBlock, other instance in /home/ubuntu/new-docs/python/build/api/gluon/mxnet.gluon.SymbolBlock.rst, use :noindex: for one of them

/home/ubuntu/anaconda3/envs/mxnet-cuda10-docs/lib/python3.7/site-packages/mxnet/gluon/parameter.py:docstring of mxnet.gluon.Parameter:1: WARNING: duplicate object description of mxnet.gluon.Parameter, other instance in /home/ubuntu/new-docs/python/build/api/gluon/mxnet.gluon.Parameter.rst, use :noindex: for one of them

/home/ubuntu/anaconda3/envs/mxnet-cuda10-docs/lib/python3.7/site-packages/mxnet/gluon/parameter.py:docstring of mxnet.gluon.Parameter:44: WARNING: duplicate object description of mxnet.gluon.Parameter.grad_req, other instance in /home/ubuntu/new-docs/python/build/api/gluon/mxnet.gluon.Parameter.rst, use :noindex: for one of them

/home/ubuntu/anaconda3/envs/mxnet-cuda10-docs/lib/python3.7/site-packages/mxnet/gluon/parameter.py:docstring of mxnet.gluon.Parameter:52: WARNING: duplicate object description of mxnet.gluon.Parameter.lr_mult, other instance in /home/ubuntu/new-docs/python/build/api/gluon/mxnet.gluon.Parameter.rst, use :noindex: for one of them

/home/ubuntu/anaconda3/envs/mxnet-cuda10-docs/lib/python3.7/site-packages/mxnet/gluon/parameter.py:docstring of mxnet.gluon.Parameter:60: WARNING: duplicate object description of mxnet.gluon.Parameter.wd_mult, other instance in /home/ubuntu/new-docs/python/build/api/gluon/mxnet.gluon.Parameter.rst, use :noindex: for one of them

Building API docs takes an excessive amount of time. When updating a single API, you should get a fast build where Sphinx keeps all unchanged files. Something in how this build is setup breaks that feature.

The cards are links, but for some reason, they don't behave like them. You can't open them in new tabs or new windows.

There a really wide margin on right that doesn't scroll which makes it seem like the page has no further content. However, if you position your mouse in the middle of the page, the scrolling works.

Around here:
https://beta.mxnet.io/guide/performance/tensorRt.html#before

https://beta.mxnet.io/api/gluon-related/mxnet.io.html#iterators

This code doesn't render properly in Chime.

<title>Install — Apache MXNet  documentation</title>

Refer to apache/mxnet#13090
Maybe just fixing it here would suffice...

To support click tracking on the website, we need unique id's for each span that represents an installation option.
This already partially implemented:

<span class="option mdl-button mdl-js-button mdl-js-ripple-effect mdl-button--raised" id="macos" data-upgraded=",MaterialButton,MaterialRipple">Macos<span class="mdl-button__ripple-container"><span class="mdl-ripple"></span></span></span>

But some buttons (like LOCAL and LINUX) don't have an id for some reason...

Error - which can break the build when an older version of nbsphinx is in place:

WARNING: toctree references unknown document 'api/ndarray/_autogen/mxnet.random.seed'

Probably from here:
https://github.com/mli/new-docs/blame/master/python/api/ndarray/routines.rst#L295

Why does the pattern change in the docs?

   ...
    random.shuffle
    random.uniform
# then it switches up to this...
    mxnet.random.seed

Refer to apache/mxnet#13429 for more info.
Unrelated maybe?

Seeing a lot of variants of MXNet and Gluon on the site. We should ensure consistency of terms when most of the content has been added. e.g. MXNet, Gluon, MXNet Gluon, MxNet, Mxnet, mxnet, gluon.

The giraffe image in develop/guide/gluon/data_augmentation.html#Data-preproecessing looks incorrect.
New docs:

Old docs:

Also, all of the other images in this tutorial seem to be rendering incorrectly.

Much of the content of the site is being hosted on other sites like the toolkit sites. When this happens there's no file locally for Sphinx to link up, but it still adds an entry to the navigation and makes it look like there are sub items when there are not.

The navigation should have some level of control/customization that doesn't also break Sphinx when you try to work around it.

Right now you get both Sphinx errors and a broken UX.

I tried setting maxdepth to 0 to stop the nav from including a carrot when there's nothing to show, but this does nothing to the nav. It seems like the theme needs updating to prevent this problem.

You also have the issue that only local items will appear in the navigation. This means that if you have 3 articles, but two of them are off-deck, then only one appears. This is only a problem because none of the articles should show up in the nav. But when you still get some, it is very confusing what is happening.

Trying to make sense of the autogen step here: https://github.com/mli/new-docs/blob/master/python/Makefile#L37
It refers to build/_templates/, but this isn't in the project.

Issue

Looking through the API, I see mx.nd.shape as an API's title. I decide to search for mx.nd.reshape. I get no results. Searching for reshape returns many more results than I want to see.

Expected Behavior

mx.nd.reshape yields a result.

It has a mention in the makefile that you can exclude markdown files by copying them to /build but this doesn't work.

It seems like a more robust method of scheduling what to convert or not is needed. I'm pretty sure having to copy every excluded file to build is not the best approach long term.

Error:

python build/md2ipynb.py install-pi.md build/./install-pi.ipynb
Traceback (most recent call last):
  File "/home/ubuntu/anaconda3/envs/mxnet-cuda10-docs/lib/python3.7/site-packages/nbformat/validator.py", line 256, in validate
    return validator.validate(nbjson, {'$ref' : '#/definitions/%s' % ref})
  File "/home/ubuntu/anaconda3/envs/mxnet-cuda10-docs/lib/python3.7/site-packages/jsonschema/validators.py", line 348, in validate
    raise error
jsonschema.exceptions.ValidationError: None is not valid under any of the given schemas

Failed validating 'oneOf' in schema['properties']['source']:
    {'oneOf': [{'type': 'string'},
               {'items': {'type': 'string'}, 'type': 'array'}]}

On instance['source']:
    None

Setup should be clear for Ubuntu and Amazon Linux and CentOS.
Currently Sphinx is set to v1.8.5 on any Linux, but 2.0.0 on Windows. We should try to get every platform to build with the same versions, preferably v2.0.0.

This article has several links that are not formatted properly and it needs editing / grammar / tech checks.

https://github.com/mli/new-docs/blob/master/python/guide/deploy/run-on-aws/use_ec2.rst

https://beta.mxnet.io/guide/module/gluon/init.html#Instantiating-a

1. Many of the code examples (or “usage”) of certain functions are actually in Python, not R. See for example: mx.symbol.ones_like, mx.symbol.square, etc. There’s generally Python code appearing all over the place in the R documentation, which should probably be translated to R.
2. mx.io.MNISTIter: “, optional” in description of prefetch.buffer argument (extra comma should be removed)
3. mx.nd.cast.storage.Rd \details{} block begins too early. This is actually a problem in many of the .Rd files: the \description{} and \details{} are actually contiguous blocks of text which have been arbitrarily split into these separate fields. Where the split between \description{} and \details{} takes place should be more sensibly chosen (definitely not in the middle of a sentence or mathematical definition). See also: mx.nd.dot, mx.nd.abs for other examples of this issue, which occurs all over the place.
4. mx.nd.Embedding: contains link to python API. Should probably specify this as “the Optimization API for the Python version of MXNet”
5. There is both: mx.nd.ctc.loss and mx.symbol.ctc_loss (seems redundant and one should probably be removed from the R package)
6. Function names ending in v1 are deprecated, e.g. “mx.nd.Convolution.v1” (should probably be removed from the R package)
7. mx.symbol.pad is missing .Rd documentation (does not automatically link to mx.symbol.Pad documention in R), whereas mx.nd.pad properly links to mx.nd.Pad documentation.
8. mx.symbol.Pad & mx.nd.Pad: "\title{pad:Pads" is missing space between 'pad:' and the subsequent description. More generally, should probably remove the function name from \title{} blocks to standardize things, since the function name is sometimes listed there (followed by colon) and sometimes not. However, please do not start including the function name in the \title{} field without colon, since my parser to convert the documentation into a Sphinx website currently critically relies on this property to hide the redundancy of the function name appearing again in the \title{} field.

Credits for this list - Jonas Mueller

When building on Windows in a Conda environment with make EVAL=0:

python build/md2ipynb.py guide/module/gluon/image-augmentation.md build/./guide/module/gluon/image-augmentation.ipynb
Traceback (most recent call last):
  File "build/md2ipynb.py", line 30, in <module>
    md2ipynb()
  File "build/md2ipynb.py", line 18, in md2ipynb
    notebook = reader.read(f)
  File "C:\Users\Aaron\AppData\Local\conda\conda\envs\mxnet-docs\lib\site-packages\nbformat\v4\rwbase.py", line 113, in read
    nbs = cast_unicode_py2(fp.read())
  File "C:\Users\Aaron\AppData\Local\conda\conda\envs\mxnet-docs\lib\encodings\cp1252.py", line 23, in decode
    return codecs.charmap_decode(input,self.errors,decoding_table)[0]
UnicodeDecodeError: 'charmap' codec can't decode byte 0x9d in position 2841: character maps to <undefined>
make: *** [Makefile:17: build/./guide/module/gluon/image-augmentation.ipynb] Error 1

Not sure why these are in there... but anyways the environment.yml file does not work for mac.
When trying to create the conda environment I get this error.

ResolvePackageNotFound: 
  - torchvision-cpu -> pytorch-cpu[version='>=1.0.0']
  - pytorch-cpu

I was able to get around it by removing -cpu from pytorch and torchvision.
Even so, this should work out of the box no matter what OS you're trying to build this on... or the instructions should guide you...

The search previews aren't very helpful at the moment.

As reported with my NDArray tutorials PR, new tutorials don't pass validation. What is required? Can we get this info added to the readme?

jsonschema.exceptions.ValidationError: None is not valid under any of the given schemas

Failed validating 'oneOf' in schema['properties']['source']:
    {'oneOf': [{'type': 'string'},
               {'items': {'type': 'string'}, 'type': 'array'}]}

On instance['source']:
    None

Running this on macOS:

$ make EVAL=0
sort: invalid option -- h
Try `sort --help' for more information.

Maybe this gsort utility hack will fix it, but it seems we shouldn't use features that require so much setup and troubleshooting.

Search for "k mean": https://beta.mxnet.io/search.html?q=k+mean
Click on a result.
Witness yellow blocks everywhere.

http://beta.mxnet.io/guide/packages/optimizer/optimizer.html

I'm noticing that the APIs reference docs are not alphabetized. I think it is ok to have them grouped in a logical way, but then listing them alphabetically makes sense.

If there's some reason not to do this, then it should be described somewhere, so that the logic is can be referenced.

For example:
https://github.com/mli/new-docs/blob/master/api/ndarray/mxnet.ndarray.NDArray.rst#attributes