nebari-dev / nebari-docs Goto Github PK

Once we have finalised the main site design we need to ensure the docs follow the same design guide

Transfer and combination of two previous issues:
nebari-dev/nebari#792
nebari-dev/nebari#796

notion doc

Add (and improve) landing pages for each main diataxis category.

• #19
• #20
• #21
• #22
• #23
• #24

Upgraded content notion doc

• https://docs.qhub.dev/en/latest/source/user_guide/dashboard.html
• nebari-dev/nebari#1285
• nebari-dev/nebari#1070

Current documentation

Need to review the current documentation for accuracy and transform into a how to doc aimed at teaching admins how to enable it on nebari

Add basic pre-commit hooks to this repo

From the governance PR

1. Markdown - prettier of Executable books md formatter
2. YAML - prettier
3. Python:

• black
• isort
• pyupgrade

4. From pre-commit hooks:

• end of file
• check case conflict
• trailing whitespace
• check-executables have shebangs

@costrouc can you please confirm which tf formatted you were suggesting?

Given the work on Infracost, we need a short tutorial on how it's used.

nebari-dev/nebari#1340

Should be written in tandem with nebari-dev/nebari#16

notion doc

Summary

Should up-to date docs on Dask - effectively replacing / updating https://docs.qhub.dev/en/latest/source/user_guide/dask_gateway.html

Tasks to complete

• [DOC] - Tutorial: Using Dask (nebari-dev/nebari#12)
• [DOC] - How-to: configure Dask (nebari-dev/nebari#68)

Examples

Flyte integrations page https://docs.flyte.org/projects/cookbook/en/latest/integrations.html

This is an internal issue regarding how we are handling instance profiles across the Quansight and Quansight-beta deployments.

Action:

• Keep the same names/resources for the user instance profiles across both deployments;

Currently, the status for both lists are:

Quansight deployment:

Quansight-beta deployment:

The details for each profile can be found in the respective qhub-config.yaml's for each deployment.

Must include

• how to use qhub destroy
• how to destroy resources manually if terraform fails (GCP, Azure, AWS, DO)
• Add troubleshooting section on what resources need to be deleted if destroy fails.

Very simplified example:

Just to say this would be very useful, I've needed this. My currently checklist for manual destruction is:

GKE: Delete cluster
GKE: Delete services
GCE: delete all instance groups (shouldn't be necessary if deleted GKE cluster)
GCE: delete all disks (shouldn't be necessary if deleted GKE cluster)
GCS: delete terraform state bucket
IAM: Delete the service account create by qhub
Also I then delete all folders locally created by qhub deploy.

Related QHub issues nebari-dev/nebari#1298 nebari-dev/nebari#606

A good example in other docs - how to write a how to

Add documentation page (md) titled Nebari architecture and conceptual guide

👉 notion page

Examples in other documentation sites

• https://kubernetes.io/docs/concepts/
• https://www.kubeflow.org/docs/started/architecture/

Good starting point may be shared/examples/kbatch_nb.ipynb on qhub-beta

• Good examples: https://docs.flyte.org/projects/cookbook/en/latest/integrations.html

notion doc

See also nebari-dev/nebari#1276 :

There are some recent issues related to Conda-store usage on qhub (see  
 https://github.com/Quansight/qhub/issues/1262 for example) that identify some issues with our [current doc of this service](https://docs.qhub.dev/en/latest/source/user_guide/environments.html?highlight=conda-store).  
 Also, there seems to be some outdated information presented in there as well.  

* Update main docs, to explain how to use conda-store, and differences between default/filesystems environments from the conda-store managed ones.
* Update paragraph related to authentication mechanic (we now support OAuth redirection)

Existing doc: https://docs.qhub.dev/en/latest/source/user_guide/environments.html?highlight=conda-store

Notion page

TODO

• add accessibility tests with pa11y
• include jest tests
• check broken links

We are lacking in-depth / up-to-date architectural diagrams

• #28
• #29
• #30
• #31

Diagrams should be added to https://lucid.app/documents#/documents?folder_id=278399173

Need to bring over the pre-commit GH Action https://github.com/Quansight/qhub/blob/main/.github/workflows/run-pre-commit.yaml

Related: #53

all the links in the readme need to be updated once the docs are complete

Given the number of services and technologies involved in a Nebari deployment, a glossary of terms would be a handy tool enabling all of us (users, devs, etc) to more precisely communicate.

Some other OS projects that have glossaries include:

• https://zero-to-jupyterhub.readthedocs.io/en/latest/resources/glossary.html
• https://kubernetes.io/docs/reference/glossary/?fundamental=true

References to the above might be a good place to start as well.

The current docs need to be expanded and migrated over to Nebari.

notion

Sometimes users want to have the docs/tutorials open on one side of their screen and another window open on the other side. In this case, collapsing the sidebar really helps improve the real estate.

Good examples: https://docs.flyte.org/projects/cookbook/en/latest/integrations.html

notion page

Summary

Accurate documentation detailing:

• How do groups and roles work. Is there a difference
• Explain special groups i.e. currently Admin/Developer/Analyst
• How do I add/remove people to QHub. (ideally, this should be doable by anyone in the admin group) not just with the root password.
• How are groups and shared folders connected.

Tasks to complete

• write documentation for the above points
• evaluate which parts of the current docs might be accurate and reused
  • login would need complete update/rewrite

How do I create a new environment on Nebari?

The conda-store UI is changing, so this may need to be on hold. Or should we go ahead with the current UI?

Maybe add a separate page for defining environments as part of the repo/IaC?

Include information on how to install something with pip (with warning signs, etc)

Good examples: https://docs.flyte.org/projects/cookbook/en/latest/integrations.html

notion doc

Need to notify users that we are in the middle of a rename

Summary

New nebari installation how-to should replace the existing installation guide, setup and provide enough information and instructions for the multiple cloud providers we support

Tasks to complete

• define which bits if any from the old docs are still applicable
  - [ ] @kcpevey to x-ref https://docs.qhub.dev/en/stable/source/installation/index.html with Diataxis items since most of this needs update/rewrite
  • setup section needs rewrite/update
  • usage whatever this is - needs a better/more accurate name
• write the installation documentation

Useful examples:

• https://www.kubeflow.org/docs/started/installing-kubeflow/
• https://zero-to-jupyterhub.readthedocs.io/en/latest/

• Need to discuss the look of the site and the tooling that will allow us to achieve what we want

TODO

• Gather requirements
• Identify tools
• Decide and design
• Implement

WIP from @kcpevey https://www.figma.com/file/7G2YapYjayeMyi2zdDcl5d/Nebari_mock_up_v1?node-id=0%3A1

How do I configure dask on nebari?

This is a doc that administrators will be reading.

• what do I need to include in the nebari config?
• what are the related infra parts?
  • Dask-gateway pod is like a manager for any job request that a user makes and it comes witht he nebari deployment (static) toghther with the dask-controller;
  • Dask-scheduler is created open request from a user, and its managed by dask-gateway
  • Dask worker is the actual place where the computation occur, for each job you can request # of workers that will run the job in parallel
• note that to use it, the env needs to include dask (or qhub-dask?)
• what is qhub-dask?

Summary

Right now, all the instance sizes are too small, they need adjusting based on:

1. Small/Large/High Mem/Cheap GPU etc
2. Is our default conda store pod size reasonable
3. Are our Dask workers configured correctly for our instance sizes
4. We may be able to use the ESIP deployment as an example.

Tasks to complete

• Determine new sizes needed
• Update the corresponding configuration/infra items
• Ensure dask workers are configured accordingly
• Increase conda store pod size

Related to

• nebari-dev/nebari#1111

See notion doc

Derives from #32

Some actions missing:

• in https://github.com/nebari-dev/nebari/blob/11fa8adc7fd5390b27abebe7371bcab2518efee0/docs/package.json#L17 need to remove the .md files to avoid conflicts with styles/syntax

Right now also eslint needs to be manually run (i.e. yarn run lint) so we need to actually run that lint as a pre commit with something like husky

Originally posted by @trallard in nebari-dev/nebari#32 (comment)

Need a doc showing how to create and share dashboards

Good examples: https://docs.flyte.org/projects/cookbook/en/latest/integrations.html

notion doc

Although there is some cleanup needed (on the intake catalog), it would be wonderful if we could integrate the intake catalog datasets into the tutorials. This way when the user tries Nebari for the first time, they can get started following our tutorials that show off a few services working together (read: data science workflows).

The idea of showing off more than one service working together need not involve intake but if you're starting with data, perhaps it would make it easier if we standardized the approach.

cc @tonyfast @sayantikabanik

The CONTRIBUTING.md doc needs to be updated.

Current documentation

Need to review the current documentation for accuracy and transform into a tutorial aimed at end users.

• Include information on how to install the dev package with pip (with warning signs, etc)
• add/remove/update packages

Good examples: https://docs.flyte.org/projects/cookbook/en/latest/integrations.html

notion doc

add nebari logo to readme once it is finalized.

Good starting point may be shared/examples/kbatch_nb.ipynb on qhub-beta