jupyter-server / jupyter-scheduler Goto Github PK

Problem

Users cannot run multiple notebooks at once.

Proposed Solution

In the file browser, let users select multiple notebooks (multiple files or directories), right-click, and choose a context menu option to run all notebooks as jobs.

Modify the create job form to accept multiple notebooks as inputs and to create multiple jobs on "Run Now".

Additional context

Should users be able to create multiple job definitions and schedules as a result?

Problem

We are currently in the process of adding the ability for users to run jobs automatically on a schedule. This issue is to describe and track work needed to enable scheduling on the create job form. The core user story here is this:

"As a user, I would like to create a job to run automatically on a schedule"

Proposed Solution

• Add UI to the create job view that allows a user to pick between "run now" and "schedule".
• When a user picks "schedule" offer two options:
  • A simplified, graphical UI for specifying a schedule.
  • Full CRON expression syntax for advanced usage cases.
• When a user create the job definition, take them to the list view, but with the job definitions table show.

Problem

Wanted to raise as a point of discussion if our REST APIs that accept parameters should take those as URL query string parameters or a JSON body. I am probably leaning towards a JSON body, which are we doing currently?

Problem

The job definition schedule is stored in cron format (e.g., */5 * * * *) which most users will not recognize in text.

Proposed Solution

In the job definition list and details view, display the job definition schedule in a human-readable way (e.g., */5 * * * * as "every 5 minutes").

Additional context

All supported JupyterLab languages should be supported; the user's chosen locale should be used for display.

cron explanation libraries exist for multiple languages.

• Python: https://github.com/Salamek/cron-descriptor
• JavaScript: https://github.com/bradymholt/cRonstrue

Summary

Implement these features for create_job API

• timeout_seconds timeout in seconds for the job to complete
• retry_on_timeout whether to retry if a timeout has occurred
• max_retries no of times to attempt retry
• min_retry_interval_millis delay in milliseconds between retry attempts

Context

#45 (comment)

This is an issue to track work to create a reusable Checkbox component for this extension that has the default JupyterLab styling.

Install via pip and then running jupyter server extension list shows a complaint about psutils not found

Installing psutils manually resolves the issue

We plan on having a job details view that contains more detailed information about a job. That view should have a breadcrumb of the form Jobs / [Job Name] or Job Definitions / [Job Def Name] that enables a user to navigate back to the list of jobs or job definitions. This is an issue to track our work to create those components.

Description

The idempotency token is not part of the schema for a job definition, but the "Idempotency token" continues to appear in the create form after the user switches modes to "Run on a schedule"

Expected behavior

After the user switches to "Run on a schedule," no longer show the idempotency token input. When the user switches back to "Run now," show the idempotency token input.

In addition to the model and modelChanged props, add two more:

1. errors — a mapping from input names to error strings.
2. setErrors — a function, in the useState style, to update the list of errors.

If any input has a "truthy" error value, display the MUI error treatment on the control and do not permit the user to submit the form.

Additional context

This is a follow-up to #24.

This is an issue to track the work to create a form label component. The create job form has a number of form components (text, checkbox, dropdown) that have a label component. This would be a simple component to help with reusability and consistency:

• Use a label element.
• Be styled using Jupyter's theme CSS properties and the overall style of this extension.

Problem

In our initial work on adding job scheduling, we probably won't get to making Job Definitions editable. This issue is to track that follow on work.

Proposed Solution

The list and detail view for a Job Definition should have an Edit action that allows the user to reconfigure elements of the scheduled job. See the REST API for which fields are editable:

#45

Description

JobDetail and buildTableRow have duplicate OutputFile element

Reproduce

1. Go to codebase
2. See OutputFile function in mainviews/job-detail.tsx and OutputFile function in components/job-row.tsx.
3. Note similarities

Expected behavior

We should be able to reuse this code instead of duplicating it

Context

• Operating System and version: N/A
• Browser and version: N/A
• Jupyter Server version: N/A

While our underlying data model for a schedule is a CRON expression, we should include a human readable form of the CRON expression in the list and detail views. Here is once such library that generates a human readable form of a CRON expression:

https://github.com/Salamek/cron-descriptor

Problem

Developers would like to call a different API when "Run Now" is clicked.

Proposed Solution

Let developers override the "Run Now" action to call a different API.

Additional context

Problem

Users don't have a way to pick and existing job definition and submit a notebook job that would run immediately.

Proposed Solution

Add a REST API that allows a job definition and runtime parameters as inputs, and enables a run now job overriding the passed in attributes in the REST API.

Additional context

#45 (comment)

Description

In Chrome, the error below the parameter name causes the parameter name to expand

Reproduce

Provide a value but not a name for a parameter. Add a new parameter.

Firefox: (no error)

Chrome: (error)

Expected behavior

The input box should not grow because of a validation error.

Description

When the user clicks "Delete Job" or "Delete Job Definition" from the details view, the job or job definition is deleted immediately, with no confirmation step. This is irreversible.

Expected behavior

The user should see a confirmation dialog (e.g., a modal) when they click a delete button so that they can confirm their decision to delete.

Problem

If the advanced options panel is collapsed, and at least one input in it has an error, the user might not know whey they cannot submit the form.

Proposed Solution

When the expandable section is collapsed and at least one error is contained in it, display an indicator on the section.

Alternatively, keep the section expanded and do not let the user collapse the section if at least one error is present.

Problem

Users may not know that they can create a job or job definition from the current notebook.

Proposed Solution

In the current notebook, add a toolbar button to open the create-job form to create a job or job definition from the current notebook. As is the case when the user right-clicks on a file in the file browser, the form should be pre-populated based on the current notebook.

Problem

Right now, we allow the user to edit the input file path in the create form. This introduces new error possibilities (user can enter a file that doesn't exist) and seems redundant at that point in the flow (they just picked a file).

Proposed Solution

I propose that we make the input file read only in the create form.

The job panel has a model that is used to render the state of the view:

https://github.com/jupyter-server/jupyter-scheduler/blob/main/src/model.ts#L142

We should serialize this model to JSON by implementing a toJSON method and wire up that state to the JupyterLab layout restoration extension. This will improve usability by fully restoring the state of this panel when JupyterLab is reloaded.

Looks like output_formats needs to be added to the StaticEnvironmentManager

Problem

Currently we are showing an error if the user does not enter text in the Input File text field. This is a good first step, but there might be more validation needed to prevent user from entering malformed input.

Proposed Solution

We can instead check for the existence of the file instead of "emptiness" for a better form of validation.

Description

If the input file for a notebook job has a relative path including a subdirectory, the "input" column shows only the subdirectory. The filename does not appear.

Reproduce

1. Create a notebook job where the input file is in a subdirectory: foo/bar.ipynb.
2. View the jobs list.
3. The directory foo, but not the filename bar.ipynb, appears in the "Input file" column.

Expected behavior

foo/bar.ipynb or bar.ipynb appears.

Context

Seen in a 2022-09-30 demo.

Problem

Our current errors object used for the create-job form uses a mapping from input name to error (as a string). For parameters, this is cumbersome, because parameters are ordered but the errors object is not.

Proposed Solution

Revise the errors object to be keyed on the possible members of the model, including an array for parameters.

Revise the removeParameter function passed to the parameters picker to modify the errors object's parameters.

Additional context

This is a follow-up to #65.

Problem

We are currently in the process of adding the ability for users to run jobs automatically on a schedule. This issue is to describe and track work needed to enable scheduling on the job list view. Users stories include:

"As a user, I would like to view and manage both jobs and job definitions"

Proposed Solution

• Add an MUI Tabs component to the list view, with two tabs, one for "Jobs" and one for "Job Definitions". The "Jobs" tab should have the existing job table.
• Add a "Job Definitions" table to the appropriate tab in the list view. Will need to work out the exact columns and actions needed for the table.
• Wire the tab to the listJobsView model property here (https://github.com/jupyter-server/jupyter-scheduler/blob/main/src/model.ts#L93)

Additional context

See issue #80 for parallel work on create view.

Problem

Users want to see whether a job definition is active or paused. Users want to pause or unpause a job definition.

Proposed Solution

• Add a "Paused" column to the Job Definitions List view. If a job definition is not "active," it is paused.
• Add a "pause" button to pause an active job or an "unpause" button to make a paused job active.

Additional context

See #81.

Description

Idempotency token value is not saved when creating job.

Reproduce

Create a job with idempotency token value.

Expected behavior

Idempotency value should appear in the Job Detail view.

Context

It seems like the idempotency_token value is not passed when the create job form is submitted.

I believe the create job form is missing some fields in the create REST API. Let's review that and make sure the UI is complete.

Problem

Currently, the job details appears below the selected item in the jobs list. This multi-column layout doesn't leave enough room to display a lot of information. In addition, the column headers do not line up with the details in the inline details view. Users can expand multiple jobs' details at once, cluttering the list view.

Proposed Solution

Add a details view that takes up the entire size of the list view's container (i.e., a main area widget). When the user clicks on a job name, they are brought to the details view. From the details view, a user can click a breadcrumb link to return to the list view.

Remove the details that appear below the row in the list.

Additional context

We can also move the action buttons out of the list view and into the details view.

We can provide for an extension point, allowing developers to customize the details displayed in the details view.

Only one job's details can be viewed at a time.

What's confusing is that model.jobsView = 'ListJobs' will render either the job list or the job definition list, dispatching based on model.listJobsModel.listJobsView.

@andrii-i For now, you'll want to somehow set the listJobsView to 'JobDefinitionDetail' immediately after line 29 above.

@jweill-aws We need to have a more thorough discussion on the purpose of these models. To my recollection, the whole point of models as a global state store was to allow persistence (serialization to JSON in local storage) and rehydration upon page reload. However, the list jobs view really doesn't have any state that needs to be persisted. Whether the jobs list or the job definitions list is being rendered should be stored in model.jobsView rather than inside model.listJobsModel.listJobsView, which means model.listJobsModel can be deleted entirely.

Another refactoring suggestion, rename JobsModel => SchedulerModel and model.jobsView => model.view.

Originally posted by @dlqqq in #92 (comment)

Problem

Some job platforms have logs that the user may want to view when the job is completed. Users will want a way to view the log files.

Proposed Solution

For now, we are going to keep it simple and have backends that support log files treat them as an output format. Those log files should automatically be copied into place with the rest of the outputs selected by the user.

There is some cleanup that we need to do to how we are handling the overall state of the extension:

• The notebook jobs panel should have a global state/model object that encodes the overall state of the panel.
• The JupyterLab layout restoration system should serialize that state to ensure the panel will be restored using that state.
• The JupyterLab command used to create the notebook jobs panel shouid take a JSON object that will override the saved state.
• The model should be wired up to a ReactWidget or VDomRenderer to automatically re-render the react component tree based on that state.

Information that will be needed in the model:

• The mode of the panel (create job, job list, job details).
• If mode is create job, the current state of the form.
• If mode is job list, should the job tab or the job definition tab show and which column is the table sorted by?
• If the mode is job details the job id of the job that is being shown.

Description

In rerunJob in job-detail.tsx, the output formats are passed as name: format, label: format where format is 'ipynb' or 'HTML':

Because these output formats do not match those in the environment's output formats list, they will never be selected on the create job form.

Expected behavior

The jobs should be in the same format as is used in the environments list's environment definition: for example, { name: 'ipynb', label: 'Notebook' }.

Context

The utility function outputFormatsForEnvironment is used to get an environment's properly formed output formats.

Problem

In certain cases, the environments dropdown in the create form will have a single value. In that case, the environment idea is likely to only confuse the user. We should simplify the UX in that case by auto-selecting the single value and hiding the dropdown and its label.

Problem

Users want to view job run errors for every job they run.

Proposed Solution

Make job run errors available in the API and make errors visible in the job details view.

I would like to suggest to extend the support of jupyter-scheduler to JupyterHub.

I feel this could be beneficial for several use cases. For example in our case we use JupyterHub to interact with smart manufacturing systems and prototype various codes in Jupyter Notebooks. Tu run on regular base some of them (e.g. to check availability of running services etc) would be definitively a use case for us.

The create job form has a parameters component that enables a user to add key/value pairs as parameters. This issue to to track improvement to that compoent:

• Use new TextField (#15) component.
• Use Box/Stack/Cluster (#13) for layout.
• Use new icon button component (#18).

This is an issue to track work to create a dropdown component for this extension. This component is used in the create form. For now, we can create a fairly simple version of a CSS-styled dropdown that uses JuptyerLab's theme variables.

Problem

From looking at the code, here's how I'm reading the idempotency token (IT) logic:

1. Customer creates job with IT=abc. Job is created with ID def.
2. Customer creates job with IT=abc. Job is not created and ID def is returned.

This is confusing because, without any warning or error message, the user is brought to the job list page where the IT is not displayed prominently.

Proposed Solution

Provide a warning message that is displayed on non-creation, as above, due to IT conflict.

In the UI, interpret this warning message and provide a link to view the job details for the job with IT=def.

Additional context

This is a follow up to #33.

Add ability to specify email notifications when submitting jobs.

In a recent PR, I added a basic Button component. This issue is to track work to enable this button to be rendered as a pure icon button or icon + text.

Problem

We are currently in the process of adding the ability for users to run jobs automatically on a schedule. This issue is to describe and track work needed to enable scheduling on the job detail view.

Proposed Solution

We should expand the current job detail view to be able to render both jobs and job details. Some key points of this work:

• Introduce a top level field into the job detail model that determines if it is a Job or Job Definition. This along with the id should be used to issue the describe call and render the appropriate detail view.
• Change the breadcrumb text and link to reflect if it is a Job or Job Definition.
• If it is a Job Definition:
  • Render the schedule in human readable form.
  • Change actions to be Delete and Pause.

Additional context

See #80 and #81 for parallel work on create and list views.

Problem

The create job API takes an output_prefix argument that determines in which directory the output files will be put when the job completes. Currently the TextField placeholder for this UI element reads "Output prefix". We should improve that to give the user more useful information with example. This information should include:

• What the output prefix is (where output files will go)
• What its format is (relative or absolute path)
• Default value
• Examples.

Problem

We want to show the most newly-created jobs at the top of the jobs list. Jobs have a start date and we sort the jobs list by default in descending order by start date.

The problem is that newly-created jobs are in state "STOPPED" with no start date, so they appear at the bottom (not at the top) of the jobs list by default.

Proposed Solution

Add two additional states for jobs to be in: "CREATED" and "QUEUED", both of which indicate that a job has not yet started.

Add a creation date/time for each job.

Sort the jobs list, by default, in descending order by creation date/time.

Additional context

Should we add the creation date/time to the jobs list so that users can sort by it?

Problem

Provide a UI extension point for output file links in the job list, so that a developer can override the appearance and behavior of output file links.

Proposed Solution

Offer a UI extension point and a component that reflects the default behavior.

Additional context

See #24 for another example of an extension point.

Description

The value of runtime_environment_parameters is not passed to the API endpoint on form submission.

Reproduce

Create a job using the advanced options extension point that modifies runtime_environment_parameters in the model.

Expected behavior

runtime_environment_parameters is present in the API call.

Description

While editing the name or value of a parameter, the cursor jumps to the end of the input box after every character entry. This is present in the code in #32.

Reproduce

Click in the middle of the name or value of a parameter. Type a character. The character is inserted at that position and the cursor moves to the end of the input.

Expected behavior

The cursor appears immediately following the inserted character.

Context

This does not occur with other text inputs in this form.

This appears to be common behavior with React when the state update is not fast enough. I attempted to remediate this using local state, as mentioned below, but I didn't have the same success with the parameter inputs as I did with the standalone inputs.

• facebook/react#955
• https://stackoverflow.com/questions/28922275/in-reactjs-why-does-setstate-behave-differently-when-called-synchronously/28922465#28922465

This is an issue to track work to create a TextField component for form input. We use this component in the create job form and need to encapsulate the visual styling and interaction behavior. Some rough guidelines:

• Visual style that uses JupyterLab's theme variables and overall visual design.
• Height that matches the needs of this extension, with possible height prop.
• Width that can be customized through a prop that takes ch units.
• Focus and hover states.
• Disabled state
• Placeholder text.
• Validation? (not sure we need this yet).