Test data for osprey processes are scattered in tests directory. Assemble and organize them under tests/data.

The parameter module is responsible for the "development of impulse response functions using input datasets such as a flow direction grid, outlet locations, etc." Wrap this into a process that can handle either a file or dict object as input. Follow the script for other inputs that may be needed.

Things we need to add:

• The process
• A pytest suite
• A notebook demo (can be an individual demo for now)

Ensure that you:

• Test both locally and on the docker server
• Make note of any changes required in RVIC in this thread

By default the project uses conda as its package management tool. We want to migrate away from this and rely solely on pip. We want to:

• Remove environment.yml
• Update requirements.txt and requirements_dev.txt as needed
• Adjust Dockerfile to use pip

rvic/parameters.py calls copy_inputs to copy the inputs in the config file to the CASEDIR. Because of this, URLs cannot be used in the config file. This call should either be removed from the package, used only when URLs are not used, or wps_parameters should be updated to copy these files from THREDDS to local storage to have detectable file paths by this function.

Currently, the file paths inside sample_parameter_config.cfg are relative to the osprey root directory. This makes it so that running the test only succeeds if run from this root directory. In order to allow it so that the test can be run from anywhere, a temporary config file should be written at run time to contain the absolute paths using os.path.abspath and resource_filename and to be passed into wps_parameters.

Describe the bug
After the multi-stage build strategy is applied, the test data can no longer be found from the Docker image, and it revealed an issue that hasn't been identified. It seems like the uhs file inputs used for wps_convert are not uploaded to the Docker container in the Docker server, thus the following messages are produced.

Traceback (most recent call last):
  File "/root/.local/lib/python3.8/site-packages/pywps/app/Process.py", line 249, in _run_process
    self.handler(wps_request, wps_response)  # the user must update the wps_response.
  File "/root/.local/lib/python3.8/site-packages/osprey/processes/wps_convert.py", line 128, in _handler
    convert(config_file)
  File "/root/.local/lib/python3.8/site-packages/rvic/convert.py", line 26, in convert
    dom_data, new_dom_data, outlets, config_dict, directories = uhs2param_init(
  File "/root/.local/lib/python3.8/site-packages/rvic/convert.py", line 98, in uhs2param_init
    outlets = read_station_file(
  File "/root/.local/lib/python3.8/site-packages/rvic/core/convert.py", line 52, in read_station_file
    raise ValueError("missing uhs_file: (%s or %s)", uhs_file, uhs2_file)
ValueError: ('missing uhs_file: (%s or %s)', './tests/data/samples/sample.uh_s2', '/tmp/pywps_process_g6cj6wty/station.uh_s2')
2020-12-22 23:21:49 ERROR: osprey: Process error: method=wps_convert.py._handler, line=128, msg=('missing uhs_file: (%s or %s)', './tests/data/samples/sample.uh_s2', '/tmp/pywps_process_g6cj6wty/station.uh_s2')

To Reproduce
Steps to reproduce the behavior:

1. Run test_notebooks_custom with a container up'd using an image built after #73
2. Check the log messages in the docker-dev03 server.

Expected behavior
Should pass the demo without failure.

sandpiper adapted to a new demo strategy, which enables us to test the demo-run results in Makefile. Deploy the same strategy in osprey.

Currently, tests that are failing due to RVIC's internal issues UW-Hydro/RVIC#96 UW-Hydro/RVIC#130. Un-comment these once RVIC's upcoming version 1.1.1 is released and the issues are resolved.

RVIC 1.1.0.post1 runs based only on config files, which degrades the user-friendliness of the software. Therefore, a config_builder program, which takes user inputs to create config files, is needed for each process.

For RVIC convolution, refer to the document page to see what contents should be included in the config file. @corviday do you have some advice on setting the direction to implement this program such as setting default values?

thunderbird Docker container is experiencing the same logging issue sandpiper was having. The solution is described here.

In order to hook osprey into birdhouse-deploy we need to put together a pavics-component. This will be similar to the component added to thunderbird

Is your feature request related to a problem? Please describe.
Detailed process errors are currently only shown on the server log. Some more detailed errors should be shown to the user.

Describe the solution you'd like
Add validation checks and ProcessErrors where possible to help show more desciptive user errors.

Describe alternatives you've considered
None

Describe the bug
RecursionError occurs whenever an osprey process is run after a failed attempt OR running multiple processes at the same time, and it causes failures in succeeding processes as well.

RecursionError: maximum recursion depth exceeded while calling a Python object

To Reproduce
Steps to reproduce the behavior:

1. Run jupyter lab
2. Run processes with invalid inputs
3. See error logs
4. Follow the above procedure in a fresh Docker container but with running multiple processes at the same time

Expected behavior
Enable running a process after a failed attempt.

Current form of the dockerfile does not support output collection for the bird processes.

Title.

We have built a utility tool for birds called wps-tools. Add this package to the project and factor out code where possible.

The Dockerfile is not working as intended with birdhouse-deploy. It seems the ports are mixed up somehow and the app isn't working correctly.

rvic-daccs 1.1.4 is released. Target the version number in requirements.txt and adapt the changes in osprey processes.

Is your feature request related to a problem? Please describe.
Not related to a problem but we should have to notebooks optionally target localhost to test dev instances without having to change any code.

Describe the solution you'd like
Use env vars to determine which instance of osprey we are targeting.

Describe alternatives you've considered
None at the moment.

• Remove multi os actions
• Upgrade to 20.04
• Remove PR trigger

Update osprey description to include an interesting blurb about the bird.

Due to the couple issues /UW-Hydro/RVIC#96 and /UW-Hydro/RVIC#130 in the latest RVIC 1.1.0.post1. Some tests that run RVIC's Parameters and Convolution modules will not pass pytest. It is suggested to ignore those tests before RVIC's internal issues are solved.

Description

wps-tools 0.3.0 is released. Use the newly added functionalities.

Requested Action

• Replace osprey's collect_args function to the wps-tools version
• Define osprey's status_percentage_steps using common_status_percentages dictionary
• Use local_path and url_path functions for constructing test parameters

Alternatives

None

Is your feature request related to a problem? Please describe.
Error handling in RVIC processes is not a safe way to terminate the software as described in #57.

Describe the solution you'd like
We would like to have an input validation method that prevents running RVIC processes when invalid inputs are given.

Describe alternatives you've considered
None

Is your feature request related to a problem? Please describe.
Local files are not accessible from Docker containers.

Describe the solution you'd like
Add ComplexInputs for each of the required files in the configurations

Describe alternatives you've considered
Allow users to upload local files to THREDDS server and use the OPeNDAP URLs for the configruations.

Cookiecutter has seen some updates since we created the bird. Update the repo using the cruft tool.

The makefile has fallen out of date.

RVIC 1.1.1 is now available at the Pypi server. Update osprey to use the new version of RVIC.

Since 'RVIC convolution' is followed by 'parameters process most of the time except for the situations using pre-generated parameters output repeatedly, I think it is worth to consider creating a single process that runs both modules. Without this, users would have to follow 6 steps to run RVIC:

1. Configure parameters input.
2. Run wps_parameters
3. Download the parameters output.
4. Configure Convolution input.
5. Run wps_convolution.
6. Download the convolution output.

We could reduce it into 3 steps of configure - run - download.
@corviday @nikola-rados @eyvorchuk What do you think? Is it worth to implement this additional process?

Describe the bug
osprey cannot access local input configuration files since they are passed into the processes as a literal input.

To Reproduce
Steps to reproduce the behavior:

1. Jupyter lab demo for any process
2. Change the relative configuration file path to an absolute file path using pkg_resources.resource_filename
3. Run the process

Expected behavior
Replace the literal inputs with complex inputs.

The convolution module is responsible for "generating streamflows by convolving the impulse response function developed in the paramaters step with runoff fluxes from a land model (e.g. VIC)." Wrap this into a process that can handle either a file or dict object as input. Follow the script for other inputs that may be needed.

Things we need to add:

• The process
• A pytest suite
• A notebook demo (can be an individual demo for now)

Ensure that you:

• Test both locally and on the docker server
• Make note of any changes required in RVIC in this thread

Is your feature request related to a problem? Please describe.
The multi-stage build is a strategy to reduce image sizes and build time. Apply this strategy in the Dockerfile

Describe the solution you'd like
Separate a single stage into two stages, build and execution, by having multiple FROM statements.

Describe alternatives you've considered
None

All data that was created for bird-specific testing has been moved on the thredds server. Urls in tests and demos need to be updated to reflect the new file paths in storage.

Is your feature request related to a problem? Please describe.
When testing notebooks we want the url to be flexible without having to modify them manually.

Describe the solution you'd like
Makefile adjustments + wps-tools version bump

Describe alternatives you've considered
None.

Some of the demo and test files are outdated. Update them to adapt to the changes.

Description

Update the osprey processes and demos using features from wps-tools 1.0.1

Requested Action

Use new module format, and output handling functions from wps-tools 1.0.1

Alternatives

None

rvic-daccs package is forked from the original RVIC software. Internal issues in the original software are resolved in the package. Replace rvic==1.1.1 dependency to rvic-daccs==1.1.2.

Describe the bug
The docker publishing action is not being triggered when creating a pull request.

To Reproduce
Steps to reproduce the behavior:

1. Create a pull request from an activate branch

Expected behavior
An image should be built and published with the pr-[#]-merge tag.

The convert module is responsible for "providing users with the ability to convert old routing model setups into RVIC parameters." Wrap this into a process that can handle either a file or dict object as input. Follow the script for other inputs that may be needed.

Things we need to add:

• The process
• A pytest suite
• A notebook demo (can be an individual demo for now)

Ensure that you:

• Test both locally and on the docker server
• Make note of any changes required in RVIC in this thread

Taking issue #2 into consideration, there are some documentation changes that need to take place. Read through the docs/ directory and make any necessary changes. Furthermore, we should update AUTHORS.rst and any other top-level rst that need work.

Add github actions as our main CI/CD tooling component.

We need to figure out how to break down the RVIC tool into usable wps processes. Explore the code and try to come to up with a few pathways for us to follow. Once we have sorted some of our ideas out we can create new issues and begin work.

Please post any notes/thoughts/discussion in this issue thread.

RVIC 1.1.0.post1 runs based only on config files, which degrades the user-friendliness of the software. Therefore, a config_builder program, which takes user inputs to create config files, is needed for each process.

For RVIC parameters, refer to the document page to see what contents should be included in the config file.

Using wps_convolution.py as a template, update wps_parameters.py to use a dictionary instead of configuration file for inputs.

osprey's last merge did not pass CI's pytest. Investigate the issue and resolve it.

replace_urls function in utils.py is modifying the original configuration files. Fix the function to make a copy of original files in tmp directory and modify the copied file.

rvic-daccs 1.1.3 is released. Target the version number in requirements.txt and adapt the changes in osprey processes.

Is your feature request related to a problem? Please describe.
Processes in Osprey currently handle one file input. It may be time efficient for the user, if the processes could handle multiple inputs.

Describe the solution you'd like
Processes handle/ allow a list of files as input.

Describe alternatives you've considered
Users could iteratively run files through the process themselves.