Git Product home page Git Product logo

pfra-hydromet's Introduction

Binder License


Description

pfra-hydromet is a collection of tools for developing pluvial (excess rainfall) and fluvial scenarios for input to hydraulic models.

Pluvial:

These tools (jupyter notebooks ) ingest data from the NOAA Hydrometeorological Design Studies Center (HDSC) and return unique, weighted excess rainfall events suitable for use in 2D hydraulic rain-on-grid models. This approach relies on:

  1. Meteorological data
  2. Random sampling
  3. Hydrologic transform
  4. Convolution algorithm for grouping

Fluvial:

These jupyter notebooks ingest HEC-SSP .rpt files containing flow frequency data for a specific USGS Stream Gage calculated at a range of confidence limits and return a series of events, statified by the annual exceedance probability, with discharge based on the mean flow frequency curve. This approach relies on:

  1. Bulletin 17C flow frequency analysis
  2. Mean flow frequency curve
  3. Stratified sampling

NOTE: EventsTable is the primary notebook for developing excess precipitation scenarios, and SSP_to_Mean_Curve is the primary notebook for calculating the mean flow frequency curve. Both of these notebooks can be called by Papermill, where papermill is designed to act as a manager to maintain consistency in computation, and ensure cells are executed in order. Manager notebooks are designated with the PM- prefix. Executed notebooks should be saved as documentation of the inputs, outputs, and results for a given project location.


Contents

notebooks:

pluvial:
  • PrecipTable: Retrieve NOAA Atlas 14 precipitation statisics for an Area of Interest (AOI).

  • PM-EventsTable: Manager notebook that executes EventsTable and/or reEventsTable.

  • EventsTable: Calculates excess rainfall using area-averaged NOAA Atlas 14 precipitation data, temporal distributions, the curve number (CN)* transform, and a convolution reduction algorithm (grouping). The output is a set of unique, weighted excess rainfall time series.

  • EventsTable_Stratified: Calculates a stratified sample of runoff events given rainfall and maximum potential retention distributions. For each each event and corresponding return interval, the event weight, runoff value, maximum potential retention value, and rainfall value are calculated.

  • reEventsTable: Calculates the reduced excess rainfall given a user-specified stormwater removal rate and capacity. Given user-specified contributing areas (stormsheds), the lateral inflow hydrograhs are also calculated for each event.

  • distalEventsTable: Calculates excess rainfall using updated randomized curve numbers and the original precipitation events calculated in EventsTable.ipynb. The events are combined using the groups determined during the convolution steps in EventsTable.ipynb. The reEventsTable notebook can be then be executed in order to calculate the reduced excess rainfall.

  • JSON_to_DSS: Converts the JSON files generated by PM-EventsTable and/or distalEventsTable to a single DSS.

  • MetadataExplorer: Explores the metadata file created by PM-EventsTable or distalEventsTable during the excess rainfall calculations.

  • Convolution_Parameters: Describes the test statistic and parameters used during the convolution step in the EventsTable notebook.

  • ProjectArea_ModelName_Pluvial_Parameters.xlsx : Excel Workbook used to store the CN, stormwater removal rate and capacity, and information on lateral inflow domains for each pluvial domain within a pluvial model. This Workbook is called by EventsTable, PM-EventsTable, distalEventsTable, and reEventsTable.

fluvial:
  • PM_Sampler_Ops: Manager notebook that executes SSP_to_Mean_Curve, Stratified_Sampler, and Make_Production_Run_List.

  • SSP_to_Mean_Curve: Calculates the mean flow frequency curve using Bulletin 17C confidence limits calculated in HEC-SSP.

  • Stratified_Sampler: Calculates the weight of a specified number of annual exceedance probabilities/recurrence intervals uniformly selected between the minimum and maximum value within log space.

  • Make_Production_Run_List: Calculates the discharge for each annual exceedance probability (AEP) within the weights table using the mean flow frequency curve.

DataRepository:

  • Temporal_Distributions: Folder containing csv files of temporal distributions of observed rainfall patterns broken down by volume, region, duration, and quartile NOAA Published. Note that the original data were compiled into csv's for uniform formatting.

  • Temporal_Distributions_Plots: Folder containing a Jupyter Notebook for each NOAA Atlas 14 volume with the plotted temporal distributions for each region, duration, and quartile.

  • NEH630_Table_10_1.json: A formatted copy of Table 10-1 from the National Engineering Handbook [Chapter 10].(https://www.wcc.nrcs.usda.gov/ftpref/wntsc/H&H/NEHhydrology/ch10.pdf.) which lists the CN values for dry and wet antecedent moisture conditions.

  • NOAA_Atlas_Volume_Codes.json: Metadata that maps the NOAA Atlas 14 volume number to the volume code. Source

  • NOAA_Temporal_Areas_US.geojson: geojson file containing the vector ploygons of the NOAA Atlas 14 temporal distribution areas. This file was constructed using the individual vector ploygons for each volume. Source

  • Temporal_Distribution_Data_Map.json: Metadata used to extract the temporal distribution data from the csv files saved within the Temporal_Distributions folder.

  • Temporal_Quartile_Ranks.xlsx: Excel Workbook that contains the percentage of precipitation events whose temporal distributions are represented by those in each quartile of a specific volume/region/duration. Source

*The (CN Method) is currently the only transform method in use for this project. Other transforms are available and can be adopted into the tool with minor modifications.


Documentation

Complete project documentation can be found in read the docs.

pfra-hydromet's People

Contributors

albrazeau avatar csmithud avatar mark-s-bartlett avatar shanemputnam avatar slawler avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

pfra-hydromet's Issues

Invalid value encountered during convolution

The following warning is produced during the first convolution step in EventsTable.ipynb:
image

This warning seems to only occur for the 6 hour duration events and only for some project areas. It does not appear to affect the results but should be investigated to identify the cause in case there is some underlying stability issue.

Unable to get PrecipData via FTP

  • Problem: In PrecipTable.ipynb, when data cannot be downloaded for a specific duration and recurrence interval, the final precipitation frequency table is incomplete. However, a complete table is needed to run EventsTable.ipynb.
  • Solution: Any file that cannot be retrieved should be added to a list and the code should attempt to redownload that file at the end of the for-loop.

Write Results from EventsTable_Stratified.ipynb

Currently, EventsTable_Statified.ipynb does not save any outputs. Add a method to calculate hydrographs and save to JSON files. @Mark-S-Bartlett recommends adding the nested hyetograph to describe the rainfall distribution and then running it through the SCS-CN equation.

Reduced Pluvial Calculation

The option to calculate the excess runoff for areas with stormwater infrastructure will be added in the branch: feature/reduced_pluvial.

Include Scope within hydromet_stratified.py

Add a scope/summary of the hydromet_stratified.py below the libraries and above the functions. This way the user can get an overview of the contents and understand the workflow before diving into the individual functions.

Add asserts to hydromet_stratified.py

It is good practice to add asserts/warningsto the code to make it more robust. For example, a warning could be added if the users enters a PMP that is unreasonably high or enters a return interval that is outside the bounds of the NOAA Atlas 14 precipitation frequency curve. @Mark-S-Bartlett recommends adding asserts/warnings for CN, PMP, and the mu parameter.

Add Examples to Docstrings in hydromet_stratified.py

For method-specific variables like S_Limit, include a short description and an example value or range of acceptable values in the docstring. Similarly, when passing an array of parameters to a function, include an example:
image. These types of descriptions make the code more readable and easier to identify issues.

Avoid Hard Coding in hydromet_stratified.py

In plot_runoff_distributions_final, the discharge range is hardcoded, ideally, you want this to be flexible for plotting in different areas, same in plot_max_potential_retentin_cond_runoff. Try to avoid hard coding when possible, can always pass the variable back to the user.

NOAA Atlas 14 Volume and Region vs Rasters

The polygons associated with the NOAA Atlas 14 volume and region do not perfectly overlap with the volume codes and their associated precipitation frequency rasters. For example, volume 8, region 4 and volume 9, region 1 have overlapping polygons but all of the se volume code raster is contained within volume 9, region 1 but does not cover the entire region. Part of the mw volume code extends into volume 9, region 1 but not across the entire portion associated with volume 8, region 4. For now, an error will be issued if the area of interest does not intersect the precipitation frequency raster which will instruct the user to manually identify the volume.

Read the Docs Update

Read the docs needs to be updated to include documentation from PR #37 on the fluvial analysis.

Unable to access NOAA ftp using binder

Testing notebooks on binder, urllib & ftplib methods timing out when attempting to access Atlas 14 datasets on file server at ftp://hdsc.nws.noaa.gov/pub/hdsc/data/.

@Tmodurr -Have you had similar issues with NOAA ftps running on remote servers?

PM-EventsTable.ipynb hangs while running the EventsTable.ipynb for the 6 hour duration

Some users have found that the PM-EventsTable.ipynb hangs or appears to not complete while running the EventsTable.ipynb for the 6 hour duration in some areas. This appears to only be a problem in areas that receive little rainfall and therefore have a small range in the precipitation frequency curve. This small range of precipitation will lead to a small range in the runoff amounts, which could cause a large number of events to be grouped into the same convolution bin, reducing the efficiency of the convolution procedure.

Coordinate Reference system Conversion (to_crs) Not working

For PrecipTable.ipynb, intersect_temporal_areas function, line: vol_gdf = vol_gdf.to_crs(geo_df.crs):

The conversion to a new crs (via to_crs) doesn't work for the first polygon in the dataframe being converted.

Picture2

If the original database was a shapefile, the problem is fixed by setting os.environ["SHAPE_RESTORE_SHX"]= 'YES'. Not sure how to fix this issue.

Thanks!

Over Constraining of GEV Optimization

When fitting the GEV to the NOAA Atlas 14 precipitation data and to the flow frequency curve in hydromet_stratified.py, for the bounds, could the loc/mu be greater than the average precipitation? It seems like this optimization may be over-constrained. @Mark-S-Bartlett recommends also revisiting.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.