What are the acceptable values for Location in the Undercut csv? In the 2019 data, almost all of them are either "Right_Bank" or "Left_Bank", which make sense. However, there are a few weird ones, such as:

• "Island": 2

• "Island,Right_Bank": 1

• "Right_Bank,Left_Bank": 3

Realizing that if a surveyor finds an undercut on an island, there should be some way to identify where it is. But then I think it should either just be "Island" for somewhere on an island, or "Island,Left_Bank" / "Island,Right_Bank".

Currently, the QA/QC function flags anything other than "Right_Bank" and "Left_Bank"

And cross-reference "key" variable names between Survey123 and Collector forms.

Beginning in 2019, the desire is that ocular substrate estimates are recorded for ALL channel units (both slow and fast), and in addition, pebble counts are taken in the first 10 fast water channel units.

1. Currently, the QC only checks for missing ocular substrate estimates in slow channel units and needs to be updated to reflect the above.
2. The DASH protocol needs to be updated to reflect this.
3. We should consider whether still missing ocular estimates can be imputed in the rollup_cu() function. These would be the remaining CU records where field staff forgot to estimate in the field AND we can't reliably estimate from drone imagery

@rcarmichael3 did, we believe, fill in all of the missing values he could in the 2018 data using drone imagery. Perhaps the same needs to be done to the 2019 and 2020 data?

Also, this brought up the idea of moving the master DASH document over to the repo to put in under versioning control. Can we move the .docx and .pdf versions over to the repo and delete from SharePoint? And we'll consider moving to a .Rmd file in the future.

The dash2018_rollup_v2.R script calculates sinuosity and braidedness based on 2 shapefiles, a centerline and a side channel one for each watershed. However, neither shapefile includes large side channels, with leaves the total length of any habitat reach with a large side channel smaller than it should be.

Sinuosity is only calculated using the main channel (centerline), but the wet_braid and wet_braid_sin metrics use the total length of the whole habitat reach, including side channels.

This oversight could also impact QRF capacity estimates, as the fish / m estimate will be multiplied by a total length that is too small, leading to total capacity estimates that are biased low for habitat reaches with large side channels. Currently the MRA_QRF repo uses the sum of all the channel unit lengths as the length for a habitat reach, and that does include large side channels.

@mackerman44 or @rcarmichael3, did I misread or misstate what's going on here?

The DASH package currently has the functionality to calculate discharge at channel units where those station width, depth, and velocity measurements are made, and those calculated discharge estimates are attached to those channel units. That is located in the calc_discharge() and rollup_cu_discharge() functions.

However, it seems we would want to "spread" those discharge estimates across all channel units e.g., the discharge est at a channel unit is applied to all downstream or upstream CUs (depending on how CUs were laid out) until you reach the TOS, BOS, or another channel where discharge is estimated. We need to determine the best way to do that and add a function that is likely applied in or right after the rollup_cu_discharge() function.

Low priority.

If I get a wild hair, consider replacing one_of() with any_of() or all_of(). First, need to determine if one_of() can universally be replaced with any_of(). What does all_of() replace?

Within the 2019 OTG DASH data, there's a miscellaneous Excel file within the Little Springs survey that appears to contain some small side channel data, presumably widths? At some point, need to figure out what to do with this...if anything.

S:/data/habitat/DASH/OTG/2019/lemhi/raw/Little Springs_Survey123_2019/Little Springs Small SC_2019.xlsx.

Perhaps wait to see if some of this information is missing during the QC/roll-up process.

This habitat reach has a large side channel with many missing metrics. Is it possible that only the main channel was surveyed on the ground? However, the large side channel has channel units with aquatic veg cover and total cover metrics (they are the only non-NA metrics recorded).

In addition, the length needs to be sorted out. I see 3 values in the dash2018_hr object:
Length.x: 588.4
Length.y: 143.4
Length.y: 141.4

Part of that issue is that the centerline file contains 2 centerlines for this reach (one follows the main channel, one follows the large side channel), but the side channel shapefile doesn't contain anything for this reach.

I recently came across some issues with the 2018 DASH data, including a duplicated habitat reach (#6 in the Pahsimeroi), and a realization that the script dash2018_rollup_v2.R contained an error that was not computing any side channel areas in the habitat rollup.

In the process of examining that, a few errors were found in the centerline shape files. Richie has fixed those and I'll be overwriting them on the NAS.

I will be working on that rollup script, and re-saving its output.

The dash_cu_points_1920.shp file contains collector points for Eighteenmile Creek, supposedly from 2020. However, I couldn't find centerlines, orthomosaics or OTG data for it. Should these points be deleted or ignored? Or is there missing data somewhere that needs to be brought into the NAS?

The first 34 channel units in the CU_1.csv file have the same parent global ID, which matches a global ID in surveyPoint_0.csv. After that the parent global IDs start increasing by 1 in the last few digits every channel unit. None of those have a match in the surveyPoint_0.csv file, so the channel unit rollup ends up with NAs in several columns for those channel units, including site name, survey date, etc.

@rcarmichael3, any thoughts about why this might happen? Should someone go into the CU_1.csv file and manually change all those parent global IDs to match the first 34?

Reminder to cross-reference the Survey123 data on the ESRI site (or SharePoint?) with what's on the NAS i.e., do we have all of the DASH OTG data on the NAS?

Also, should add instructions for creating, using, saving, etc. Survey123 survey templates for DASH in the DASH repo.

There are four channel units in the dash_cu_points_ shape files that have some points with one habitat unit listed, and other points with another. All points show the same channel unit type. The four are:

1. Little Springs 2019 CU 209 (Riffle): habitat reaches 30 (2 pts) and 31 (1 pt)
2. Lower Lemhi 3 2019 CU 3 (SSC): habitat reaches 1 (3 pts) and 2 (3 pts)
3. Upper Lemhi 1 2019 CU 69 (SSC): habitat reaches 11 (3 pts) and 12 (3 pts)
4. Lower Pahsimeroi MRA 2018 CU 9 (Riffle) habitat reaches 2 (1 pt) and 3 (2 pts)

At some point, somebody still needs to review each of the latest "qc_results_YYYYMMDD.csv" files in the "2019/lemhi", "2019/nf_salmon", and "2020/secesh" directories and resolve errors in the "/2_qcd_csvs/" data where possible. Further directions are provided in the "01_otg_import_qc.R" script and below:

# At this point, someone very familiar with the OTG data (preferably a field technician or field coordinator,
# secondarily a project leader) should likely intervene, review the remaining QC errors the we just wrote
# to file, and attempt to resolve those, and ideally, make notes for those QC errors that can't be resolved.
# In addition, for the QC errors that are resolved, it is useful to provide notes on how they are resolved. Notes
# on how errors were or were not resolved can be useful towards improving data validation (e.g., during field
# collections) or quality control steps in the future.

# Above, we imported data from the "/1_formatted_csvs/" directory for 3 year x watershed combinations ("2019/lemhi",
# "2019/nf_salmon", and "2020/secesh"), and in the case of "2019/lemhi" resolved some common issues in ocular
# estimates using the rescale_values() function and fish cover estimates using fix_fish_cover(), and then wrote out all
# of the remaining identified errors to a file in each respective directory titled "qc_results_YYYYMMDD.csv" which
# is the file that should be reviewed.

# The next suggested step is to copy/paste all of the Survey123 data to a new directory for each year x watershed
# combination; we used "/2_qcd_csvs/". And then, only data in the "/2_qcd_csvs/" directories should be modified while
# reviewing the QC results. Doing so preserved the integrity of the raw data in "/1_formatted_csvs/", which is good
# general practice. However, it is fine to add notes in the "qc_results_YYYYMMDD.csv" about how errors were or were
# not resolved (those qc results can always be replicated in the future using this script).

# Note: the data in the "/1_formatted_csvs/" directory and "/0_raw_csvs/" directories only differ in that file
# formatting issues may have been identified during data import (i.e., no data has changed).

# Now that errors have been resolved (which as of 20201209 they all have not, still need review), we can move on and
# re-import the "/2_qcd_csvs/" data in which some errors have been resolved.

Need to add a QC to qc_cu() to check that pebble values match expected. Should be able to get the expected values from Richie and/or the DASH protocol. Just a reminder.

We need to review and resolve the station widths in the "2020/secesh" surveys which includes data for Grouse, Lake, and Summit creeks. Any changes should occur in the "/2_qcd_csvs/" folder. From Kevin's e-mail:

I also noticed that a number of the discharge metrics for 2020 are 0, which seems odd. Looking at the discharge measurements, it looks like maybe the station widths weren’t captured correctly? For example, in the Grouse Discharge_Measurements_6.csv file, all the station widths are either 0.25 or 0.64. Seems odd, no?

Might also be worth adding a QC to make sure that station widths are unique i.e., station widths, it seems, should occur in consecutive increments.

At some point, I need to go back and ensure that NAs are being flagged during the QC process in any column that is later imputed during the rollup process. These should be the impute_cols = columns in rollup_cu_wood(), rollup_cu_jam(), and rollup_cu_undercut() (others?).

For tape distance, depth, and velocity.

Largely just requires lumping qc_disch_meas() into qc_disch()

Reminder to review and update the DASH protocol from lessons learned developing the pipeline.

In the README.txt file, under Getting Started, consider adding instructions on how to clone, build, and install DASH, especially for internal folks, and perhaps, also include brief instructions on making a pull request.

Make sure edit_date_2 is eliminated during otg_to_cu() after re-processing data when back in "office"

There's 2 NAs in jam data, both in Lower Lemhi

I created a function check_col_names() to compare a list of column names from a data frame being imported to a list of expected column names. However, I see there is a function janitor::compare_df_cols() to do a similar thing. Consider seeing if I can use compare_df_cols() instead.

A reminder to myself to, at some point, review and improve my error messages in the QC process. In many cases, I say the value is outside the expected range, but I should also just provide the value.

Some of the site_names in the OTG survey data don't match with the centerline Site_ID. Things like LowerLemhi3_2019 vs. Lowerlemhi3_2019, or Lower Lemhi_2, or even "Lemhi River". We can fix this by hand when matching the two datasets up, but it would be good to establish a better process, maybe pre-loading site names into Survey123 or something?

In the 2018 data, there are some habitat reaches (9 of them) that contain no pool channel units. Therefore, the metrics related to max pool depth are NAs. Is that appropriate, or should we set those to 0?

When used for QRF prediction of capacity, these could end up being imputed, or we'll need to flag them to not be

We need to create a function that calculated discharge and any location where those measurements were made.

Here is a description of Discharge (Q), which I grabbed, that I believe is from the CHaMP metric dictionary:

The sum of station discharge across all stations. Station discharge is calculated as depth x velocity x station increment for all stations except first and last. Station discharge for the first and last station is 0.5 x station width x depth x velocity.

There is a file within the Little Springs 2019 DASH survey data that appears to contain width measurements for small side channels there:

S:/data/habitat/DASH/OTG/2019/lemhi/0_raw_csvs/Little Springs Small SC_2019.xlsx.

Need to "resolve" that file. Perhaps that data just needs to be entered into the CU data in the 2_qcd_csvs?

Looking into it now...

I need to standardize how pebble counts are recorded btw 2018-2020 & 2021 data.

TOS/BOS have not been assigned in the channel unit data for any data 2018-2020.

I need to add a step a QC to verify that the "site_name" field in the otg_type = "surveyPoint_0.csv" is filled out. Just identified that the site name for the LowerLemhi3_2019 field is not filled.

In the 2018 data, there are 2 small side channels with an average width of 0. They are:
UpperLemhi_2018, hab_rch 3, cu_id 012
UpperSalmon_2018, hab_rch 10, cu_id 062

This seems incorrect.

In the 2018 data:

There are a number of habitat reaches (3 of them) that contain no riffle channel units, so no pebble counts were conducted and the D50 and D84 metrics are NA.

Similarly, there are 6 habitat reaches with no pools or runs, so it appears no percent fines/gravel/cobble/boulders was recorded in those reaches, and the percent substrate metrics are NA.

That is accurate, but if a QRF model uses one of those metrics, we'll end up imputing it. Are we OK with that?

@rcarmichael3 is going to provide the data entry templates for both Survey123 (used to collect the OTG data) and Collector and/or information on how to log into our account and access that info so that we can change variable names or improve templates, if needed.

Also, might be worthwhile to include those templates in the repo, maybe inst/templates/ so that they are included during package install.

Reminder to start a metric dictionary for DASH metrics, likely to include metrics that are collected OTG and metrics that are summarized both to the channel unit (CU) and habitat reach (HR) scale. Something similar to the CHaMP metric dictionary, but better!

I just added the rescale_values() function as a simplified version of rescale_percents() that simply:

1. sums the designated col_names to calculate a new column sum_values
2. identifies records where sum_values does not fall within an expected min_value and max_value
3. rescales those values to be equal to sum_to (default = 100)

And returns a corrected data frame. My intent is then to use those within functions to remedy errors in the ocular estimates and fish cover data. For now, I left rescale_percents(), but leaving this issue to remind myself to eventually remove rescale_percents() after we're certain all its functionality is used in the functions to fix that data.

Just a reminder to organize the OTG, collector, and survey template forms on ESRI online.

The sum of the various cover metrics can be more than 100%. However, I would suggest changing the qc_cu function to flag the following issues related to cover (instead of the current "does it sum to 100"?):

• Are some (but not all) of the cover metrics for a particular Global ID NA? These could be made 0, before moving on.

• Are any of the cover measurements in decimals, rather than multiples of 10? If they are all less than 1, this could be fixed before moving on.

• For multiples of 10, if we add all the cover metrics except "Total No Cover", it should be less than (100 - Total No Cover). If it's not, flag it.

• Is "Total No Cover" more than 100?

• If we add all the cover metrics including "Total No Cover", it should be at least 100. Flag any that are smaller

According to @rcarmichael3 Ricardo, "station width refers to the distance between each depth and velocity measurement, so they should be the same across each measurement. So if all of them are 0.25 or 0.64 that would make sense to me. At least that is how it is supposed to be recorded."

This was in response to @KevinSee's earlier remarks to me "I also noticed that a number of the discharge metrics for 2020 are 0, which seems odd. Looking at the discharge measurements, it looks like maybe the station widths weren’t captured correctly? For example, in the Grouse Discharge_Measurements_6.csv file, all the station widths are either 0.25 or 0.64. Seems odd, no?"

So it seems we need to review how measurements are collected and discharge is calculated. 1) It appears that station width's may have been collected incorrectly in 2018 & 2019, at least after first glance. However, 2) this doesn't necessarily explain Kevin's remark above where a number of the discharge metrics for 2020 are 0.

In any case, worth a review.

In the 2018 data, Richie noticed a missing QRF capacity prediction in the MRA_QRF repo for, we believe, Pahsimeroi Hab_Roll 9 which Richie says is channel units 73-77. These channel units have data, but Hab_Roll 9 doesn't appear in the centerline shapefile used to calculate sinuosity and braidedness metrics. Why not?

Here's a case for SCOOBY DOO (or Mike or Kevin)!

Currently, datetime imports occurring in read_otg_csv() and with column specs provided in get_otg_col_specs() are a little problematic. Some are being read in with an AM/PM designation and in 12-hr format whereas others are being read in as 24-hr format. I initially tried to remedy using readr::col_datetime(format = "%m/%d/%Y %H:%M"), but to no avail.

For the moment, I'm simply reading in date time columns (e.g., Survey Date, EditDate, CreationDate) as readr::col_character(), which is maybe okay? i.e., is it up to DASH to determine datetime specifications? Or do we need to be better about date formatting in Survey123?

Related to Issue #15

We need to determine the best method for dealing with NAs in the length_m, diameter_m, width_m, height_m, and estimated_number_of_pieces columns in the otg_type = "Wood_2.csv" and otg_type = "Jam_3.csv" data. These are individual LWD and jam measurements that are missed in the field (or less likely, fall outside of expected QC values and we don't know how to resolve).

Currently, within rollup_cu_wood() and rollup_cu_jam, I use the median among all other measured values within a channel unit to fill in the NA. However,

1. This may not be the most appropriate solution, and
2. This fails when no other pieces of wood or jams were measured within the channel unit.

For now, I'm proceeding w/ the above, temporary solution, but we should identify better and then add that to the above functions for imputing missing values during the rollup step.

We need to determine how to deal w/ NAs in the wet, channel_forming, and ballasted columns of the wood data. These are records where the field tech forgot or had an error when recording that information. We currently aren't using that information "downstream", but it could be useful at some point...

One idea is to calculate the proportion Yes/No for each column within a given channel unit OR survey (which is important), and then draw from those proportions to determine whether a given NA should be Yes/No (e.g., if 60% of all wood pieces in a survey are wet, draw from that to determine Yes/No with a 60% chance of being "Yes"). Stated that very inelegantly.

Station widths in the "DischargeMeasurements_6.csv" files and /2_qcd_csvs/ folders are incorrect in 2019 data. We need to identify the root of the problem and resolve in the 01_otg_import_qc.R script and using values from the /1_formatted_csvs/ folders.

When pulling together the updated centerline files, I found 6 instances of duplicated channel unit numbers. They are:

• Big Timber 1, 2019, CU 11
• Grouse, 2020, CU 41
• Hawley, 2019, CU 176
• Kenney 2, 2019, CU 34
• Kenney 2, 2019, CU 98
• Upper Lemhi 3, 2019, CU 162

Some of these appear to be adjacent segments, but some of these appear to be distinct channel units (and the Big Timber 1 appears to have a segment far, far away from the rest of the centerline). These centerline files are meant to be all cleaned up before we start joining other data to them, so these need to be addressed.

There are 7 pools in 2018 data with a max depth of 0. This seems incorrect.