For better readability, maintenance and collaboration, I'd suggest to split bioRad.R (and the other R files) in logical files: one per function (and its helper functions), with the same name as that function.

@adokter these are all the functions I found, per current file. It's quite the list 😅 At first glance, think it would be better to group these by what they do (e.g. bind, vintegrate) that their data type (vp, vplist).

bioRad.R

bioRad.plot.R

bioRad.ppi.R

download_vp.R

When using the readvp.table function on a vol2bird text output, I got the following error:

Warning message: In Ops.factor(heights[-1], heights[-length(heights)]) : '-' not meaningful for factors

This heights seems to be a factor, the problem can be solved by putting heights = as.numeric(as.character(heights)) under heights = data[[1]]$HGHTin the function.

scan class needs:

• radar
• datetime (nominal time)
• wavelength

param class needs all essential attributes needed for merging parameters into a scan object
We need at least:

• NI
• startdate,endate,starttime,endtime

The bird density loaded from vol2bird has bird density zero when sd_vvp is not defined / has nodata value. If I recalculate densities within bioRad (when sd_vvp threshold is changed), the bird densities are not set to zero.

Related to adokter/vol2bird#86

Need to make this consistent

It is only used in the vertical integrated bird density plot

composite now requires specification of one scan parameter by argument param. Change to default to generate composite for all scan parameters when param left unspecified.

For example is.vpts and dim.vpts are missing, while is.ppi and dim.ppi do show up correctly.

For example add

FALSE %in% file.exists(fs)

While listing the functions in #50, I noticed there are a lot of name conventions used (lowerCamelCase, period.separated , underscore_separated ) for those functions. Not sure if we want to tackle this now (but better now than later), but the it would be good to stick to 1. 😄 The link suggests underscore_separated.

Currently the ppi's and scans don't have a time stamp or radar ID. Suggestion to have this automatically added from the info stored in the pvols.

rhdf5 is not available in CRAN, while h5 is

Reading in a pvol from Baltrad (so real life data) gives the error that dataset1 does not contain 'how', which is true.

filename <- read.pvol("bewid_pvol_20170110T0900Z.h5")
Error in h5readAttributes(filename, paste(scan, "/how", sep = "")) : Object dataset1/how does not exist in this HDF5 file.

Possible that the read.pvol function needs to be adapted to the current structure of the ODIM h5's?

I can read vol2bird output vp.h5 files, though cannot convert them to a time series. I get following error:
Error in radars == radar : comparison of these types is not implemented

When looking what is inside the vplist, I get the following:
List of vertical profiles (class vplist)
radars: Error in cat(list(...), file, sep, fill, labels, append) : argument 2 (type 'list') cannot be handled by 'cat'

Looking in a single vp gives me this:
Vertical profile (class vp)
radar:
source: WMO:11480,NOD:czbrd
nominal time: 2017-01-07 23:30:00
generated by: vol2bird 0.3.6

So it seems to me that the readvp function does not read in the radar ID properly, making that no time series can be made eventually?

README.md:
install note 1:
add 'reboot R' after
install_github("hadley/ggplot2")
install_github("dkahle/ggmap")

COMMANDS:
cmt:
twice "of" in first sentence: ... cumulative sum of OF the..
missing "of" in last sentence: ... as a function OF time ...

is.scan (and other commands with 'geo' element in de description)
it's should be its:
... geo element of it's parent ...

[.vplist and [.vpts
are these commands?

It's now shown under adriaandokter.com my personal webpage, not ideal...

currently you get a warning message that this is not possible

e.g. vplist class has no plot method, but users may think there is

Suggest to change the application of the gap criterium. The gap detection in the birds-only loop (iProfile=1) is used, while it makes more sense to use the gap detection of the all-scatterers loop (iProfile=2).

In mixed birds/precipitation events, you can then have a valid bird density, but not a valid bird speed.

Could also leave the gap criterium like this (maybe more conservative), but then we should set bird density to UNDETECT instead of zero, i.e.

if (hasGap && birdDensity>0 && alldata->misc.scatterersAreNotBirds[iLayer] == FALSE){
       alldata->profiles.profile[iLayer*alldata->profiles.nColsProfile + 12] = NODATA;
}

Requires a fix of issue adokter/vol2bird#97

the radial velocity standard deviation threshold is only applied to the bird density quantity, not to the reflectivity and reflectivity factor.

This is confusing, as people may extract these quantities for further analysis.

Possible solution: book-keeping of eta both with and without sd_vvp applied, which adds an extra quantities. Using wavelength information, dbz could be recalculated from eta

Another option is to only adjust the plotting functions, but that has a higher risk of confusion

This gives an error:

vintegrate(VPTS,alt.min=0,alt.max=200)

for larger altitudinal ranges no error:

vintegrate(VPTS,alt.min=0,alt.max=400)

Seems to be related to lapply:

vpFromVptsList=function(vptslist,date,dt=60,trim=T){
  vps=lapply(vptslist,vpFromVpts,date=date,trim=trim)
  idx.dt=sapply(vps,function(x) abs(as.double(x$datetime-date,unit="mins"))<dt)
  vps[idx.dt]
}

This is because /Library isn't one of the directories that is mountable by default. So either

• ask users to add this folder to the File Sharing tab in the Docker settings.
• have the package copy files in extdata folder to working directory in examples

for vplist class:

radarpos.vplist=function(x){
  stopifnot(inherits(x,"vplist"))
  lat=sapply(vpl, function(x) x$attributes$where$lat)
  lon=sapply(vpl, function(x) x$attributes$where$lon)
  radar=sapply(vpl, function(x) x$radar)  
  data.frame(radar=radar,lat=lat,lon=lon)
}

The function read.pvol allows parameters param = c("DBZH", "VRADH", "RHOHV", "ZDR", "PHIDP", "CELL"), however in the OPERA polar volumes, velocity is stored under the name VRAD. As a consequence, VRAD-scans are not read by read.pvol. Simply adding "VRAD" to the parameter list solves the problem. See also issue #39 for another problem with the same function.

in read.pvol function, the param='all' option is not implented, although it is listed in the manual

I think it would be useful to add http://adokter.github.io/bioRad as the repository website, like here:

(For now that URL will redirect to adriaandokter.com, see #54)

I was just wandering if we could make readvp.list and/or readvp faster.

Replacing

profile=as.data.frame(lapply(groups,function(x) readOdimProfileData(filename,x)))

with

x=h5read(filename,'/')
profile = data.frame(matrix(unlist(x$dataset1),nrow=length(x$dataset1$data1$data)))

can reduce 1/3 of the running time with my quick test. But I'm not sure if it's really valid for all the radar file. Also, do we really need to read all the attribute when using readvp.list ?

vp objects have a datetime slot (vp$datetime), while vpts has a dates slot (vpts$dates).

confusing to have different names for these

plot.colours is twice specified, causing a recursive-problem.

It's first specified in line 75, and then again in line 177, causing trouble when you want to execute lines 182 and 183.

Changing the name of plot.colours in line 75 and accordingly in line 177 solves the problem.

1. pvol exploration: structure of pvol
2. ppi: ppi plot
3. vol2bird: pvol → vp, docker
4. vp exploration: structure of vp
5. vpts exploration: vp → vpts, timeseries plots
6. vpts filtering: df + yaml
7. vpts aggregation

Suggested improvement by Baptiste Schmidt:

If processing a non-existing or broken file, vol2bird function now gives a hard exit, terminating all processes. Instead it would be better to have a warning message, as then it won't terminate for-loops in which the vol2bird function is called.

Aim:

Add the functionality of vp-processing in selecting data and outputting a simple data.frame for multiple radars, that you can easily write to a text file.

Several steps in vp-processing's extract_vp.Rmd have already been integrated in bioRad, with the advantage that people have the benefits of the vp-processing examples, while still making use of the 'bioRad' objects and the their class functions.

New functionality (to be) added:

• function to go from vplist class object, to a list of vpts objects, implemented in the bind() function. This allows users to organize a list of single profiles into a collection of vertical profile time series organized by radar
• A function that converts a vertical profile time series object (class 'vpts') to a simple 'data.frame' that people can use for own analyses. A as.data.frame class function as been added for for vpts objects (as.data.frame.vpts)
• have as.data.frame optionally add day/night binary indicator
• hvae as.data.frame optionally add sunrise/sunset time
• take a yaml configuration file and filter the data.frame generated by as.data.frame according to the specified selections.

while installing bioRad in R I recieved the following error:

> install_github("adokter/bioRad")
Downloading GitHub repo adokter/bioRad@master
from URL https://api.github.com/repos/adokter/bioRad/zipball/master
Installing bioRad
"C:/PROGRA~1/R/R-33~1.0/bin/x64/R" --no-site-file --no-environ --no-save --no-restore --quiet CMD INSTALL  \
  "C:/Users/jshamou1/AppData/Local/Temp/RtmpsjV7bM/devtools17344a633b4/adokter-bioRad-df64641" --library="C:/Program Files/R/R-3.3.0/library" --install-tests 

* installing *source* package 'bioRad' ...
** R
** data
*** moving datasets to lazyload DB
** inst
** preparing package for lazy loading
Error in loadNamespace(i, c(lib.loc, .libPaths()), versionCheck = vI[[i]]) : 
  there is no package called 'spam'
ERROR: lazy loading failed for package 'bioRad'
* removing 'C:/Program Files/R/R-3.3.0/library/bioRad'
Error: Command failed (1)

I tried rerunning this after installing Docker but still recieved the same error

Reported by Baptiste Schmid on 7 Apr 2017:

plot(ts, quantity="dens")
#>>>  Error in if (is.na(x) || is.na(y)) next : missing value where TRUE/FALSE needed

It looks like quite some countries (NL, BE, FR for example) do not have a subfolder how under dataset1/2/3/.... However, the function read.pvol() requires this and trying to read in pvols that do not have the how subfolder makes the function crash with the error: Error in h5readAttributes(filename, paste(scan, "/how", sep = "")) : Object dataset1/how does not exist in this HDF5 file.

One way to solve this would be to make the how attribute optional in the function read.pvol().

suggested function attached to download NEXRAD 2 .gz files from AWS

NEXRAD_L2_massdownload.txt

##SET STATION ID - four letter character string
STATION <- "KOAX"

##vector of dates as a string in the YYMMDD format (
#DATES <- c("20150708","20150715", "20150722", "20150730", "20150905", "20150912","20150919", "20150927")
DATES <- c("20150722", "20150905")
##Start and end hours of interest for each day (must be same for all days) as integer 0-23
startHOUR <- 0
endHOUR <- 14

##folder to save radard.gz files to
saveLoc <- "./Bioscatter_data"

AWSbucket = "https://noaa-nexrad-level2.s3.amazonaws.com/"

system.time(
AWS_objects <- get_NEXRAD2(STATION,DATES,saveLoc,startHOUR,endHOUR)
)

# Function to download NEXRAD2 files from AWS -----------------------------
##version: 0.1.1
##Author: Michael Whitby
##EMAIL: [email protected]
##July 28, 2017 


get_NEXRAD2 <- function(STATION,##SET STATION ID - four letter character string
                        DATES,##vector of dates as a string in the YYMMDD format
                        saveLoc=getwd(),
                        startHOUR = 00,##Start and end hours of interest for each day (must be same for all days) as integer 0-23
                        endHOUR = 23,
                        AWSbucket = "https://noaa-nexrad-level2.s3.amazonaws.com/"
                        ) {

dates.year <- substring(DATES, 1, 4)
dates.month <- substring(DATES, 5, 6)
dates.day <- substring(DATES, 7,9)

times <- seq(startHOUR, endHOUR)

times <- stringr::str_pad(times, 2, pad="0")

#files <- paste0(STATION,DATES,"_", times)
files <- c(t(outer(DATES, times, paste)))
files <- sub(" ", "_", files)
files <- paste0(STATION, files)
location <- paste0(dates.year, "/", dates.month, "/", dates.day, "/", STATION, "/")
obj <- c(t(outer(location, files, paste0)))

library(aws.s3)
all <- list()
  all <- lapply( obj, function(o){
    get_bucket_df(bucket = "noaa-nexrad-level2", o)
  }
  )

toDWNLD <- plyr::ldply(all)$Key

DWNLDURLS <- c(t(outer(AWSbucket, toDWNLD, paste0)))

saveLoc=paste0(saveLoc, "/", STATION, "/")
saveFile=substr(toDWNLD, 17, 42)
dub <- (saveFile  %in% list.files(saveLoc))
saveFile = saveFile[!dub]

saveFile=paste0(saveLoc, saveFile)
toDWNLD =toDWNLD[!dub]


lapply(1:length(DWNLDURLS), function (i){ 
  download.file(DWNLDURLS[i], destfile = saveFile[i])
})

}

I can access v data for a single vp file:

bioRad::fetch(vp_files[1], quantity = "v")

But if I want to access it for 4 vp files (= vplist object):

bioRad::fetch(vp_files, quantity = "v")

I get the dens values back instead of v (which is not even the first attribute), even though according to the bioRad documentation, the fetch function should work on 3 class types:

Arguments

x: a vp,vplist or vpts object
quantity: a profile quantity, one of "HGHT","u","v","w","ff", "dd","sd_vvp","gap","dbz","eta", "dens","DBZH","n","n_all","n_dbz","n_dbz_all".

I know I can get at the data with vp_files[1]$data, but fetch is the recommended function... is the fact that dens is returned a bug or am I doing something wrong?

This is the error I get in my terminal (I'm using Ubuntu 14.04)

checkDocker()
docker: Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Post http://%2Fvar%2Frun%2Fdocker.sock/v1.26/containers/create?name=hello-world: dial unix /var/run/docker.sock: connect: permission denied.
See 'docker run --help'.

Thanks

--
Sergio Cabrera