What is the relationship between hydrofunctions, Ulmo, dataretrieval, HyRiver, and others? about hydrofunctions HOT 13 OPEN

Thanks Martin,

The summary for the HyRiver is good enough. As you said, I mostly focus on watershed data data. Your package was the reason that I didn't add coverage for all NWIS services and just added daily data which I needed for my research at the time. I designed the HyRiver project with extensibility in mind. Each one of the six packages in this software stack has a specific purpose that can be used as a standalone project and can be used in other packages. For example, PyGeoOGC and PyGeoUtils are the engines of the project that all the other packages rely on for generating queries and conversion to dataframes for other web services. These two packages are general and can be used for any geospatial web services.

Regarding coordination for further development, I agree. @emiliom created hydro_pycommunity repo for this purpose.

I think a good starting point would be establishing a framework for the projects that are within the scope of this collaboration. For example, we can create a repo that provides some guidelines for developers for starting a new project such as a categorized list of existing efforts (an awesome-style repo), and steps for creating a new project. We can come up with a template (maybe using cookiecutter) for projects to have some minimum requirements for linting, documentation, etc.. The README file should include some specific sections for example, Motivation, Scope, Usage, Installation, Credits, etc..

from hydrofunctions.

Thank you @aaraney ! I've been looking over HydroTools and love so many of its features:

• the subpackage structure to import what you want
• the separate _restclient, with built-in caching and async requests
• Great docstrings! And they look even better in Sphinx with your Furo style
• I like the tidy long-form dataframes... and then saving space by making some repetitive strings categorical data.

from hydrofunctions.

@mroberge this is an interesting issue, it's good to see so much community engagement! @cheginit told me about this issue and suggested I mention the project I help develop and maintain, HydroTools, at the Office of Water Prediction.

HydroTools is a namespace, toolbox like, package that is designed with data scientists in mind. As such, we've taken a different approach than it appears hydrofunctions has, we have a canonical pandas data frame format that all of our tools output and comply with and enforce opinionated data representations like naive UTC datetimes and heavy usage of categorical data types. Currently we have tools for retrieving and caching National Water Model forecasts from google cloud, an event based evaluation metrics package, a package of common hydrologic metrics, and a NWIS IV data service which implements caching, asynchronous data retrieval, and include data quality flags.

The two motivations for building our nwis_client was (IMO I can't speak for my colleagues) (1) to support large scale model evaluation activities where NWIS data at the continental scale is often required and (2) enforce UTC time on both input and outputs.

Our work does not access any NWIS services other than the instantaneous value service and we don't offer any plotting, quality control, or data resampling methods.

from hydrofunctions.

I think this is a good summary, Martin. I would actually be keen to transferring my maintenance efforts to supporting your script. I personally haven't contributed to other repositories because I don't feel experienced enough to make meaningful improvements. Everyone has their own way of accessing the NWIS services and making them available through commands. I am used to the commands in my library and know exactly what they are doing, but I wouldn't be opposed to collaborating. I like how you have added tests and good documentation to your library. I am also an earth-science educator, but just for introductory geology.

from hydrofunctions.

Thanks Martin,
You summarize dataretrieval well. In the Unix spirit, dataretrieval tries to do one thing - data retrieval of (mostly) USGS hydrologic data - and do that one thing well. I'm happy to help support additional USGS datasets or to work to standardize and integrate dataretrieval into a more cohesive ecosystem of tools. Although I often leave it on the back burner, I don't plan to abandon it unless a well-supported and stable alternative emerges. At this point it requires minimal maintenance, and I know I can fix it quickly whenever the REST APIs change.
-Tim

from hydrofunctions.

@mroberge @stijnvanhoey this all sounds great to me! i am guessing we will begin real work in May or June but i'd love to see how PyOS can work with you both and this community. we also have a cookie cutter -- and will be developing better standards in our contributing & dev guide over the next year. i'd love to get this communities input as we develop resources to support exactly this use case!

from hydrofunctions.

Hi all, I got invited to give a 15-min (virtual) talk at Pangeo Showcase about HyRiver, tomorrow (May 25th) at 4 pm EDT. I am going to going to talk about the state of the project and future direction. I think it would be a good opportunity to meet and have a discussion. I would be happy to see you there.

Edit: The correct date as Martin mentioned is May 26th, 4 pm EDT.

from hydrofunctions.

Looking forward to seeing it! (4pm Wednesday, May 26)

from hydrofunctions.

Thanks for sharing @cheginit. Unfortunately I have a conflict.

While I'm here: so sorry for not chiming in on this great thread 😞 ! I'll use Taher's ping as motivation for finally following up hopefully by the end of this week. Thanks to @mroberge for starting it, and great seeing everyone's input.

from hydrofunctions.

@emiliom Sure, I understand. The recording and link to the presentation material is here.

from hydrofunctions.

hi @mroberge et al! i just wanted to mention that i just received funding for @pyOpenSci and we will be starting an effort to help subcommunities organize for exactly this purpose. we also have considered needs such as finding other maintainers and such and i'm super open to what exactly the needs are to better support open source python.

I'm a little under the weather this week with what has happened in my town of Boulder, but wonder if there is a way in a few weeks to circle back and check in on whether pyopensci could facilitate helping you and this group build some community around your (and our) tools. I use hydrofunctions in my courses and really appreciate the package and the effort it takes to maintain a package like this.

from hydrofunctions.

Hi @lwasser ! Congratulations on getting funding! I've been following earthlab since @mbjoseph contacted me.

I'm interested! @pyOpenSci looks like a great idea and I would be happy to contribute and work with you.

from hydrofunctions.

Thanks @mroberge. With respect to pywaterinfo, it is indeed a (small) Python wrapper around the API used by the Flemish environmental agency to access the data available on https://www.waterinfo.be/Meetreeksen/ (they provide stream data, but also water quality parameters). In terms of maintenance and development, I cite @thodson-usgs

I don't plan to abandon it unless a well-supported and stable alternative emerges. At this point it requires minimal maintenance, and I know I can fix it quickly whenever the REST APIs change.

I'm certainly interested in a more community oriented approach. I do have the impression pywaterinfo is the only non-USGS data oriented package, but we can always see on which level some common ground can be found. For example, agree on an output (dataframe) format that would align with the other packages so users can easily reuse a certain analysis on data sets from different sources (waterinfo, USGS,...)?

In terms of documentation/cookiecutter-template/... guidelines as described by @cheginit. This is a very useful proposal, but I think we should build on the excellent work @lwasser and @pyOpenSci are already doing instead of defining a new/separate set of guidelines.

Looking forward to further collaboration.

from hydrofunctions.