ratshan / ds-pipelines-3 Goto Github PK
View Code? Open in Web Editor NEWHome Page: https://lab.github.com/USGS-R/many-task-pipelines
Home Page: https://lab.github.com/USGS-R/many-task-pipelines
In the last issue you noted a lingering inefficiency: When you added Illinois to the states
vector, your task table pipeline built WI_data
, MN_data
, and MI_data
again even though there was no need to download those files again. This happened because those three targets each depend on oldest_active_sites
, the inventory object, and that object changed to include information about a gage in Illinois. As I noted in that issue, it would be ideal if each task target only depended on exactly the values that determine whether the data need to be downloaded again. But we need a new tool to get there: a splitter.
The splitter we're going to create in this issue will split oldest_active_sites
into a separate table for each state. In this case each table will be just one row long, but there are plenty of situations where the inputs to a set of tasks will be larger, even after splitting into task-size chunks. Some splitters will be quick to run and others will take a while, but either way, we'll be saving ourselves time in the overall pipeline!
In general, scipiper best practices require that each command
creates exactly one output, either an R object or a file. To follow this policy, we could write a function that would take the full inventory and one state name and return a one-row table.
get_state_inventory <- function(sites_info, state) {
site_info <- dplyr::filter(sites_info, state_cd == state)
}
And then we could insert an initial task table step where we pulled out that state's information before passing it to the next step, such that each state's resulting targets in 123_state_tasks.yml would follow this pattern:
WI_inventory:
command: get_state_inventory(sites_info=oldest_active_sites, I('WI'))
WI_data:
command: get_site_data(sites_info=WI_inventory, state=I('WI'), parameter=parameter)
...but ugh, what a lot of text to add for such a simple operation! Also, suppose that sites_info
was a file that took a long time to read in - we've encountered cases like this for large spatial data files, for example - you'd have to re-open the file for each and every call to get_state_inventory()
, which would be excruciatingly slow for a many-state pipeline.
Fortunately, there's a better way.
Instead of calling get_state_inventory()
once for each state, we'll go ahead and write a single splitter function that accepts oldest_active_sites
and creates a single-row table for each state. It will be faster both to type (fewer targets) and to run (no redundant reloading of the data to split). The key is that instead of just creating as many tables as there are states, we'll also create a single summary file that concisely describes each of those outputs all in one place. In this way we only sort of break the "one-function, one-output" rule - we are creating many outputs, true, but one of them is an output that we can use as a scipiper target because it reflects the overall behavior of the function.
Ideally, we should design our summary file so that it changes anytime any of the split-up outputs change. We'll get more into the rationale in a later issue (very briefly, this approach makes it possible to use the summary file as an effective dependency of other pipeline targets). For now, I'm asking you to accept on faith that summary files should reflect the split-up outputs in a meaningful way.
To meet this need, our favorite format for summary files is a YAML of key-value pairs, where each key is a filename and each value is a hash of that file. A hash is a shortish character string (usually 16-32 characters) that describes the contents of a larger file or object. There are a number of algorithms for creating hashes (e.g., MD5, SHA-2, and SHA-3), but the cool feature that unites them all is that hashes are intended to be unique - if the contents of two files are even just slightly different, their hashes will always (OK, almost always) be different.
Explore the hash idea yourself using the digest::digest()
function on R objects or the tools::md5sum()
function on files. For example, you can hash your remake.yml file and get something like this:
> tools::md5sum('remake.yml')
remake.yml
"96b30271e93f4729ba4cbc6271c694df"
and then if you change it just a tiny bit - add a newline, for example - you are more or less guaranteed to get a different hash:
> tools::md5sum('remake.yml')
remake.yml
"9268e621b96ec9d35e9d9c0c2d58c19d"
Hash algorithms come from cryptography, which makes them extra cool ๐ต๏ธ
In this issue you'll create a splitter to make your task table more efficient in the face of a changing inventory in oldest_active_sites
. Your splitter function will generate a separate one-row inventory file for each state, plus a YAML file that summarizes all of the single-state inventories with a filenames and hashes roughly like this:
1_fetch/tmp/inventory_IL.tsv: 4c59d14d16b3af05dff6e6d6dfc8aac9
1_fetch/tmp/inventory_MI.tsv: fed321c051ee99e2c7b163c5c4c10320
1_fetch/tmp/inventory_MN.tsv: d2bff76a0631abf055421b86d033d80c
1_fetch/tmp/inventory_WI.tsv: b6167db818f91d792ec639b6ec4faf68
Ready?
Congratulations, @RAtshan, you've finished the course! โจ
Now that you're on the USGS Data Science team, you'll likely be using data pipelines like the one you explored here in your visualization and analysis projects. More questions will surely come up - when they do, you have some options:
vignette(package='scipiper')
In the last issue you noted some inefficiencies with writing out many nearly-identical targets within a remake.yml:
scmake()
if any target breaks along the way.In this issue we'll fix those inefficiencies by adopting the task table approach supported by scipiper.
A task table is a concept. The idea is that you can think of a split-apply-combine operation in terms of a table: each row of the table is a split (a task) and each column is an apply activity (a step). In the example analysis for this course, each row is a state and the first column contains calls to get_site_data()
for that state's oldest monitoring site. Later we'll create additional steps for tallying and plotting observations for each state's site. A task step is a cell within this conceptual table.
A task table is "conceptual" because it doesn't exist as a table in R. We implement it in two ways: as a task plan, which is a nested R list defining all the tasks and steps, and as a task remakefile, which is an automatically generated scipiper YAML file much like the remake.yml
you're already working with.
In this issue you'll create a task plan and task remakefile for this analysis of USGS's oldest gages.
Your pipeline is looking pretty good! Now it's time to add complexity. I've just added these two files to the repository:
In this issue you'll add these functions to the task table in the form of two new steps.
The goal of this issue is to expose you to multi-step task tables. They're not hugely different from the single-step task table we've already implemented, but there are a few details that you haven't seen yet. The changes you'll make for this issue will also set us up to touch on some miscellaneous pipeline topics. Briefly, we'll cover:
force = TRUE
argument to scmake()
in the task-table contextcreate_task_step()
The create_task_step()
function offers a lot of flexibility if you're willing to figure out the syntax. Specifically, you might have noticed the ...
in the arguments list of each function you pass to create_task_step()
. You don't get to decide what's available in the ...
s, but you do get do decide whether you use that information.
So far, the only named argument you've likely used is task_name
, but other named arguments are passed in to the target_name
and command
functions when the task step is turned into a task plan with create_task_plan()
. These other arguments allow you to look to other information about this task-step or previous steps within the task.
target_name
, you have access to the evaluated step_name
for this task-step. Therefore, your function definition can be function(...)
or function(step_name, ...)
depending on whether you plan to use the step_name information or not. It'll get passed in regardless of whether you include that argument in the definition.depends
function, you have access to the evaluated target_name
and step_name
. Therefore, your function definition can be either of these: function(target_name, ...)
, function(target_name, step_name, ...)
command
function, you have access to the evaluated target_name
, step_name
, and depends
. Therefore, your function definition can be any of these: function(target_name, ...)
, function(target_name, step_name, ...)
, or function(target_name, step_name, depends, ...)
step_name
, target_name
, depends
, or command
for any of the preceding steps for this task; all of these are available elements of the steps
argument, which is a list named steps
. Therefore, your function definition can be any of these: function(steps, ...)
, function(target_name, steps, ...)
, function(target_name, step_name, steps, ...)
, or function(target_name, step_name, depends, steps, ...)
. This is most useful if you're defining the 2nd, 3rd, etc. step, because then there's information to look back at. For example, if step 2 uses the target from step 1 as input, and if step 1 is named "prepare", then your definition for step 2 might look something like this:step2 <- create_task_step(
step_name = 'plot',
target_name = function(target_name, ...) {
sprintf("out/%s.png", target_name)
},
command = function(steps, ...) {
sprintf('visualize(\'%s\', target_name)', steps[['prepare']]$target_name)
}
)
You can always reference this information in the task steps vignette when you're working on real-world pipelines.
It's time to meet the data analysis challenge for this course! Over the next series of issues, you'll connect with the USGS National Water Information System (NWIS) web service to learn about some of the longest-running monitoring stations in USGS streamgaging history.
The repository is already set up with a basic scipiper data pipeline that:
Your pipeline is looking great, @RAtshan! It's time to put it through its paces and experience the benefits of a well-plumbed pipeline. The larger your pipeline becomes, the more useful are the tools you've learned in this course.
In this issue you will:
loop_tasks()
functionBefore you get started, make sure you have the most up-to-date version of scipiper:
packageVersion('scipiper')
## [1] โ0.0.20โ
You should have package version >= 0.0.20. If you don't, reinstall with:
remotes::install_github('USGS-R/scipiper')
So far we've implemented split and apply operations; now it's time to explore combine operations in scipiper pipelines.
In this issue you'll add two combiners to serve different purposes - the first will combine all of the annual observation tallies into one giant table, and the second will summarize the set of state-specific timeseries plots generated by the task table. We'll use the second combiner in a target within the main remake.yml, downstream of the whole task table, to illustrate how task tables can fit into the flow of a longer pipeline.
Combiners in scipiper are functions that accept a set of task-step targets as arguments and produce a single file or R object as output. We define their corresponding targets within the task remakefile (as opposed to within the main remake.yml) for several reasons:
remake::diagram(remake_file='remake.yml')
and searching in vain for targets such as WI_tally
:For the same reasons that purrr and dplyr allow you to implement split, apply, and combine operations all in a single expression, it's conceptually tidier to think of those three operations as a bundle within scipiper pipelines, and therefore to code them all in the same place.
The number of inputs to a combiner should change if the number of tasks changes. If we hand-coded a combiner target with a command
function that accepted a set of inputs (e.g., command: combine_tallies(WI_tally, MI_tally, [etc])
), we'd need to manually edit the inputs to that function anytime we changed the states
vector. That would be a pain and would make our pipeline susceptible to human error if we forgot or made a mistake in that editing. It's safer if we can automatically generate that list of inputs along with the rest of the task remakefile.
The scipiper way to use combiners is to work with the finalize_funs
, final_targets
, as_promises
, and tickquote_combinee_objects
arguments to create_task_makefile()
. Once you've set up these arguments properly, create_task_makefile()
will write combiner targets into your task remakefile for you, autopopulating the arguments to match the final step of each task.
The finalize_funs
argument is a vector of one or more function names, each one of which will be called as the command
to create a combiner target. Note that you don't get to specify anything else about your combiners here except their names. You can write your own combiner function or you can use the built-in combine_to_ind()
combiner for a common type of combining (you'll see when we try it out). If you write your own combiner function, it should be defined within one of the sources
or packages
specified in create_task_makefile()
.
You don't get choices about the list of arguments that a combiner function will accept: If the output of your combiner will be a file, your combiner function must accept arguments that are a summary filename, the output from task 1, the output from task 2, and so on, in that order, and your combiner must write the output to the filename given by the summary filename argument. If the output of your combiner will be an R object, your combiner function should skip straight to the outputs from the tasks (and not accept an initial filename argument) and should return an R object. The declaration of the combiner function should therefore either be function(out_file, ...)
or function(...)
(where the name of the outfile
argument is up to you).
The final_targets
argument is a vector of target names for your one or more combiner targets. This argument's length needs to exactly match that of finalize_funs
and will be mapped 1:1 to those target functions. If a finalize_fun
(combiner) will return a file, the corresponding final_targets
value should be a filename. It the combiner will return an R object, the final_targets
value should be a valid object target name.
The utility of as_promises=TRUE
will become clearer with an illustration shortly. For now you can use as_promises=FALSE
.
We've been working with tickquote_combinee_objects=FALSE
largely because you don't have any combiners. Once you add them, you should pretty much always use tickquote_combinee_objects=TRUE
. This argument helps create_task_makefile
format the remakefile correctly across a range of possible apply and combine operations.
In this course you'll learn about tools for data-intensive pipelines: how to download many datasets, run many models, or make many plots. Whereas a for
loop would work for many quick tasks, here our focus is on tools for managing sets of larger tasks that each take a long time and/or are subject to occasional failure.
A recurring theme in this activity will be the split-apply-combine paradigm, which has many implementations in many software languages because it is so darn useful for data analyses. It works like this:
There can be variations on this basic paradigm, especially for larger projects:
Assign yourself to this issue to explore the split-apply-combine paradigm further.
Please remember to be patient with me, too - sometimes I need a few seconds and/or a refresh before you'll see my response.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.