Organize your project files about ds-pipelines-targets-1 HOT 1 CLOSED

We're going to combine several things into this assignment. You'll be asked to make some modifications to your training repository and also to create a pull request that captures these changes.

Background

We use team conventions for how our pipelines are organized, which make it easier to hop in and out of collaborative projects and to rapidly understand what is going on where.

We refer to major elements of a pipeline as "phases", and name phases according to their purpose, such as 1_fetch or 2_process. These phases are used to separate files and data based on the intent of the code we are writing, and make it tractable to figure out where you'd need to edit code if you were coming in fresh to the project.

For medium to large pipelines projects, you'll see these workflow phases explicitly named by a number often followed by a verb (separated by an underscore). We use these phases to create different folders 📁 for data and code, and also to specify how we orchestrate the running of code (more on that later).

So, if we have a 1_fetch phase, code in the fetch folder 📁 would be used to do things like get data from web services, google drive, an FTP, or to scrape a website. 2_process (or 2_munge) might contain code that transforms the "fetched" data into more usable formats.

We recommend having src and out folders within each phase folder that contain code for this phase (src) and data (or other files) produced by this phase (out). When seeing some of our existing pipelines in action, you will also see other folders 📁 named in, log, and tmp to represent manually added files, logged/diagnostic output, and temporary data files, respectively.

⌨️ Activity: Restructure your code repository to follow our team's conventions for folders and files

Create a two phase directory structure for "fetch" and "process" concepts, and include src and out folders. Move the example script (my_happy_script.R) from the my_work_R folder into one of the src folders (at this time, it doesn't matter which one you choose) and delete any existing folders that aren't part of the intended structure.

When you are done, open a pull request with the changes.

Check your new pull request for a comment from me (you might have to wait a few seconds).

from ds-pipelines-targets-1.