rcgsheffield / cured Goto Github PK

Describe the solution you'd like

A function that takes these inputs:

• Working directory
• The unique ID of the data set
• Metadata (the data type of each column)

and has these outputs

• The path of the output directory
• A new set of data files in a binary format (e.g. Apache Parquet)

graph LR
    func["csv_to_binary()"]
    input1("Raw data directory") --> func
    input2("Metadata") --> func
    func --> output1("Output directory")
    func --> output2("Binary data files")

Describe alternatives you've considered

• Using R data frames to convert from CSV to binary
• Using Apache Arrow using the R library
• Using DuckDB (via the R library) using SQL language

Additional context

https://github.com/Joe-Heffer-Shef/cured/tree/master/csv_to_parquet

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

Describe the solution you'd like
A clear and concise description of what you want to happen.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

• https://quarto.org/
• R markdown
• Excel

Additional context
Add any other context or screenshots about the feature request here.

Assignee: Madina Hassan

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

We need to extract the metadata from the TOS in a format that's usable in our data pipeline.

Describe the solution you'd like
A clear and concise description of what you want to happen.

• Load the Excel TOS workbook(s) (There are many different ones! For now we can start with APC.)
• Find the relevant part of the spreadsheet
• Clean up these cells, capturing as much of the information as possible
• Save to a machine-readable format (JSON)

Ideally we'd save the metadata using some kind of widely-accepted standard (RO-Crate, JSON Schema, or something)

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

Describe the solution you'd like
A clear and concise description of what you want to happen.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.