francescoperera / entree Goto Github PK

Given new FILE IO pipeline, apply class balancing.
Plan:

• create a map of counters for each class
• as a data point is read and labeled, updated appropriate counter
• if class counter reaches the limit, then ignore any other data point that will be labeled with that class.

Currently, S3 file listing works fine when the file is under a folder path.
This doesn't seem to work when the files are right under the bucket ( without folders).

Create branch, use S3 dev mode, and fix this by using the data in cb-json-data bucket

enforce encoding utf-8

update existing files in doc
update README
add doc for user-input.json

in the instance that important parameters in user-input.json are not present, set up default values. In particular, set up a default data format object.

Given new File IO pipeline, output file sizes are random.

• Use a mutable variable (integer) to the pipeline.
• As you writing data points to output, check that the lines in the output file are not above limit. IF above limit, close file, send to S3 , delete file, update var ,create new file and write next line to it.

Optimize and improve the current process of creating a data object:

• createDataObject
• getKeyValuePair
• createUnknownObjects

getKeyValuePair does not need all of these arguments. How do we differentiate between
createDataObject and createUnknownObjects.

add functionality to divide dump data into training and test files.
By default, do 80/20 split, have these values defined by the user.

potentially define cfnMap as input in user-input.json and use current cfnMap in CFNMappingCook as default map

potentially look at Alpakka

column field name mapping ( i.e email_address -> (email_address,emailaddress,email) is currently hardcoded. This is inefficient because any change in the mapping, requires a change in the code.

add tests for the basic Entree functionality

Connect to Kaggle or download relevant Kaggle datasets to dump. Create a column field name mapping & breakdown map for Kaggle datasets.

put every sensitive token/cred or var in the config file and use sys env. vars