palashio / libra Goto Github PK

process for cropping: find median of height/width and interpolate each image up/down based on > or < then value.

Most image data sets consist of a CSV file with image path and labels as well as an overall image folder. Currently, the need to enter a path for every class seems redundant and might be difficult for most large image datasets. We might want to accept a CSV and Image directory path.

just text file highlighted common developer problems, and how to overcome

need to test single_reg_preprocesser preprocesser with many structured datasets. Need to report how it fails, and what it needs improvement upon.

because Libra is training multiple models we need to mute the logging that Keras has, where it outputs accuracies after every epoch and just do it for every model. This'll help clean up output when you're running the models.

need to figure out a way to make the log just stay at the bottom of the console and then update instead of re-printing out everytime.

try a recurrent neural network for identifying target instructions. PM me for this; i've tried using seq2seq and it didn't work.

querying google images is too unreliable; gonna get rid of this.

Use Scaler fit in the preprocess function to alter data before prediction

we need to be able to call .tune() for NLP tasks. Look into keras-tuner for this.

some of the queries like regression_querry_ann are quite tacky, a system for naming queries needs to be made.

need to create way in the structured_preprocesser() to identify columns that will reduce performance because of noise before training. This can be because columns are similar to each other and/or aren't coorelated whatsoever.

right now we're only allowing users to upload .csv files. Need support for excel files. We also need to make sure that when we do read in these files into pandas dataframes they're maintaining the same format/scheme as .csv files.

Both regression_query_ann() and classification_query_ann() need to be modified so that it's just one simple ann_query().

parts of queries that are used in each one like initial file reading etc. should be moved to one separate usable function to avoid code repetition.

sometimes datasets have columns with ID's, we need to remove them.

Here's my idea for removing them: columns that hold ID's are just non-numerical columns (which can be found using data[column].dtype.name != 'object where the number of unique elements obtained by np.unique['column'] is equal to the number of rows in the dataset.

so currently this line of code: while(all(x > y for x, y in zip(losses, losses[1:]))): inside the regression_query_ann checks to see if the new loss (with one more layer) is lower then the last loss. This results in always low stoppage. Need to find better mechanism.

documentation for library is very weak right now; we need to develop a method to document properly. Also create a file describing how the documentation works.

include a folder where we include example projects people can do with libra.

Many queries are lacking possible hyperparameters that users could specify. Examples would be for decision_tree_query(), user currently can't specify min_samples_leaf.

We should think about adding more machine learning models, such as the Linear Regression or Logistic Regression model.

currently, all the queries are just shoved into predictionQueries.py under client class. What's a better way of distributing these?

sentiment analysis NLP query. This should definitely be implemented. Could even be a pre-existing algorithm.

Queries should first see what sort of file is need to be read. If it's excel file, a different reading function should be applied etc.

when calling .tune() on functions, i haven't figured out how to extract the history from keras-tuner so right now, the tune() only updates the compiled model, not the accuracy, losses etc. Need to figure out how to get ALL values.

part of the textual module. a sequence to sequence query that converts any set of text to a new text.

Instead of filling the Nan values with Zero, replace the numerical value with the column mean/median value and Categorical value with last value before the missing value in the column. For better accuracy.

need to figure out best patience value for callback; read papers online and figure out when it's safe to say the model has overfit.

Image Augmentation: The downloaded images may not be enough so augmentation may play a vital role.

dates and times can come in so many different shapes and sizes; we need a whole method that can be integrated into structured_preprocesser() to deal with all possibilities.

python has a built in Levenshtein distance module. I can use this to compare two strings and get rid of produceMask().

We are leaking data by scaling train and test together currently

the main framework for the queries should be similar to how the pipeline is setup under the dev-pipeliner module. This whole conversion needs to happen immediately.

TensorFlow currently allows users to load multiple datasets within TensorFlow itself (i.e. MNIST, COCO, etc). We could add this to Libra, but include relevant datasets to the time right now, such as COVID-19 related datasets, Election-Related Datasets, etc.

look into implementing some sort of reinforcement learning query? how are most users setting up these queries. This is a bit more difficult because RL problems require action and agents are constantly changing. Maybe look into something more practical? Deep Q Networks?

Libra currently cannot be installed by using pip install libra command. This needs to be completed by using PyPi

sometimes the datasets people provide to Libra are gonna be massive; how're we managing this before we accept the dataset.

The final accuracy and some other items should stand out by being displayed in other colors.

Add GANs as either a new query to generate similar images to a given dataset, and maybe use this for data generation.

Textblob is a pretty weak part of speech identification algorithm. Try others like google's natural language api + research better ones.

https://github.com/Palashio/libradocs

sentences with a comma don't work well; i've assumed that users will only do things like "model x" or "please estimate y". When you create complex sentences it fails because there are many different varying part of speech locations

Create a document summarization query that can be used to analyze document information etc.

right now, produceMask() just creates an array of size 26 (representing the alphabet) and adds one to every element for every character. We need to find a better way to do this. Maybe use word2vec?

The models dictionary shouldn't be accessible directly by the user. We need to make an easy way to access the fields/values in this dictionary.

currently the labels (y) that we find after preprocessing are being scaled etc.

Something we could improve upon in terms of formats of data to be preprocessed could be an integration with Google Sheets/docs as well as an expansion enabling .tsv files.

merge the two queries into one. Depending on whether the target column is categorical vs numerical we choose a different loss function, activation, etc.