speediedan / deep_classiflie_db Goto Github PK

add date aligned flow prior to convergence in dataprep and retrain

change db password to be read from env config file

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

Describe the solution you'd like
A clear and concise description of what you want to happen.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

add dataprep path that doesn't depend on previous model-based filtering (for initial model runs among other uses)

create a hard link of the datasets used in each model training session

refactor envconfig into class

refactor deep_classiflie_db core modules

refactor log_report in dc_tweebot into simpler funcs

create all_tweet_statements_tmp code path branch for filtering "false" truth tweets (truncate tweet temp table, etc.)

create tweet-based version of dist_based_filter_vw

expand exception handling to include tracebacks like tweetbot

save all report pics grouped by day into folders

Perform dataset ablations

update pt_converged_tweet_truths/falsehoods defs to use date bounds like pt_converged_dt_truths/falsehoods

refactor dataloading class, moving some functions to dataprep_utils

refactor config processing to avoid configuration complexity hell

Add stmt source scalar to data pipeline

Generate single truths sid-embedding cache, sid-falsehoods cache, find proper distance threshold for eliminating only falsehoods in truths table by truth sid

refactor deep_classiflie_db scraping_classes modules

Add static randomization seed to convergence subclass sql and main convergence sql for reproducibility

refine tweet-only dataset to use only tweets since 2017.01.20

update model analysis to reflect analysis of highest and lowest confidence predictions as well as fp and fn, making initial notes of data differences and enumerating approaches (in issues) to improve future post-release model performance on them

refactor dataloading class and underlying schema to generalize to multiple statement issuers/agents

remove dcbot inference_output dir

Find tweet stream source

address tweet index out of range bug (thread building related?)

update secondaryft_converge_dist_subclasses, secondaryft_converge_class_dist, secondaryft_converge_dist_class_sql, secondaryft_converge_dist_class_card_sql, secondaryft_converge_dist_dt_class_bound_card_sql defs

Add factbase tweet parser conditionally into initial tweet history loading path

create all_tweet_statements_tmp_v and all_tweet_statements_tmp temp table

add a flag (and logic) to enable publishing of only projected falsehoods on main thread

Fine tune model with tweet-only datasets

make auto and manual PEP8 code compliance changes

find appropriate "false truth" threshold (l2buckets) for tweets only dataset

add report retention period in days

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

Describe the solution you'd like
A clear and concise description of what you want to happen.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

fix name error fbase_statement childid truth test

update tweet_target_dist def

Write core tests

create tweet version of falsehood_data_driver on pt_converged_tweet_falsehoods, add secondaryft_dist_dt_bound_sql

add type hints to all relevant functions

Adapt secondary ft (tweet dataset) to include model-based filtering

Change truths view to canonical truths table/materialized view which includes sid

Add additional inputs to model such as geolocation etc.

Get stmt timings and stmt sources

increase factbase retry exponential backoff to accommodate longer factbase downtime

Score all falsehoods w/ model then remove sufficiently similar truths statements for each falsehood in training set only for further fine tuning

change reporting to report on all statements with accuracy > n

Purchase GPU for running model daemon

create dcbot service instead of aliases

remove unnecessary randomization seed from converged dt falsehoods