yota-p / kaggle_ieee-fraud-detection Goto Github PK
View Code? Open in Web Editor NEWLicense: MIT License
License: MIT License
Currently feature scripts are not executable by itself, which needs to be called from featurefatory. Thus it is difficult to debug, I need to make it executable by itself.
Features raw.py, altgor.py are executable by itself.
Remove feature creating process and add feature loading to the main script.
Easy to debug feature creating scripts.
https://github.com/hakubishin3/kaggle_ieee
Currently train process is not logged.
Thus I can't check model training process after the batch process.
To output training log.
Be able to save & check training results after batch experiment.
Currently I'm using Full-process piped architecture, which means preprocess, train, predict are connected with pipe. However, Pipeline definition causes a lot of modification for switching pipes. I recently noticed it doesn't have to be luigi pipeline, it can be altered by creating a method to judge the task itself is done. Thus I want to rebuild my architecture.
To implement full architecture introduced by TIS Kubo 2018.
Full project
TIS Kubo 2018
Project architecture has been refined. Now it's time to reproduce the top Kaggler's scores.
Reproduce gold solution, which is Public LB > 0.946627.
Gets evidence of this project being able to reproduce top solutions.
Chirs Deotte - 1st place solution
https://www.kaggle.com/c/ieee-fraud-detection/discussion/111284
https://www.kaggle.com/c/ieee-fraud-detection/discussion/111308
https://www.kaggle.com/cdeotte/xgb-fraud-with-magic-0-9600
Currently we run training script by typing python src/main.py 0000
, which is not self-evident (here 0000 indicates a parameter set).
I want to gather commands in Makefile.
Run training by 'make train 0000'.
Run prediction by 'make pred 0000'.
Easy to run for beginners.
Easy to type.
Need to separate training and predicting code.
make run
to MakefileCurrently output filename is defined at both config and run.
Directory structure(log/, data/...) requires filenames to be determined on certain rules, thus defining filename at config is not necessary.
In this issue, naming rules will be defined, and modify configs not to define filenames, dirs.
Define filenames based on RULE below:
RULE
{filetype}_{VERSION}(.small){.extention}
data/feature/name_test(.small).pkl
data/feature/name_train(.small).pkl
data/raw/*.csv
data/model/model_0000(.small).json
data/submission/submission_0000.csv
log/main_0000(.small).log
log/train_0000(.small).tsv
Config file to be slim.
Dynamically change filename at run_{VERSION}.py.
None
If transformer reads .small data under --small run, it resamples already .small data.
xgboost.train does not recognize num_boost_round in params and default value used.
Feature importance score is not logged.
Deprecate train_and_validate. Validation data is optional.
Transformer to skip sampling step if --small and read data from file.
Pass num_boost_round to xgboost.train().
Log feature importance for full dataset.
python run_gbdt.py --small
To finish the objectives above.
None
I cloned source and tried to run, but couldn't finish running.
This because the required directoriy did not exist.
Check for directory data/interim, raw, processed, exterim
before running.
Create them if not exist.
python src/main.py 0000
Can't run further than downloading raw data.
Check for directory doesn't need to be executed automatically done when running.
Add directory creating step to Makefile. Later it will be depended from make run
in other issue.
Add creating directory step to Makefile.
Currently preprocess.py contains all preprocessing steps in one method.
Thus if this method raised error while running, we have to execute every steps from the head of method.
I want preprocess method to be divided and piped.
Preprocess method detects and skip finished steps.
Reduce re-executing time.
Current directory is having datas at top level, though
Finish Tasks below
Better access to each files.
Remove unneeded files.
None
This is how the directories will look like:
(move src/* to top, remove 's' from directory name, data in data/*)
data/model *ignore
data/submission *ignore
data/raw *ignore
data/feature *ignore
log/train_0000.tsv
log/main_0000.log
log/importance_0000.json
log/params_0000.json
util/
feature/
config/
run0000.py
...
run9999.py
Currently my highest score is Private 0.932896, Public 0.958595.
1st place solution is 0.967722 public, 0.945884 private.
To achieve Private score > 0.945884.
Proof of correct model.
1st Place Solution - Part 1
https://www.kaggle.com/c/ieee-fraud-detection/discussion/111284
1st Place Solution - Part 2
https://www.kaggle.com/c/ieee-fraud-detection/discussion/111308
Very short summary
https://www.kaggle.com/c/ieee-fraud-detection/discussion/111257
IEEE - Notebooks series (LB 0.9480/no blend)
https://www.kaggle.com/c/ieee-fraud-detection/discussion/104142
How to find UIDs
https://www.kaggle.com/c/ieee-fraud-detection/discussion/111510
How the magic works
https://www.kaggle.com/c/ieee-fraud-detection/discussion/111453
XGB Fraud with Magic - [0.9600]
https://www.kaggle.com/cdeotte/xgb-fraud-with-magic-0-9600
Kaggle Ensembling Guide
https://mlwave.com/kaggle-ensembling-guide/
None
Now I'm struggling to reproduce past experiment.
The main cause is because the dependent modules & utils change oftenly.
Discontinue complex class dependencies.
Integrate important methods & parameters for each experiments.
Reproductivity increases.
None
None
None
Current log file doesn’t include evaluation scores at each steps.
Evaluation scores are logged in each log files.
Able to analyze score variation over steps
None
None
GBDT models (LightGBM, XGBoost, CatBoost) have similar but slightly different methods.
Let's use them at same procedure.
To Switch models from config
i.e. To finish the tasks below.
Easy to switch GBDT models.
None
None
Currently the models are gradient boost trees only, we need to try other models such as neutral networks.
Implement neural network model
For future knowledge
None
None
None
Small enhancements
None
None
None
This project doesn't have a test code.
Need a test suite for essential modules.
One can run test suite by make test
.
Keep codes consistent to existing functions.
Reduce bugs in production code.
Reduce time for bug fixing.
make test
Currently the models are called directly from the training method.
Thus model & library specific codes are having dense union to the common code.
I want to call model using common method.
Model & library specific codes not implemented out of src/model.
Create two models implementing Model, Trainer, ModelAPI.
Can change model from config (without modifying the code!)
Be able to handle different models using same method.
Reduce risks of changing improper codes.
Use architecture from TIS Kubo.
TIS_Kubo_2016
TIS_Kubo_2018
Each models contain Model, Trainer, ModelAPI.
Class Model implements the model & library specific codes.
To train: Trainer.train() calls Model.train()
To predict: ModelAPI.predict() calls Model.predict()
This is a issue to gather small refactoring requests to improve productivity.
Add them in this issue and will be modified at once.
Note: This issue will accept up to 5 requests.
Each requests should be able to be fixed within 20 lines of coding.
Unsuitable tasks will not be fixed in this issue.
To complete all requests in this issue.
Usability imporvement.
Currently oof training data is being outputted to src/ directory, which is not for output.
Change output directory to log/train/.
Key files like .slack_token, .kaggle.json are dispersed in the project or system directory.
I need to gather keys to one directory to make them manageable.
Currently class AbstractTask() is for just forcing it's inheritors to implement output() and run().
This is replaceable by luigi.Task(), so AbstractTask is no longer needed.
Current trainer, modelapi are not executable.
To execute trainer, modelapi by itself
Easy to debug
Reduce execution time
None
None
None
Currently config is handled by save_log
module to call get_options
module.
This structure is taken over from external repository, but it looks weird to handle config files through log handler module.
Sample methods save_log.py contains:
get_version()
# for reading configget_training_logger()
# logsend_message()
# for slackI want to remove methods for config and slack from save_log module.
Separate configure and save_log module into individual codes.
Clear architecture.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.