innixma / autogluon-benchmarking Goto Github PK

Hi Nick

I've read yours https://arxiv.org/abs/2003.06505
I think that mentioning that for at least couple of kaggle competitions obtaining top-scores is not possible without using data-leaks would be more fair to automated ml and would add some value to paper.
For example:

1. https://www.kaggle.com/c/santander-customer-transaction-prediction: https://www.kaggle.com/c/santander-customer-transaction-prediction/discussion/85125
2. https://www.kaggle.com/c/santander-value-prediction-challenge: https://www.kaggle.com/c/santander-value-prediction-challenge/discussion/63907
3. https://www.kaggle.com/c/ieee-fraud-detection: https://www.kaggle.com/c/ieee-fraud-detection/discussion/111284
   Detection and exploit such leaks is hard even for human.

Btw. do you have plans for paper release with updated benchmarks?

Anyways great work :)

When I ran python3 on a default AMI instance I got:

running develop
/home/ec2-user/.local/lib/python3.7/site-packages/setuptools/command/easy_install.py:159: EasyInstallDeprecationWarning: easy_install command is deprecated. Use build and pip and other standards-based tools.
  EasyInstallDeprecationWarning,
/home/ec2-user/.local/lib/python3.7/site-packages/setuptools/command/install.py:37: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
  setuptools.SetuptoolsDeprecationWarning,
error: can't create or remove files in install directory

The following error occurred while trying to add or remove files in the
installation directory:

    [Errno 13] Permission denied: '/usr/lib/python3.7/site-packages/test-easy-install-9513.write-test'

The installation directory you specified (via --install-dir, --prefix, or
the distutils default setting) was:

    /usr/lib/python3.7/site-packages/

So far I've got a bit further with:

sudo python3 setup.py develop

Hello!

Thanks a lot for providing this repository for the reproducibility of results. I have a question about the setup of the predictors.

In the paper, one can see that the machine employed for the runs has a pretty big memory in line with some of the big Kaggle datasets.

To ensure no AutoML framework is resource-limited, we ran the Kaggle benchmark for longer than the AutoML
datasets (4h and 8h time limits), and used more powerful AWS m5.24xlarge EC2 instances (384 GiB memory, 96 vCPU
cores).

If I look into the predictors, for instance autosklearn it looks like it is using 4 cores and dividing all of the virtual memory available among them, which makes me think each core of the got like 384/4 = 96 Gb for 4h/8h.

Is this a correct assumption? Thanks a lot for the clarification!

I would like to run my AutoML package on the same train/test split of OpenML datasets. Could you please give some instructions on how to obtain exact split for train/test?

Hi everyone,

thanks for this great repo, I am really looking forward to reproduce the benchmark results from your paper.
I have a question though: I've been working my way through the README_automlbenchmark.txt. I executed the bash script and installed AutoGluon's custom automlbenchmark, that worked so far. Now, I want to execute the run_automlbenchmark.py, but I am a bit confused about how I should properly set the variables in the script. Maybe you could give me some help there? Would really appreciate that.

Thanks & Cheers!