innixma / autogluon-benchmarking Goto Github PK
View Code? Open in Web Editor NEWBenchmarking Utilities for AutoGluon
License: Apache License 2.0
Benchmarking Utilities for AutoGluon
License: Apache License 2.0
Hi Nick
I've read yours https://arxiv.org/abs/2003.06505
I think that mentioning that for at least couple of kaggle competitions obtaining top-scores is not possible without using data-leaks would be more fair to automated ml and would add some value to paper.
For example:
Btw. do you have plans for paper release with updated benchmarks?
Anyways great work :)
When I ran python3 on a default AMI instance I got:
running develop
/home/ec2-user/.local/lib/python3.7/site-packages/setuptools/command/easy_install.py:159: EasyInstallDeprecationWarning: easy_install command is deprecated. Use build and pip and other standards-based tools.
EasyInstallDeprecationWarning,
/home/ec2-user/.local/lib/python3.7/site-packages/setuptools/command/install.py:37: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
setuptools.SetuptoolsDeprecationWarning,
error: can't create or remove files in install directory
The following error occurred while trying to add or remove files in the
installation directory:
[Errno 13] Permission denied: '/usr/lib/python3.7/site-packages/test-easy-install-9513.write-test'
The installation directory you specified (via --install-dir, --prefix, or
the distutils default setting) was:
/usr/lib/python3.7/site-packages/
So far I've got a bit further with:
sudo python3 setup.py develop
Hello!
Thanks a lot for providing this repository for the reproducibility of results. I have a question about the setup of the predictors.
In the paper, one can see that the machine employed for the runs has a pretty big memory in line with some of the big Kaggle datasets.
To ensure no AutoML framework is resource-limited, we ran the Kaggle benchmark for longer than the AutoML
datasets (4h and 8h time limits), and used more powerful AWS m5.24xlarge EC2 instances (384 GiB memory, 96 vCPU
cores).
If I look into the predictors, for instance autosklearn it looks like it is using 4 cores and dividing all of the virtual memory available among them, which makes me think each core of the got like 384/4 = 96 Gb
for 4h/8h.
Is this a correct assumption? Thanks a lot for the clarification!
I would like to run my AutoML package on the same train/test split of OpenML datasets. Could you please give some instructions on how to obtain exact split for train/test?
Hi everyone,
thanks for this great repo, I am really looking forward to reproduce the benchmark results from your paper.
I have a question though: I've been working my way through the README_automlbenchmark.txt
. I executed the bash script and installed AutoGluon's custom automlbenchmark, that worked so far. Now, I want to execute the run_automlbenchmark.py
, but I am a bit confused about how I should properly set the variables in the script. Maybe you could give me some help there? Would really appreciate that.
Thanks & Cheers!
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.