Comments (7)
It seems like it is still loading one core at time although I specify to use all available cores. I would appreciate any advice here.
from deep-symbolic-optimization.
Your CPUs should definitely be pegged at 100% using this configuration.
As you know, setting n_cores_batch
to -1 should be shorthand to use all available cores. From core.py
:
if n_cores_batch == -1:
n_cores_batch = cpu_count()
if n_cores_batch > 1:
pool = Pool(n_cores_batch, initializer=set_task, initargs=(self.config_task,))
I'm wondering if there could be some OS/machine-specific issue with cpu_count
? Could you try the following in a Python shell?
import multiprocessing
print(multiprocessing.cpu_count())
Also, how are you running this config? If it's via command-line, is it the simple python -m dso.run config.json
or do you have other command-line arguments in that command?
One other thing to try that might help narrow things down is if the problem persists when removing the const
token from function_set
? On that note do you know if your Cython executable successfully installed? (Sometimes this fails but if so DSO falls back on a pure Python version)?
from deep-symbolic-optimization.
@IgorK7 FYI we just updated the repo to a new release. It had a lot of refactoring, so apologies if it breaks any existing configs; you might have to shuffle some things around in your configs following the new template config_common.json
.
Not sure if any of the edits will address the two issues you have raised.
from deep-symbolic-optimization.
Hi Brenden,
Thank you very much for getting back to me.
I am using your package in my research project where I investigate properties of symbolic regressions and DSR in particular in the set up with very high noise (true R2 at 5%). So all calculations require a lot of computation capacity and time.
I can confirm that with
print(multiprocessing.cpu_count())
the core count is correct. It loads all the cores but only for a fraction of capacity.
Yes, I tested it via command-line with
python -m dso.run config.json
and the effect is the same. The load per core is very low. Tracking average load on the build-in datasets is not reasonable (unless lopping) as they have no noise and the program just runs too fast. I supply my own data of 10000 rows and 2 explanatory variables with noise and the average load per core is low (like 5%).
The inclusion/exclusion of const
does not affect the load.
Cython is installed.
I run the algorithm on 3 different machines: Intel Mac with 10 physical cores, 2 instances of 48-cores on Google Compute Engine and Windows Subsystem for Linux (WSL) on an Intel PC with 8 physical cores. The results are the same - low load per core although all cores are loaded.
By the way, running the program on WSL should be a faster alternative to Docker if one needs to run it on Windows PC.
I am not sure how to make it run on CPUs at the full capacity.
Also, is there an option in the package set up to make it run on GPU? I could not find it but I am not a programmer.
Thank you very much!
from deep-symbolic-optimization.
I will also test the new version of the package. Thank you!
from deep-symbolic-optimization.
Regarding using GPU, I was able to make it utilize GPU but it still does not load to max capacity. The behavior is the same regardless of whether I use a small dataset (10,000 by 3) or a large one (10,000,000 by 9).
I was able to install tensorflow==1.14 + CUDA==10.2+cuDNN==8.6.0 on Ubuntu18.04 with some intel and Nvidia TeslaT4. (resolving compatibility issues on the go as tf==1.14 should only work with CUDA=10.0).
Still, any advice for speeding up computations would be appreciated.
from deep-symbolic-optimization.
GPU is not going to help. GPU will be used on the neural network (which is on the TensorFlow compute graph) but not computing the MSE (which is done off the compute graph). Since the DSO LSTM is a very small network, GPU just doesn't help. CPU (e.g., computing MSE on the dataset for the regression
task) is going to be the computational bottleneck.
from deep-symbolic-optimization.
Related Issues (20)
- Kernel crashes when importing tensorflow version 1.14 HOT 5
- Running dso.run will create a log folder but does not store or load the results csv files HOT 7
- Deap fitness objective HOT 1
- create primitive set mapping name clash HOT 2
- Deap varying with constraints HOT 4
- Methods for Running DSO HOT 4
- Normalization for input variables to domain (0,1) HOT 1
- Normalization for input variables to domain (0,1) HOT 1
- Different Learned Equation with Different Numpy Array Shape HOT 4
- Doubts about output HOT 1
- Ratio of training set to testing set HOT 1
- training iteration N, current best R : 0.9 HOT 1
- Trouble Installing, sample virtual environment config? HOT 7
- How to install configuration and generate an interactive platform? HOT 1
- Defining custom gaussian function HOT 3
- Ignoring errors... HOT 1
- adjusting best R or custom best R HOT 1
- how to run dso multiple times in a loop HOT 2
- How can parallel computing improve speed in a single-task scenario? HOT 1
- Installation error HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from deep-symbolic-optimization.