Comments (5)
In that example (https://ax.dev/tutorials/gpei_hartmann_developer.html) there are two main sources of randomness: The scrambling in the Sobol generator used for the initialization, and the randomness in the (quasi) Monte Carlo approximation used in the acquisition function.
Like @kkashin says, the first can easily be handled by making the change to
sobol = Models.SOBOL(search_space=exp.search_space, seed=1000)
And yes, this will require using the more explicit API rather than optimize
, whose purpose is to do a bunch of stuff under the hood for you.
Most of the randomness will come from that (the initialization). The actually Bayesian optimization phase will still have some randomness in that the acquisition function is evaluated with a Monte Carlo approximation. The seed for that is set here:
Ax/ax/models/torch/botorch_defaults.py
Line 162 in eebb8c7
So, you can just run
torch.manual_seed(1000)
before every call to gpei.gen
.
As a side note, for the purposes of benchmarking / performance evaluation I'd recommend setting the number of reps high enough that the seed doesn't matter. But of course there are other settings where reproducibility of a single run would be useful, and this should give that.
from ax.
I'm going to close out this issue for now, given that you're unblocked. We will look into making it possible to pass a seed directly to the optimize
function so that reproducibility is easier to achieve in the future, and will follow up with developments.
from ax.
@Irynei - what level are you looking for reproducibility at? The whole optimization loop or a specific generator?
You can set the seed for (quasi)-random generators, such as the Sobol generator (see https://ax.dev/api/models.html#ax.models.random.sobol.SobolGenerator.seed).
from ax.
I was looking for the whole optimization loop, in the optimize
method, ideally.
As far as I understand a (quasi)-random generator is chosen somehow under the hood?
So I can only set seed when I am using something similar to what explained here https://ax.dev/tutorials/gpei_hartmann_developer.html, right?
from ax.
@bletham, thank you for a great explanation! It's clear to me now.
from ax.
Related Issues (20)
- [Bug]: Custom metric issue HOT 4
- [GENERAL SUPPORT]: CI_Level Paretofrontier
- [Bug]: Large sample time increase in ax-platform >= version 0.3.5 HOT 6
- [GENERAL SUPPORT]: Reference Point for Multi-Objective Bayesian Optimization HOT 5
- [GENERAL SUPPORT]: Plotting Pareto fronts / Posterior mean model HOT 6
- [GENERAL SUPPORT]: Getting best predicted point of a botorch model HOT 10
- [GENERAL SUPPORT]: Logical-or in outcome constraints HOT 4
- [GENERAL SUPPORT]: Parallelism and arbitrary parameter type support. HOT 5
- [GENERAL SUPPORT]: How to Standardize and Normalize Data for Service API HOT 6
- [FEATURE REQUEST]: Implement ImprovementGlobalStoppingStrategy for batch trials HOT 2
- [GENERAL SUPPORT]: Plot observed pareto frontier alongside remaining Arms HOT 2
- [GENERAL SUPPORT]: Why `qLogNEI` by default? HOT 2
- [Bug]: No module named 'ax.analysis' HOT 2
- Performance issue HOT 11
- [GENERAL SUPPORT]: Multi-Objective, scale about the objectives HOT 3
- Multi-Objective, scale about the objectives HOT 2
- [GENERAL SUPPORT]: MultiTask with Choice Parameter using Ax Service API HOT 2
- [GENERAL SUPPORT]: Seeking Guidance on Surrogate Model and Acquisition Function for High-Dimensional Optimization Problem HOT 10
- [FEATURE REQUEST]: Issues with Customizing Data Classes HOT 3
- [FEATURE REQUEST]: RangeParameter handling periodic variables (such as angles) HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from ax.