Comments (9)
Hey, @hyzhak, glad to hear that it will be useful; it is almost done!
from ax.
@dkatz23238, thank you so much for this detailed feedback!
Re: docs on saving experiment –– that will be in soon! We didn't add it yet because we were not encouraging accessing experiment through the service API just yet, so that will change shortly.
Re: allowing to grab some plotting platform-agnostic data for plots –– understood, great idea, thank you!
Re: unique name for altered experiments –– yes, experiments do need to have unique names for storage, we'll make that more clear in the docs. Just to make sure: you can alter the optimization config on an existing experiment, but if you are creating a new experiment, it will need a unique name, even if it's an exact copy of an existing experiment.
from ax.
Hey, @dkatz23238! Just to clarify, are you looking to just save the underlying experiment in the service API, or are you looking to save the full state of optimization you're doing through service API, so that you can continue it at a later time (this latter option involves saving both the experiment and the state of the optimization, e.g. which model was being used, what its state was. etc.)?
Both of those use cases are currently in the works. However, if you are looking for the former, there is a workaround for now: you can save the experiment in the same way you would save it in the developer API (thorough save_experiment(experiment=ax_client.experiment)
. If the latter, that functionality is under development and will be available in later versions of Ax.
from ax.
I was looking for the best practice you gives you to store results for later inspection. The ax_client.experiment should work for my use case. It was just a bit confusing at first since the developer and loop API give you an explicit experiment object.
I am trying to store experiments in a backend to later query for visualization purposes using something like Power BI that would be the defacto tool we use in my organization.
Again, thanks for your help Lena!
from ax.
@dkatz23238, that makes total sense, thank you for pointing that out! The current workaround works for you for now, then?
Just out of curiosity, do you plan to use Ax visualizations or your own separate visualizations? Would it help if the service API provided an ability to directly retrieve (as plotly configs of AxPlotConfig
-s) the contour plot of response surfaces and trace of optimization on objective?
from ax.
My use case for having a db is to store many experiments and later visualize them among a small team sharing code. I don't think I will be using plotly because in my case we tend to stick to other solutions. From what I have seen so far I can easily join among the arm_v2 and metrics_v2 table to create the plots I need. This also makes it easy to create web visualizations using javascript and html.
I like the normalization work that has already done, thats why I wanted to test these features out instead of just storing to json or to a json style db like mongo.
The ax.experiment suggestion definitely worked for me however maybe the documentation can be a little bit more clear as there is no direct tutorial on storing an experiment. I wanted to make sure that sqa_store.db.create_all_tables(engine) did not delete any tables that already existed.
In the end my code looks something like this:
import ax.storage as axst
url = "mysql://root:password@localhost/axdb"
axst.sqa_store.db.init_engine_and_session_factory(url=url)
engine = axst.sqa_store.db.get_engine()
axst.sqa_store.db.create_all_tables(engine)
axst.sqa_store.save(experiment=ax.experiment)
Regarding your last question I do think this would be useful for plotly users, but maybe there can be a simple "framework" agnostic way to return only the data needed to produce a series of recommended plots.
What I plan on doing is just retrieving the data from SQL into a pandas dataframe for exploratory analysis.
Just an additional note:
Seems like every experiment stored in a mySQL db should have a unique name even if there have been changes to say the parameters or objective function. I dont know if this is documented anywhere but I have not found it.
My quick fix is the following
isonow = lambda : datetime.datetime.now().isoformat()
ax.create_experiment(
name="GaussianProcessRegression-%s" % isonow(),
parameters=PARAMETERS,
objective_name="mean_square_error",
minimize=True)
from ax.
I've faced similar problem. I need to preserve state of experiments on a long run of search (Service API), tried to use ax.storage.sqa_store.structs.DBSettings
but constantly getting :
NotImplementedError:
Saving and loading experiment in `AxClient` functionality currently under development.
So I can't wait to finally get that functionality it will be extremely useful
from ax.
There is now an ability to use SQA storage for the Service API. @hyzhak, @dkatz23238, you can make use of it by specifying DBSettings
here: https://github.com/facebook/Ax/blob/master/ax/service/ax_client.py#L103.
Not closing this issue yet as I would still like to add an example of how to use the DBSettings
to the Service API, but just letting you know!
from ax.
@lena-kashtelyan hurray! Thank you, will try soon!
from ax.
Related Issues (20)
- [Bug]: Custom metric issue HOT 4
- [GENERAL SUPPORT]: CI_Level Paretofrontier
- [Bug]: Large sample time increase in ax-platform >= version 0.3.5 HOT 6
- [GENERAL SUPPORT]: Reference Point for Multi-Objective Bayesian Optimization HOT 5
- [GENERAL SUPPORT]: Plotting Pareto fronts / Posterior mean model HOT 6
- [GENERAL SUPPORT]: Getting best predicted point of a botorch model HOT 10
- [GENERAL SUPPORT]: Logical-or in outcome constraints HOT 4
- [GENERAL SUPPORT]: Parallelism and arbitrary parameter type support. HOT 5
- [GENERAL SUPPORT]: How to Standardize and Normalize Data for Service API HOT 6
- [FEATURE REQUEST]: Implement ImprovementGlobalStoppingStrategy for batch trials HOT 2
- [GENERAL SUPPORT]: Plot observed pareto frontier alongside remaining Arms HOT 2
- [GENERAL SUPPORT]: Why `qLogNEI` by default? HOT 2
- [Bug]: No module named 'ax.analysis' HOT 2
- Performance issue HOT 11
- [GENERAL SUPPORT]: Multi-Objective, scale about the objectives HOT 3
- Multi-Objective, scale about the objectives HOT 2
- [GENERAL SUPPORT]: MultiTask with Choice Parameter using Ax Service API HOT 2
- [GENERAL SUPPORT]: Seeking Guidance on Surrogate Model and Acquisition Function for High-Dimensional Optimization Problem HOT 10
- [FEATURE REQUEST]: Issues with Customizing Data Classes HOT 3
- [FEATURE REQUEST]: RangeParameter handling periodic variables (such as angles) HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from ax.