Comments (2)
@nzw0301 Thank you for your reply. #5118 seems to have a confusing objective, and the answers seem to solve the issue using features already implemented in Optuna.
The proposed feature here might also help in #5118 , but I would consider this as an independent issue. See below a diagram of the proposed structure. The role of Parent Wrapper is to allow the comparison between different runs of nested cross-validation, keeping the study database table organized and the optuna-dashboard tidy. Maybe we can adopt the term "Experiment" for this new class?
The results of individual Studies are irrelevant when assessing the results of nested cross-validation, as we are interested in the overall performance of all studies (i.e mean accuracy of the best model of each Study). The "Experiment" class could implement a method set_metric()
or equivalent to manually set the overall performance metric of the experiment. This Parent Wrapper could be used for different uses than nested cross-validation, allowing users to organize studies into groups.
I think users should not be forced to create an Experiment to be able to create Studies, as this would alter current implementations with Optuna and many users won't need Experiments at all. Instead, an Experiment object could be created and passed when creating Study objects. The Study database table could then store a Foreign Key to the parent Experiment. See below an example usage:
import optuna
import numpy as np
from sklearn.model_selection import KFold, cross_val_score
'''
Suppose that `X` and `y` are the input features and
input labels, respectively.
Suppose that `pipeline` is an sklearn Pipeline.
'''
search_spaces = {
'param_1': optuna.distributions.FloatDistribution(0.001, 0.4, log=True),
'param_2': optuna.distributions.IntDistribution(10, 500),
}
storage = 'sqlite:///example-storage.db'
# Here, we create the new Experiment object
experiment = Experiment(
experiment_name = 'nestedCV-1',
storage = storage
)
outer_scores = []
cv_outer = KFold()
for split, (train_idx, test_idx) in enumerate(cv_outer.split(X, y)):
study = optuna.create_study(
study_name=f'study_{split}',
storage=storage,
direction='maximize',
experiment=experiment # We assign this study to the experiment object we defined
)
X_train, X_test = X[train_idx], X[test_idx]
y_train, y_test = y[train_idx], y[test_idx]
def objective(trial):
trial_params = {key: trial._suggest(key, dist) for key, dist in search_spaces.items()}
pipeline.set_params(**trial_params)
cv_inner = KFold()
scores = cross_val_score(pipeline, X_train, y_train, cv=cv_inner)
return scores.mean()
study.optimize(objective, n_trials=100)
best_params = study.best_params
pipeline.set_params(**best_params)
pipeline.fit(X_train, y_train)
test_score = pipeline.score(X_test, y_test)
outer_scores.append(test_score)
experiment.set_metric('Mean Accuracy', np.mean(outer_scores)) # We add the overall performance metric (similar to `set_user_attr()`)
from optuna.
Thank you for feature request. Do you think #5118 is the same as this request?
from optuna.
Related Issues (20)
- Optuna Hyperband Algorithm Not Following Expected Model Training Scheme.
- Optuna Hyperband Algorithm Not Following Expected Model Training Scheme
- (sqlite3.DatabaseError) database disk image is malformed HOT 7
- Is there any plan for Maximal Update Parametrization support ? HOT 1
- Hyper-parameter optimization of YOLOv7 model
- Hyper-parameter optimization of YOLOv7 model
- Duration of iteration increases as the number of iterations increase, with distributed optimization. HOT 14
- Local Minima Skip HOT 8
- Enhance `plot_parallel_coordinate()` by eliminating redundant for-loops
- Use CPU-only PyTorch wheels on GitHub Actions
- from optuna.integration.xgboost import XGBoostPruningCallback error ModuleNotFoundError: No module named 'optuna_integration' HOT 3
- Inconsistent Dynamic Parameter Handling in Optuna Ask-Tell Interface with Conditional Trials
- Suggesting same parameters in several trials with discrete parameter grid HOT 1
- How to optimize the parameters to make the RMSE fitting value close to 0 when predicting multiple output items through multiple input items?
- Custom Executor Support HOT 1
- The study.best_params belong to several functions. Is it possible to group params to indicate which function they belong to? HOT 2
- trial.suggest_float doesn't work properly. HOT 1
- while opptimizing hyperparameter with optuna I get stuck in first trial and the trial 0 is not terminated
- Optuna restrict to only to the first trial and display it as the best one HOT 1
- Cannot build the document on Mac HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from optuna.