Following a similar pattern to what is in the documentation for auto-lightgbm but for the m4 parquet weekly file and receiving an error several steps in. Maybe the error has to do with how much data is held out and the number of lags coupled with some short time series but not quite sure. Any thoughts?
[flaml.tune.tune: 07-23 00:17:53] {773} INFO - trial 1 config: {'n_estimators': 40, 'num_leaves': 2, 'reg_alpha': 0.002624570650559949, 'reg_lambda': 0.6080101522778687, 'colsample_bytree': 0.6176855370868456, 'subsample': 0.6211956701608718, 'subsample_freq': 4, 'min_child_samples': 10}
[flaml.tune.tune: 07-23 00:17:57] {773} INFO - trial 1 config: {'n_estimators': 40, 'num_leaves': 2, 'reg_alpha': 0.002624570650559949, 'reg_lambda': 0.6080101522778687, 'colsample_bytree': 0.6176855370868456, 'subsample': 0.6211956701608718, 'subsample_freq': 4, 'min_child_samples': 10}
[flaml.tune.tune: 07-23 00:18:04] {773} INFO - trial 1 config: {'n_estimators': 40, 'num_leaves': 2, 'reg_alpha': 0.002624570650559949, 'reg_lambda': 0.6080101522778687, 'colsample_bytree': 0.6176855370868456, 'subsample': 0.6211956701608718, 'subsample_freq': 4, 'min_child_samples': 10}
[flaml.tune.tune: 07-23 00:18:12] {773} INFO - trial 1 config: {'n_estimators': 40, 'num_leaves': 2, 'reg_alpha': 0.002624570650559949, 'reg_lambda': 0.6080101522778687, 'colsample_bytree': 0.6176855370868456, 'subsample': 0.6211956701608718, 'subsample_freq': 4, 'min_child_samples': 10}
[flaml.tune.tune: 07-23 00:18:20] {773} INFO - trial 1 config: {'n_estimators': 40, 'num_leaves': 2, 'reg_alpha': 0.002624570650559949, 'reg_lambda': 0.6080101522778687, 'colsample_bytree': 0.6176855370868456, 'subsample': 0.6211956701608718, 'subsample_freq': 4, 'min_child_samples': 10}
[flaml.tune.tune: 07-23 00:18:26] {773} INFO - trial 1 config: {'n_estimators': 40, 'num_leaves': 2, 'reg_alpha': 0.002624570650559949, 'reg_lambda': 0.6080101522778687, 'colsample_bytree': 0.6176855370868456, 'subsample': 0.6211956701608718, 'subsample_freq': 4, 'min_child_samples': 10}
[flaml.tune.tune: 07-23 00:18:35] {773} INFO - trial 1 config: {'n_estimators': 40, 'num_leaves': 2, 'reg_alpha': 0.002624570650559949, 'reg_lambda': 0.6080101522778687, 'colsample_bytree': 0.6176855370868456, 'subsample': 0.6211956701608718, 'subsample_freq': 4, 'min_child_samples': 10}
[flaml.tune.tune: 07-23 00:18:45] {773} INFO - trial 1 config: {'n_estimators': 40, 'num_leaves': 2, 'reg_alpha': 0.002624570650559949, 'reg_lambda': 0.6080101522778687, 'colsample_bytree': 0.6176855370868456, 'subsample': 0.6211956701608718, 'subsample_freq': 4, 'min_child_samples': 10}
[flaml.tune.tune: 07-23 00:18:52] {773} INFO - trial 1 config: {'n_estimators': 40, 'num_leaves': 2, 'reg_alpha': 0.002624570650559949, 'reg_lambda': 0.6080101522778687, 'colsample_bytree': 0.6176855370868456, 'subsample': 0.6211956701608718, 'subsample_freq': 4, 'min_child_samples': 10}
[flaml.tune.tune: 07-23 00:19:01] {773} INFO - trial 1 config: {'n_estimators': 40, 'num_leaves': 2, 'reg_alpha': 0.002624570650559949, 'reg_lambda': 0.6080101522778687, 'colsample_bytree': 0.6176855370868456, 'subsample': 0.6211956701608718, 'subsample_freq': 4, 'min_child_samples': 10}
[flaml.tune.tune: 07-23 00:19:12] {773} INFO - trial 1 config: {'n_estimators': 40, 'num_leaves': 2, 'reg_alpha': 0.002624570650559949, 'reg_lambda': 0.6080101522778687, 'colsample_bytree': 0.6176855370868456, 'subsample': 0.6211956701608718, 'subsample_freq': 4, 'min_child_samples': 10}
[flaml.tune.tune: 07-23 00:19:20] {773} INFO - trial 1 config: {'n_estimators': 40, 'num_leaves': 2, 'reg_alpha': 0.002624570650559949, 'reg_lambda': 0.6080101522778687, 'colsample_bytree': 0.6176855370868456, 'subsample': 0.6211956701608718, 'subsample_freq': 4, 'min_child_samples': 10}
[flaml.tune.tune: 07-23 00:19:32] {773} INFO - trial 1 config: {'n_estimators': 40, 'num_leaves': 2, 'reg_alpha': 0.002624570650559949, 'reg_lambda': 0.6080101522778687, 'colsample_bytree': 0.6176855370868456, 'subsample': 0.6211956701608718, 'subsample_freq': 4, 'min_child_samples': 10}
[flaml.tune.tune: 07-23 00:19:43] {773} INFO - trial 1 config: {'n_estimators': 40, 'num_leaves': 2, 'reg_alpha': 0.002624570650559949, 'reg_lambda': 0.6080101522778687, 'colsample_bytree': 0.6176855370868456, 'subsample': 0.6211956701608718, 'subsample_freq': 4, 'min_child_samples': 10}
[flaml.tune.tune: 07-23 00:19:57] {773} INFO - trial 1 config: {'n_estimators': 40, 'num_leaves': 2, 'reg_alpha': 0.002624570650559949, 'reg_lambda': 0.6080101522778687, 'colsample_bytree': 0.6176855370868456, 'subsample': 0.6211956701608718, 'subsample_freq': 4, 'min_child_samples': 10}
[flaml.tune.tune: 07-23 00:20:10] {773} INFO - trial 1 config: {'n_estimators': 40, 'num_leaves': 2, 'reg_alpha': 0.002624570650559949, 'reg_lambda': 0.6080101522778687, 'colsample_bytree': 0.6176855370868456, 'subsample': 0.6211956701608718, 'subsample_freq': 4, 'min_child_samples': 10}
[flaml.tune.tune: 07-23 00:20:22] {773} INFO - trial 1 config: {'n_estimators': 40, 'num_leaves': 2, 'reg_alpha': 0.002624570650559949, 'reg_lambda': 0.6080101522778687, 'colsample_bytree': 0.6176855370868456, 'subsample': 0.6211956701608718, 'subsample_freq': 4, 'min_child_samples': 10}
[flaml.tune.tune: 07-23 00:20:36] {773} INFO - trial 1 config: {'n_estimators': 40, 'num_leaves': 2, 'reg_alpha': 0.002624570650559949, 'reg_lambda': 0.6080101522778687, 'colsample_bytree': 0.6176855370868456, 'subsample': 0.6211956701608718, 'subsample_freq': 4, 'min_child_samples': 10}
[flaml.tune.tune: 07-23 00:20:50] {773} INFO - trial 1 config: {'n_estimators': 40, 'num_leaves': 2, 'reg_alpha': 0.002624570650559949, 'reg_lambda': 0.6080101522778687, 'colsample_bytree': 0.6176855370868456, 'subsample': 0.6211956701608718, 'subsample_freq': 4, 'min_child_samples': 10}
[flaml.tune.tune: 07-23 00:21:04] {773} INFO - trial 1 config: {'n_estimators': 40, 'num_leaves': 2, 'reg_alpha': 0.002624570650559949, 'reg_lambda': 0.6080101522778687, 'colsample_bytree': 0.6176855370868456, 'subsample': 0.6211956701608718, 'subsample_freq': 4, 'min_child_samples': 10}
[flaml.tune.tune: 07-23 00:21:18] {773} INFO - trial 1 config: {'n_estimators': 40, 'num_leaves': 2, 'reg_alpha': 0.002624570650559949, 'reg_lambda': 0.6080101522778687, 'colsample_bytree': 0.6176855370868456, 'subsample': 0.6211956701608718, 'subsample_freq': 4, 'min_child_samples': 10}
[flaml.tune.tune: 07-23 00:21:36] {773} INFO - trial 1 config: {'n_estimators': 40, 'num_leaves': 2, 'reg_alpha': 0.002624570650559949, 'reg_lambda': 0.6080101522778687, 'colsample_bytree': 0.6176855370868456, 'subsample': 0.6211956701608718, 'subsample_freq': 4, 'min_child_samples': 10}
[flaml.tune.tune: 07-23 00:21:53] {773} INFO - trial 1 config: {'n_estimators': 40, 'num_leaves': 2, 'reg_alpha': 0.002624570650559949, 'reg_lambda': 0.6080101522778687, 'colsample_bytree': 0.6176855370868456, 'subsample': 0.6211956701608718, 'subsample_freq': 4, 'min_child_samples': 10}
[flaml.tune.tune: 07-23 00:22:09] {773} INFO - trial 1 config: {'n_estimators': 40, 'num_leaves': 2, 'reg_alpha': 0.002624570650559949, 'reg_lambda': 0.6080101522778687, 'colsample_bytree': 0.6176855370868456, 'subsample': 0.6211956701608718, 'subsample_freq': 4, 'min_child_samples': 10}
[flaml.tune.tune: 07-23 00:22:25] {773} INFO - trial 1 config: {'n_estimators': 40, 'num_leaves': 2, 'reg_alpha': 0.002624570650559949, 'reg_lambda': 0.6080101522778687, 'colsample_bytree': 0.6176855370868456, 'subsample': 0.6211956701608718, 'subsample_freq': 4, 'min_child_samples': 10}
[flaml.tune.tune: 07-23 00:22:43] {773} INFO - trial 1 config: {'n_estimators': 40, 'num_leaves': 2, 'reg_alpha': 0.002624570650559949, 'reg_lambda': 0.6080101522778687, 'colsample_bytree': 0.6176855370868456, 'subsample': 0.6211956701608718, 'subsample_freq': 4, 'min_child_samples': 10}
[flaml.tune.tune: 07-23 00:23:00] {773} INFO - trial 1 config: {'n_estimators': 40, 'num_leaves': 2, 'reg_alpha': 0.002624570650559949, 'reg_lambda': 0.6080101522778687, 'colsample_bytree': 0.6176855370868456, 'subsample': 0.6211956701608718, 'subsample_freq': 4, 'min_child_samples': 10}
[flaml.tune.tune: 07-23 00:23:17] {773} INFO - trial 1 config: {'n_estimators': 40, 'num_leaves': 2, 'reg_alpha': 0.002624570650559949, 'reg_lambda': 0.6080101522778687, 'colsample_bytree': 0.6176855370868456, 'subsample': 0.6211956701608718, 'subsample_freq': 4, 'min_child_samples': 10}
[flaml.tune.tune: 07-23 00:23:39] {773} INFO - trial 1 config: {'n_estimators': 40, 'num_leaves': 2, 'reg_alpha': 0.002624570650559949, 'reg_lambda': 0.6080101522778687, 'colsample_bytree': 0.6176855370868456, 'subsample': 0.6211956701608718, 'subsample_freq': 4, 'min_child_samples': 10}
[flaml.tune.tune: 07-23 00:23:57] {773} INFO - trial 1 config: {'n_estimators': 40, 'num_leaves': 2, 'reg_alpha': 0.002624570650559949, 'reg_lambda': 0.6080101522778687, 'colsample_bytree': 0.6176855370868456, 'subsample': 0.6211956701608718, 'subsample_freq': 4, 'min_child_samples': 10}
[flaml.tune.tune: 07-23 00:24:18] {773} INFO - trial 1 config: {'n_estimators': 40, 'num_leaves': 2, 'reg_alpha': 0.002624570650559949, 'reg_lambda': 0.6080101522778687, 'colsample_bytree': 0.6176855370868456, 'subsample': 0.6211956701608718, 'subsample_freq': 4, 'min_child_samples': 10}
[flaml.tune.tune: 07-23 00:24:37] {773} INFO - trial 1 config: {'n_estimators': 40, 'num_leaves': 2, 'reg_alpha': 0.002624570650559949, 'reg_lambda': 0.6080101522778687, 'colsample_bytree': 0.6176855370868456, 'subsample': 0.6211956701608718, 'subsample_freq': 4, 'min_child_samples': 10}
[flaml.tune.tune: 07-23 00:25:02] {773} INFO - trial 1 config: {'n_estimators': 40, 'num_leaves': 2, 'reg_alpha': 0.002624570650559949, 'reg_lambda': 0.6080101522778687, 'colsample_bytree': 0.6176855370868456, 'subsample': 0.6211956701608718, 'subsample_freq': 4, 'min_child_samples': 10}
[flaml.tune.tune: 07-23 00:25:26] {773} INFO - trial 1 config: {'n_estimators': 40, 'num_leaves': 2, 'reg_alpha': 0.002624570650559949, 'reg_lambda': 0.6080101522778687, 'colsample_bytree': 0.6176855370868456, 'subsample': 0.6211956701608718, 'subsample_freq': 4, 'min_child_samples': 10}
[flaml.tune.tune: 07-23 00:25:47] {773} INFO - trial 1 config: {'n_estimators': 40, 'num_leaves': 2, 'reg_alpha': 0.002624570650559949, 'reg_lambda': 0.6080101522778687, 'colsample_bytree': 0.6176855370868456, 'subsample': 0.6211956701608718, 'subsample_freq': 4, 'min_child_samples': 10}
[flaml.tune.tune: 07-23 00:26:11] {773} INFO - trial 1 config: {'n_estimators': 40, 'num_leaves': 2, 'reg_alpha': 0.002624570650559949, 'reg_lambda': 0.6080101522778687, 'colsample_bytree': 0.6176855370868456, 'subsample': 0.6211956701608718, 'subsample_freq': 4, 'min_child_samples': 10}
[flaml.tune.tune: 07-23 00:26:34] {773} INFO - trial 1 config: {'n_estimators': 40, 'num_leaves': 2, 'reg_alpha': 0.002624570650559949, 'reg_lambda': 0.6080101522778687, 'colsample_bytree': 0.6176855370868456, 'subsample': 0.6211956701608718, 'subsample_freq': 4, 'min_child_samples': 10}
[flaml.tune.tune: 07-23 00:26:56] {773} INFO - trial 1 config: {'n_estimators': 40, 'num_leaves': 2, 'reg_alpha': 0.002624570650559949, 'reg_lambda': 0.6080101522778687, 'colsample_bytree': 0.6176855370868456, 'subsample': 0.6211956701608718, 'subsample_freq': 4, 'min_child_samples': 10}
[flaml.tune.tune: 07-23 00:27:22] {773} INFO - trial 1 config: {'n_estimators': 40, 'num_leaves': 2, 'reg_alpha': 0.002624570650559949, 'reg_lambda': 0.6080101522778687, 'colsample_bytree': 0.6176855370868456, 'subsample': 0.6211956701608718, 'subsample_freq': 4, 'min_child_samples': 10}
[flaml.tune.tune: 07-23 00:27:45] {773} INFO - trial 1 config: {'n_estimators': 40, 'num_leaves': 2, 'reg_alpha': 0.002624570650559949, 'reg_lambda': 0.6080101522778687, 'colsample_bytree': 0.6176855370868456, 'subsample': 0.6211956701608718, 'subsample_freq': 4, 'min_child_samples': 10}
[flaml.tune.tune: 07-23 00:28:09] {773} INFO - trial 1 config: {'n_estimators': 40, 'num_leaves': 2, 'reg_alpha': 0.002624570650559949, 'reg_lambda': 0.6080101522778687, 'colsample_bytree': 0.6176855370868456, 'subsample': 0.6211956701608718, 'subsample_freq': 4, 'min_child_samples': 10}
[flaml.tune.tune: 07-23 00:28:33] {773} INFO - trial 1 config: {'n_estimators': 40, 'num_leaves': 2, 'reg_alpha': 0.002624570650559949, 'reg_lambda': 0.6080101522778687, 'colsample_bytree': 0.6176855370868456, 'subsample': 0.6211956701608718, 'subsample_freq': 4, 'min_child_samples': 10}
[flaml.tune.tune: 07-23 00:29:00] {773} INFO - trial 1 config: {'n_estimators': 40, 'num_leaves': 2, 'reg_alpha': 0.002624570650559949, 'reg_lambda': 0.6080101522778687, 'colsample_bytree': 0.6176855370868456, 'subsample': 0.6211956701608718, 'subsample_freq': 4, 'min_child_samples': 10}
[flaml.tune.tune: 07-23 00:29:26] {773} INFO - trial 1 config: {'n_estimators': 40, 'num_leaves': 2, 'reg_alpha': 0.002624570650559949, 'reg_lambda': 0.6080101522778687, 'colsample_bytree': 0.6176855370868456, 'subsample': 0.6211956701608718, 'subsample_freq': 4, 'min_child_samples': 10}
[flaml.tune.tune: 07-23 00:29:52] {773} INFO - trial 1 config: {'n_estimators': 40, 'num_leaves': 2, 'reg_alpha': 0.002624570650559949, 'reg_lambda': 0.6080101522778687, 'colsample_bytree': 0.6176855370868456, 'subsample': 0.6211956701608718, 'subsample_freq': 4, 'min_child_samples': 10}
[flaml.tune.tune: 07-23 00:30:18] {773} INFO - trial 1 config: {'n_estimators': 40, 'num_leaves': 2, 'reg_alpha': 0.002624570650559949, 'reg_lambda': 0.6080101522778687, 'colsample_bytree': 0.6176855370868456, 'subsample': 0.6211956701608718, 'subsample_freq': 4, 'min_child_samples': 10}
[flaml.tune.tune: 07-23 00:30:46] {773} INFO - trial 1 config: {'n_estimators': 40, 'num_leaves': 2, 'reg_alpha': 0.002624570650559949, 'reg_lambda': 0.6080101522778687, 'colsample_bytree': 0.6176855370868456, 'subsample': 0.6211956701608718, 'subsample_freq': 4, 'min_child_samples': 10}
---------------------------------------------------------------------------
ShapeError Traceback (most recent call last)
[<ipython-input-12-453b2f0928c1>](https://localhost:8080/#) in <cell line: 11>()
9 time_budget=time_budget,
10 )
---> 11 forecaster.fit(y=weekly_train_pl)
12
13 # Get best lags and model hyperparameters
8 frames
[/usr/local/lib/python3.10/dist-packages/functime/base/forecaster.py](https://localhost:8080/#) in fit(self, y, X)
78 X = self._enforce_string_cache(X.lazy().collect())
79 X = X.lazy()
---> 80 artifacts = self._fit(y=y, X=X)
81 cutoffs = y.groupby(y.columns[0]).agg(pl.col(y.columns[1]).max().alias("low"))
82 artifacts["__cutoffs"] = cutoffs.collect(streaming=True)
[/usr/local/lib/python3.10/dist-packages/functime/forecasting/automl.py](https://localhost:8080/#) in _fit(self, y, X)
108 from functime.forecasting._ar import fit_cv
109
--> 110 return fit_cv(
111 y=y,
112 X=X,
[/usr/local/lib/python3.10/dist-packages/functime/forecasting/_ar.py](https://localhost:8080/#) in fit_cv(y, forecaster_cls, freq, min_lags, max_lags, max_horizons, strategy, test_size, step_size, n_splits, time_budget, search_space, points_to_evaluate, low_cost_partial_config, num_samples, cv, X, **kwargs)
149 scores_path = []
150 for lags in lags_path:
--> 151 score = evaluate(
152 **{
153 "lags": lags,
[/usr/local/lib/python3.10/dist-packages/functime/forecasting/_evaluate.py](https://localhost:8080/#) in evaluate(lags, n_splits, time_budget, points_to_evaluate, num_samples, low_cost_partial_config, test_size, max_horizons, strategy, freq, forecaster_cls, y_splits, X_splits, search_space)
134 score = result["mae"]
135 else:
--> 136 tuner = flaml.tune.run(
137 partial(
138 evaluate_windows,
[/usr/local/lib/python3.10/dist-packages/flaml/tune/tune.py](https://localhost:8080/#) in run(evaluation_function, config, low_cost_partial_config, cat_hp_cost, metric, mode, time_budget_s, points_to_evaluate, evaluated_rewards, resource_attr, min_resource, max_resource, reduction_factor, scheduler, search_alg, verbose, local_dir, num_samples, resources_per_trial, config_constraints, metric_constraints, max_failure, use_ray, use_spark, use_incumbent_result_in_evaluation, log_file_name, lexico_objectives, force_cancel, n_concurrent_trials, **ray_args)
774 result = None
775 with PySparkOvertimeMonitor(time_start, time_budget_s, force_cancel):
--> 776 result = evaluation_function(trial_to_run.config)
777 if result is not None:
778 if isinstance(result, dict):
[/usr/local/lib/python3.10/dist-packages/functime/forecasting/_evaluate.py](https://localhost:8080/#) in evaluate_windows(config, lags, n_splits, test_size, max_horizons, strategy, freq, forecaster_cls, y_splits, X_splits)
78 y_train, y_test = y_splits[i]
79 X_train, X_test = X_splits[i] if X_splits is not None else None, None
---> 80 result = evaluate_window(
81 y_train=y_train,
82 y_test=y_test,
[/usr/local/lib/python3.10/dist-packages/functime/forecasting/_evaluate.py](https://localhost:8080/#) in evaluate_window(config, lags, test_size, max_horizons, strategy, freq, forecaster_cls, y_train, y_test, X_train, X_test)
44 y_test = y_test.sort([entity_col, time_col])
45 y_pred = y_pred.sort([entity_col, time_col])
---> 46 y_test = y_test.with_columns(**{time_col: y_pred.get_column(time_col)})
47 score = mae(y_true=y_test, y_pred=y_pred).get_column("mae").mean()
48 res = {"score": score}
[/usr/local/lib/python3.10/dist-packages/polars/dataframe/frame.py](https://localhost:8080/#) in with_columns(self, *exprs, **named_exprs)
7331 """
7332 return (
-> 7333 self.lazy()
7334 .with_columns(*exprs, **named_exprs)
7335 .collect(no_optimization=True)
[/usr/local/lib/python3.10/dist-packages/polars/lazyframe/frame.py](https://localhost:8080/#) in collect(self, type_coercion, predicate_pushdown, projection_pushdown, simplify_expression, no_optimization, slice_pushdown, common_subplan_elimination, streaming)
1529 streaming,
1530 )
-> 1531 return wrap_df(ldf.collect())
1532
1533 def sink_parquet(
ShapeError: unable to add a column of length 7644 to a dataframe of height 9334