I'm getting ERROR:flask_migrate:Error while upgrading. about superset HOT 6 OPEN

Hey there, @lei1402! I'm here to help you with any bugs, questions, or contributions you have for Superset. Let's tackle this challenge together!

I found a similar issue that might help you resolve the "ERROR:flask_migrate:Error while upgrading" when upgrading Superset from version 2.0.0 to 4.0.2 using Python 3.10.9. Here are the steps to follow:

1. Shut down the running containers (if using Docker Compose):

   docker compose down

2. Update the Superset repository:

   git pull origin master

3. Restart the containers (if using Docker Compose):

   docker compose up

4. Upgrade Superset manually (if not using Docker Compose):

   pip install apache-superset --upgrade

5. Upgrade the metadata database:

   superset db upgrade
   superset init

Ensure you have backed up your metadata database before performing these steps ^[1].

Additionally, the warning about using in-memory storage for Flask-Limiter indicates that you should configure a proper storage backend for rate limiting. You can configure Redis as the storage backend by setting the RATELIMIT_STORAGE_URI in the superset/config.py file:

RATELIMIT_STORAGE_URI = "redis://host:port"

Replace host and port with the appropriate values for your Redis instance ^[2].

If you encounter any issues with the migration scripts or directories, you might need to check the specific migration scripts involved in the upgrade process. However, I couldn't find detailed information on the specific migration scripts or directories involved in the Superset upgrade process from version 2.0.0 to 4.0.2 ^[1].

To continue talking to Dosu, mention @dosu.

from superset.

The error during superset init after a successful superset db upgrade is likely due to an issue in the migration script 4a33124c18ad_mig_new_columnar_upload_perm.py. Specifically, the error handling in the upgrade function might not be sufficient to catch and handle all potential issues that could arise during the permission migration process.

Here is the relevant part of the migration script:

def upgrade():
    bind = op.get_bind()
    session = Session(bind=bind)

    do_upgrade(session)

    try:
        session.commit()
    except SQLAlchemyError as ex:
        session.rollback()
        raise Exception(f"An error occurred while upgrading permissions: {ex}") from ex

If an error occurs during the do_upgrade(session) call, it will be caught by the except block, which rolls back the session and raises a new exception with a message indicating that an error occurred while upgrading permissions. This could be the source of the error you encountered during superset init.

To diagnose the specific cause, you would need to check the exact error message that was raised. This message should provide more details about what went wrong during the permission migration process ^[1].

To continue talking to Dosu, mention @dosu.

from superset.

The error you're encountering, "Directory already exists and is not empty," indicates that the /home/superset/anaconda3/envs/superset/lib/python3.9/site-packages/superset/migrations directory already exists and contains files. This can cause issues during the superset db init process.

To resolve this, you can try the following steps:

1. Backup the existing migrations directory:

   mv /home/superset/anaconda3/envs/superset/lib/python3.9/site-packages/superset/migrations /home/superset/anaconda3/envs/superset/lib/python3.9/site-packages/superset/migrations_backup

2. Recreate the migrations directory:

   mkdir /home/superset/anaconda3/envs/superset/lib/python3.9/site-packages/superset/migrations

3. Run the superset db init command again:

   superset db init

This should allow the superset db init process to proceed without encountering the existing directory issue.

Additionally, to address the in-memory storage warning for Flask-Limiter, you can configure a proper storage backend like Redis. Add the following configuration to your superset/config.py file:

RATELIMIT_STORAGE_URI = "redis://localhost:6379"

Replace localhost:6379 with the appropriate values for your Redis instance ^[1].

To continue talking to Dosu, mention @dosu.

from superset.

superset db upgrade
superset init
Superset db upgrade was successful, but an error occurred during init
@dosu

from superset.

superset db upgrade
logging was configured successfully
2024-09-10 20:57:54,361:INFO:superset.utils.logging_configurator:logging was configured successfully
2024-09-10 20:57:54,365:INFO:root:Configured event logger of type <class 'superset.utils.log.DBEventLogger'>
/home/superset/anaconda3/envs/superset/lib/python3.9/site-packages/flask_limiter/extension.py:333: UserWarning: Using the in-memory storage for tracking rate limits as no storage was explicitly specified. This is not recommended for production use. See: https://flask-limiter.readthedocs.io#configuring-a-storage-backend for documentation about configuring the storage backend.
warnings.warn(
WARNI [alembic.env] SQLite Database support for metadata databases will be removed in a future version of Superset.
INFO [alembic.runtime.migration] Context impl SQLiteImpl.
INFO [alembic.runtime.migration] Will assume transactional DDL.
INFO [alembic.runtime.migration] Running upgrade e09b4ae78457 -> f3afaf1f11f0, add_unique_name_desc_rls
INFO [alembic.runtime.migration] Running upgrade f3afaf1f11f0 -> 7fb8bca906d2, permalink_rename_filterState
INFO [alembic.runtime.migration] Running upgrade 7fb8bca906d2 -> cdcf3d64daf4, Add user_id and dttm composite index to Log model
INFO [alembic.runtime.migration] Running upgrade cdcf3d64daf4 -> c747c78868b6, Migrating legacy TreeMap
INFO [alembic.runtime.migration] Running upgrade c747c78868b6 -> 06e1e70058c7, Migrating legacy Area
Upgraded 8/8 charts
INFO [alembic.runtime.migration] Running upgrade 06e1e70058c7 -> a39867932713, query_context_to_mediumtext
INFO [alembic.runtime.migration] Running upgrade a39867932713 -> 409c7b420ab0, add created_by_fk as owner
INFO [alembic.runtime.migration] Running upgrade 409c7b420ab0 -> ffa79af61a56, rename report_schedule.extra to extra_json
INFO [alembic.runtime.migration] Running upgrade ffa79af61a56 -> 6d3c6f9d665d, fix_table_chart_conditional_formatting_colors
INFO [alembic.runtime.migration] Running upgrade 6d3c6f9d665d -> 291f024254b5, drop_column_allow_multi_schema_metadata_fetch
INFO [alembic.runtime.migration] Running upgrade 291f024254b5 -> deb4c9d4a4ef, parameters in saved queries
INFO [alembic.runtime.migration] Running upgrade deb4c9d4a4ef -> 4ce1d9b25135, remove_filter_bar_orientation
INFO [alembic.runtime.migration] Running upgrade 4ce1d9b25135 -> f3c2d8ec8595, create_ssh_tunnel_credentials_tbl
INFO [alembic.runtime.migration] Running upgrade f3c2d8ec8595 -> 9c2a5681ddfd, convert key-value entries to json
INFO [alembic.runtime.migration] Running upgrade 9c2a5681ddfd -> c0a3ea245b61, remove_show_native_filters
INFO [alembic.runtime.migration] Running upgrade c0a3ea245b61 -> d0ac08bb5b83, invert_horizontal_bar_chart_order
INFO [alembic.runtime.migration] Running upgrade d0ac08bb5b83 -> b5ea9d343307, bar_chart_stack_options
INFO [alembic.runtime.migration] Running upgrade b5ea9d343307 -> 07f9a902af1b, drop postgres enum constrains for tags
INFO [alembic.runtime.migration] Running upgrade 07f9a902af1b -> 7e67aecbf3f1, chart-ds-constraint
INFO [alembic.runtime.migration] Running upgrade 7e67aecbf3f1 -> 4ea966691069, cross-filter-global-scoping
INFO [alembic.runtime.migration] Running upgrade 4ea966691069 -> 9ba2ce3086e5, migrate-pivot-table-v1-to-v2
INFO [alembic.runtime.migration] Running upgrade 9ba2ce3086e5 -> 4c5da39be729, migrate_treemap_chart
INFO [alembic.runtime.migration] Running upgrade 4c5da39be729 -> ae58e1e58e5c, migrate_dual_line_to_mixed_chart
INFO [alembic.runtime.migration] Running upgrade ae58e1e58e5c -> 83e1abbe777f, drop access_request
INFO [alembic.runtime.migration] Running upgrade 83e1abbe777f -> 90139bf715e4, add_currency_column_to_metrics
INFO [alembic.runtime.migration] Running upgrade 90139bf715e4 -> 6fbe660cac39, add on delete cascade for tables references
INFO [alembic.runtime.migration] Running upgrade 6fbe660cac39 -> 8e5b0fb85b9a, Add custom size columns to report schedule
INFO [alembic.runtime.migration] Running upgrade 8e5b0fb85b9a -> 240d23c7f86f, update_tag_model_w_description
INFO [alembic.runtime.migration] Running upgrade 240d23c7f86f -> f92a3124dd66, drop rouge constraints and tables
INFO [alembic.runtime.migration] Running upgrade f92a3124dd66 -> 6d05b0a70c89, add on delete cascade for owners references
INFO [alembic.runtime.migration] Running upgrade 6d05b0a70c89 -> 863adcf72773, delete obsolete Druid NoSQL slice parameters
INFO [alembic.runtime.migration] Running upgrade 863adcf72773 -> a23c6f8b1280, cleanup erroneous parent filter IDs
INFO [alembic.runtime.migration] Running upgrade a23c6f8b1280 -> bf646a0c1501, json_metadata
INFO [alembic.runtime.migration] Running upgrade bf646a0c1501 -> e0f6f91c2055, create_user_favorite_table
INFO [alembic.runtime.migration] Running upgrade e0f6f91c2055 -> ee179a490af9, deckgl-path-width-units
INFO [alembic.runtime.migration] Running upgrade ee179a490af9 -> 0769ef90fddd, Fix schema perm for datasets
INFO [alembic.runtime.migration] Running upgrade 0769ef90fddd -> 2e826adca42c, Fix schema for log
INFO [alembic.runtime.migration] Running upgrade 2e826adca42c -> 8ace289026f3, add on delete cascade for dashboard_slices
INFO [alembic.runtime.migration] Running upgrade 8ace289026f3 -> 4448fa6deeb1, add on delete cascade for embedded_dashboards
INFO [alembic.runtime.migration] Running upgrade 4448fa6deeb1 -> 9f4a086c2676, add_normalize_columns_to_sqla_model
INFO [alembic.runtime.migration] Running upgrade 9f4a086c2676 -> ec54aca4c8a2, Increase ab_user.email field size
INFO [alembic.runtime.migration] Running upgrade ec54aca4c8a2 -> 317970b4400c, Added always_filter_main_dttm to datasource
INFO [alembic.runtime.migration] Running upgrade 317970b4400c -> 4b85906e5b91, add on delete cascade for dashboard_roles
INFO [alembic.runtime.migration] Running upgrade 4b85906e5b91 -> b7851ee5522f, replay 317970b4400c
INFO [alembic.runtime.migration] Running upgrade b7851ee5522f -> 06dd9ff00fe8, add_percent_calculation_type_funnel_chart
INFO [alembic.runtime.migration] Running upgrade 06dd9ff00fe8 -> 65a167d4c62e, add indexes to report models
INFO [alembic.runtime.migration] Running upgrade 65a167d4c62e -> 59a1450b3c10, drop_filter_sets_table
INFO [alembic.runtime.migration] Running upgrade 59a1450b3c10 -> a32e0c4d8646, migrate-sunburst-chart
INFO [alembic.runtime.migration] Running upgrade 59a1450b3c10 -> 96164e3017c6
/home/superset/anaconda3/envs/superset/lib/python3.9/site-packages/superset/migrations/versions/2024-01-17_13-09_96164e3017c6_tagged_object_unique_constraint.py:58: SAWarning: WARNING: SQL-parsed foreign key constraint '('database_id', 'dbs', 'id')' could not be located in PRAGMA foreign_keys for table tab_state
metadata.reflect(bind=bind)
/home/superset/anaconda3/envs/superset/lib/python3.9/site-packages/superset/migrations/versions/2024-01-17_13-09_96164e3017c6_tagged_object_unique_constraint.py:58: SAWarning: WARNING: SQL-parsed foreign key constraint '('latest_query_id', 'query', 'client_id')' could not be located in PRAGMA foreign_keys for table tab_state
metadata.reflect(bind=bind)
/home/superset/anaconda3/envs/superset/lib/python3.9/site-packages/superset/migrations/versions/2024-01-17_13-09_96164e3017c6_tagged_object_unique_constraint.py:58: SAWarning: WARNING: SQL-parsed foreign key constraint '('database_id', 'dbs', 'id')' could not be located in PRAGMA foreign_keys for table table_schema
metadata.reflect(bind=bind)
INFO [alembic.runtime.migration] Running upgrade 96164e3017c6, a32e0c4d8646 -> 15a2c68a2e6b, merging two heads
INFO [alembic.runtime.migration] Running upgrade a32e0c4d8646 -> 214f580d09c9, migrate_filter_boxes_to_native_filters
INFO [alembic.runtime.migration] Running upgrade 214f580d09c9 -> e863403c0c50, drop_url_table
INFO [alembic.runtime.migration] Running upgrade e863403c0c50, 15a2c68a2e6b -> 1cf8e4344e2b, merging
INFO [alembic.runtime.migration] Running upgrade 1cf8e4344e2b -> 87d38ad83218, Migrate can_view_and_drill permission
INFO [alembic.runtime.migration] Running upgrade 87d38ad83218 -> 17fcea065655, change_text_to_mediumtext
(base) [superset@superset ~]$ superset db init
logging was configured successfully
2024-09-10 20:58:41,123:INFO:superset.utils.logging_configurator:logging was configured successfully
2024-09-10 20:58:41,126:INFO:root:Configured event logger of type <class 'superset.utils.log.DBEventLogger'>
/home/superset/anaconda3/envs/superset/lib/python3.9/site-packages/flask_limiter/extension.py:333: UserWarning: Using the in-memory storage for tracking rate limits as no storage was explicitly specified. This is not recommended for production use. See: https://flask-limiter.readthedocs.io#configuring-a-storage-backend for documentation about configuring the storage backend.
warnings.warn(
2024-09-10 20:58:42,031:ERROR:flask_migrate:Error: Directory /home/superset/anaconda3/envs/superset/lib/python3.9/site-packages/superset/extensions/../migrations already exists and is not empty
@dosu Still the same error

from superset.

@dosu Ignore the speed limit first and let him go through init first.

from superset.