Propagate DDL commands to workers v2 about citus HOT 9 CLOSED

We have 4 other issues that relate to propagating DDL changes. For now, I'm using this issue as a high level one and documenting other issues below.

• Issue #168 talks about extending DDL commands for Alter, Delete, and Create Table to support table sharding. I'm happy to at least remove the Alter Table part from this issue. @samay-sharma, what do you think?
• Issue #265 relates to failure handling for DDL command propagation. We decided to think about this as part of v2 changes (#25).
• Issue #312 relates to propagating CREATE INDEX commands that don't have index names.
• Issue #1007 relates to running CREATE INDEX CONCURRENTLY.
• Issue #1239 relates to adding support for ALTER TABLE ADD/DROP CONSTRAINT commands.

After release, we must remember to reply to citusdata/pg_shard#98 to direct future readers to this project so they can benefit from the DDL stuff it provides. At that point, we can close that ticket.

The following 4 issues don't directly relate to DDL command propagation, but are worth mentioning nonetheless.

• Issue #54 and #184 talks about DDL propagation for Unlogged Tables
• Issue #192 talks about DDL commands and defining primary keys
• Issue #40 talks about creating distributed materialized views

The following issues were related to DDL propagation. They have since been resolved:

• Issue #131 improves DDL propagation's performance by running these commands in parallel.
• Issue #274 touches on a potential usability issue with our current behavior -- for Citus pre-4.1 customers who built their own DDL propagation scripts.
• Issue #132 relates adding error or warning messages for unsupported ALTER TABLE commands.
• Issue #86 relates to propagating TRUNCATE to shards.

from citus.

Adding 2 more issues which we should evaluate during DDL propagation v2:

#356 relates to avoiding passing the text query string directly to the worker nodes.
#357 relates to considering if we should run the command on the master before the workers.

from citus.

Adding two more issues to evaluate during DDL propagation v2:

#354 relates to improving DDL propagation support for index related commands.
#376 relates to improving DDL propagation support for database related commands.

from citus.

#513 also relates to this issue. ALTER TABLE ... SET NOT NULL can have partial failures if some shards have rows with null values

from citus.

I have a question here. Are there any blockers not to support CONCURRENTLY for index creation?

from citus.

#633 relates to improving DDL propagation support for rename related commands.

from citus.

@onderkalaci -- We're now tracking CREATE INDEX CONCURRENTLY in #1007.

(edited based on @anarazel's comment below)

The reason we currently don't support it is failure semantics. With CREATE INDEX, we can issue the command within a transaction block. If we observe a failure, we can roll it back. CREATE INDEX CONCURRENTLY doesn't run inside a transaction, so we may need to incorporate more logic to handle out-of-band errors.

from citus.

from citus.

Most of the issues referenced here are closed. DDL is now transactional, parallel, and supports many more commands, including CREATE INDEX CONCURRENTLY. We can use individual issues to track missing features (or open a new parent issue).

from citus.