Pick a terminology for AWS DMS’s method for capturing data changes and stick with it consistently throughout. I like “Ongoing Replication” with no reference to CDC/change data capture because as a SQL DBA, CDC and change data capture are features of the SQL Server product so when you refer to Aws DMS’s technology as CDC/change data capture it is extremely confusing what is being referred to. Stick to AWS DMS technology as “Ongoing Replication” and MS-Replication and MS-CDC for Microsoft technologies.

To use CDC with SQL Server, you must create a publication for each table that is participating in ongoing replication.

• Instead of CDC say “Ongoing Replication”

9. Choose Security Settings and choose Run under the SQL Server Agent service account. Make sure to choose By impersonating the process account for publisher connection. Choose OK.

I assume this is for the Snapshot Agent “Security Settings” since the wizard won’t let you move past the Agent Security screen without this set. But there’s two “Security Settings” buttons so it’s not 100% clear

Provide a name of the publication in the format AR_PUBLICATION_000.

• If the DBID is a single digit (ex: 6) does the name have to be AR_PUBLICIATION_00006 (add another 0)? That is what is created when the publication is created by the DMS tool.

• If this is required then this should be noted, otherwise I would create the publication as AR_PUBLICIATION_0006 when following the instructions explicitly and without knowing how the automated process works.

Enable the Distribution option on all SQL Server instances in your Availability Replicas.

• This sounds like you’re talking about a feature of Always ON Availability Groups, when in reality I think you’re referring to setting up the distribution database for the MS-Replication install on the replicas.

The source database is postgres and target database is Redshift. So, DMS not able to skip or modify the query not supported by redshift like atler datatype as in the following example:
ALTER TABLE table_name
ALTER COLUMN column_name [SET DATA] TYPE new_data_type;

The DMS task is broken and the solutions to fix this by excluding the table and start the task, or restarting the task from the beginning.

In doc_source/CHAP_Troubleshooting.md there is mentioned we can create the DDL artifacts in a separate schema by setting the schema name in the parameter ddlArtifactsSchema. Please clarify if then DMS creates this schema itself or does DMS expects this schema to exist already?

This bug causes that not all data being loaded to the target database.

The solution is to add an ECA.

If you ok guys, I will add this to the Mysql troubleshooting section.

Cheers!

CHAP_Target.S3.DatePartitioning document does not exist even though the feature was announced via https://aws.amazon.com/about-aws/whats-new/2020/10/aws-database-migration-service-supports-transaction-commit-date-partitioning-cdc-mode-using-s3-target/. See “Use date partitioning when using S3 as a target” link leads to the DMS welcome page.

DMS Agent is not starting the service on the Linux Box and status always shows as not running.
Is there a log file that can be viewed to see the errors.

There seem to be more settings on the JSON for Validation than the documentation shows. I see:

    "ValidationSettings": {
        "EnableValidation": true,
        "ValidationMode": "ROW_LEVEL",
        "ThreadCount": 5,
        "PartitionSize": 10000,
        "FailureMaxCount": 10000,
        "RecordFailureDelayInMinutes": 5,
        "RecordSuspendDelayInMinutes": 30,
        "MaxKeyColumnSize": 8096,
        "TableFailureMaxCount": 1000,
        "ValidationOnly": false,
        "HandleCollationDiff": false,
        "RecordFailureDelayLimitInMinutes": 0
    },

I haven't found docs on other settings than EnableValidation and ThreadCount. Googling for "Data Migration Service" "ValidationSettings" "MaxKeyColumnSize" for instance turns out no results.

how often is AWS DMS triggered? Even with verbose logging I don't get any information every time I upload a new file to s3

such as:

ConnectTimeout 
ExecuteTimeout
CdcTimeout 
UnloadTimeout
ResumeFetchForXRows
...
...
...

https://github.com/awsdocs/aws-dms-user-guide/blob/master/doc_source/CHAP_Source.S3.md

When working with DMS using S3 as source ingesting csv files, is not clear whether there is a limit on the size of the csv files. I've personally experienced failures on big file sizes (+200Mb) which I resolved splitting files in multiple smaller files.

Is there any known limit on s3 csv file size?

Thanks 🙏

Due to this problem: https://forums.aws.amazon.com/thread.jspa?messageID=860564&tstart=0

Also confirmed by AWS support.

The documentation doesn't reflect this.

https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Target.Kinesis.html

In the section on Object Mapping the guide explains what map-record-to-record means, but all the examples only show rules for map-record-to-document. Conversely, there's no verbal explanation of what the latter type of mapping does.

Additionally, it's not clear whether the generated events are the same for each type of Operation or if e.g. the update one only contains the updated fields.

Also, why is a read event generated if it doesn't seem to correspond to a change in the source database?

Hi, I am migrating a Microsoft SqlServer database to RDS MySql. There is an error in the task because the source index name has empty space:
2018-11-21T04:47:05 [TARGET_LOAD ]E: RetCode: SQL_ERROR SqlState: HY000 NativeError: 1280 Message: [MySQL][ODBC 5.3(w) Driver][mysqld-5.7.23-log]Incorrect index name 'fuelsolutionheader_pk_fuelsolutionheader ' [1022502] (ar_odbc_stmt.c:4615)

The definition of the index in the source database is:
CREATE UNIQUE INDEX [pk_fuelsolutionheader ] ON dbo.fuelsolutionheader (lgh_number,request_id)

Notice that there is empty space at the end of the [pk_fuelsolutionheader ].
Since our goal is to migrate data over and don't worry about the extra space in the index name, can AWS DMS handle such typo cases (for example, trim the empty space) and finish the migration successfully?