Migrating MySQL to Kafka

UDTS supports migration from MySQL to Kafka. Supported MySQL versions include MySQL (including the Percona version) 5.6/5.7/8.x. Supported Kafka versions are 2.x: including 2.0, 2.1, 2.2, 2.3, 2.4, 2.5, 2.6, 2.7, 2.8, etc.

1. Functional Limitations

1.1 Source MySQL Limitations

For incremental/complete + incremental migration, the source database needs to have binlog enabled, with the format set to ROW and image set to FULL.



Query method:
show global variables like 'binlog_format';
show global variables like 'binlog_row_image';

Setting method:
set global binlog_format = "ROW";
set global binlog_row_image = "FULL";

1.2 Target Kafka Limitations

Set auto.create.topics.enable to true.
Set delete.topic.enable to true.

2. Migration Content

Migration Content	Description
Migration Structure	Database, Table structure, and data
Migration Scope	Only migrates the databases and tables available when creating the task. Tables added during task operation will not be automatically migrated.
DDL	CREATE, ALTER, DROP statements
DML	Snapshot/insert/update/delete

3. Form Filling

Data Source Form

Parameter Name	Description
Address Type	Supports three types: internal network address, external network address, and dedicated line address. Internal network address requires filling in VPC and subnet information; external network address supports both IP and domain name; dedicated line address supports both IP and domain name. If using a domain name, the user network must have an external network exit.
Port	MySQL connection port
Username	MySQL connection username
Password	Password corresponding to the MySQL user
Database Name	MySQL database name. For all databases transmission, fill in *; to specify a single database, fill in the database name; to specify multiple databases, enter multiple database names separated by a comma. (Incremental migration cannot be performed if the database name contains spaces.)
Table Name	MySQL transmission table name. Only valid when “Database Name” specifies a single database. If not filled in, the default is to migrate all tables in the specified database; to specify a single table transmission, fill in the table name; to specify multiple tables, enter multiple table names separated by a comma.
Nolocks	Disabled by default. For partner RDS services that cannot obtain SUPER privileges, enable this option. See FAQ for how to obtain SUPER privileges for UDB.

Transmission Target Form

Parameter Name	Description
Internal Address	Connection address of the Kafka cluster, example: 92.168.1.10:9092,192.168.1.11:9092,192.168.1.12:9093
Maximum Rate	Limit value for Kafka transmission speed, used to adjust the transmission rate
Topic Prefix	UDTS creates corresponding topics on the target Kafka when migrating MySQL to Kafka. A topic is created for each table, and each topic will use this parameter as a prefix.
Default Partition Number	The default number of partitions for the corresponding topic created on the target Kafka when migrating MySQL to Kafka with UDTS.

4. Kafka Transmission Data Format

The content migrated to Kafka by UDTS is in the Debezium JSON format. Downstream synchronization can use Flink CDC, specifying the data format as debezium-json.