Skip to Content
Data TransmissionSource is SQLServerSQL Server migration to Kafka

SQL Server Migration to Kafka

UDTS supports migration from SQL Server to Kafka. The supported SQL Server version is SQL Server 2008 R2 and above. Kafka supports version 2.x: including 2.0, 2.1, 2.2, 2.3, 2.4, 2.5, 2.6, 2.7, 2.8, etc.

1. Feature Limitations

1.1 Source SQL Server Limitations

  1. The source database and the tables to be migrated need to have the CDC feature enabled.
-- Specify the database name use dbname -- Enable CDC feature exec sys.sp_cdc_enable_db -- Check if the CDC is enabled for the dbname database, a return value of 1 means it is enabled select is_cdc_enabled from sys.databases where name = "dbname"; -- Enable CDC for the dbo.tablename table exec sys.sp_cdc_enable_table @source_schema = 'dbo',@source_name = 'tablename',@role_name = null; -- Check if the CDC is enabled for the dbo.tablename table, a return value of 1 means it is enabled select is_tracked_by_cdc from sys.tables where name = 'tablename' -- Modify the CDC data retention time, at least set to 1440 minutes (1 day) or more, recommended 14400 minutes (7 days). EXECUTE sys.sp_cdc_change_job @job_type = N'cleanup', @retention = 14400;

1.2 Target Kafka Limitations

  1. Set auto.create.topics.enable to true.
  2. Set delete.topic.enable to true.

2. Migration Content

Migration ContentDescription
Migration StructureDatabase, Table structure, and data
Migration ScopeOnly migrates tables that can be found when creating the task, newly added tables during the task will not be automatically migrated temporarily
DDLCREATE, ALTER, DROP statements
DMLSnapshot/insert/update/delete

3. Form Filling

Data Source Form

Parameter NameDescription
AddressSupports two methods: internal network address and dedicated line address. Internal network address requires VPC and subnet information; dedicated line address supports both IP and domain name, if using a domain name, a user network with an external network exit is required
PortSQL Server connection port
UsernameSQL Server connection username
PasswordPassword corresponding to the SQL Server database user
Database NameSQL Server database name, only supports single database migration
Table NameSQL Server table name to be transferred, use English commas to separate table names. Example: dbo.tablename1,dbo.tablename2, when the schema is dbo, it can be omitted, and you can fill only tablename, example: tablename1,tablename2

Target Transfer Form

Parameter NameDescription
Internal AddressKafka cluster connection address, example: 92.168.1.10:9092,192.168.1.11:9092,192.168.1.12:9093
Maximum RateKafka transfer rate limit, adjust the transfer rate
Topic PrefixUDTS migrating MySQL to Kafka will create the corresponding Topic on the target Kafka; the rule is one Topic per table, each Topic will have this parameter as a prefix
Default Partition NumberThe default number of partitions for the corresponding Topic when migrating MySQL to Kafka using UDTS

4. Kafka Transfer Data Format

The content migrated from UDTS to Kafka is in Debezium JSON format. For downstream synchronization, Flink CDC can be used, specifying the data format as debezium-json.