SQL Server Migration to Kafka
UDTS supports migration from SQL Server to Kafka. The supported SQL Server version is SQL Server 2008 R2 and above. Kafka supports version 2.x: including 2.0, 2.1, 2.2, 2.3, 2.4, 2.5, 2.6, 2.7, 2.8, etc.
1. Feature Limitations
1.1 Source SQL Server Limitations
- The source database and the tables to be migrated need to have the CDC feature enabled.
-- Specify the database name
use dbname
-- Enable CDC feature
exec sys.sp_cdc_enable_db
-- Check if the CDC is enabled for the dbname database, a return value of 1 means it is enabled
select is_cdc_enabled from sys.databases where name = "dbname";
-- Enable CDC for the dbo.tablename table
exec sys.sp_cdc_enable_table @source_schema = 'dbo',@source_name = 'tablename',@role_name = null;
-- Check if the CDC is enabled for the dbo.tablename table, a return value of 1 means it is enabled
select is_tracked_by_cdc from sys.tables where name = 'tablename'
-- Modify the CDC data retention time, at least set to 1440 minutes (1 day) or more, recommended 14400 minutes (7 days).
EXECUTE sys.sp_cdc_change_job @job_type = N'cleanup', @retention = 14400;
1.2 Target Kafka Limitations
- Set
auto.create.topics.enable
to true. - Set
delete.topic.enable
to true.
2. Migration Content
Migration Content | Description |
---|---|
Migration Structure | Database, Table structure, and data |
Migration Scope | Only migrates tables that can be found when creating the task, newly added tables during the task will not be automatically migrated temporarily |
DDL | CREATE, ALTER, DROP statements |
DML | Snapshot/insert/update/delete |
3. Form Filling
Data Source Form
Parameter Name | Description |
---|---|
Address | Supports two methods: internal network address and dedicated line address. Internal network address requires VPC and subnet information; dedicated line address supports both IP and domain name, if using a domain name, a user network with an external network exit is required |
Port | SQL Server connection port |
Username | SQL Server connection username |
Password | Password corresponding to the SQL Server database user |
Database Name | SQL Server database name, only supports single database migration |
Table Name | SQL Server table name to be transferred, use English commas to separate table names. Example: dbo.tablename1,dbo.tablename2 , when the schema is dbo, it can be omitted, and you can fill only tablename, example: tablename1,tablename2 |
Target Transfer Form
Parameter Name | Description |
---|---|
Internal Address | Kafka cluster connection address, example: 92.168.1.10:9092,192.168.1.11:9092,192.168.1.12:9093 |
Maximum Rate | Kafka transfer rate limit, adjust the transfer rate |
Topic Prefix | UDTS migrating MySQL to Kafka will create the corresponding Topic on the target Kafka; the rule is one Topic per table, each Topic will have this parameter as a prefix |
Default Partition Number | The default number of partitions for the corresponding Topic when migrating MySQL to Kafka using UDTS |
4. Kafka Transfer Data Format
The content migrated from UDTS to Kafka is in Debezium JSON format. For downstream synchronization, Flink CDC can be used, specifying the data format as debezium-json.