Migrate from Amazon DynamoDB to Tair Serverless KV

Use ddb-migrate to copy existing data and continuously synchronize changes, then use ddb-data-check to compare the two tables. This guide covers preparation, synchronization, validation, and application cutover.

Migration flow

1. ddb-migrate scans the source table and writes existing items to the target table.
2. After the scan, ddb-migrate continuously reads DynamoDB Streams and synchronizes inserts, updates, and deletes.
3. ddb-data-check compares the source and target item by item.
4. Stop source writes, wait for incremental synchronization to finish, and switch application traffic.

Before migration

1. Download the archive for the server operating system and architecture from GitHub Releases.
2. Create the Tair Serverless KV (DynamoDB-compatible) instance and target table. The partition key and optional sort key must have the same names and data types as the source table. Non-key attributes do not need to be declared in advance.
3. Enable DynamoDB Streams on the source table with NEW_IMAGE or NEW_AND_OLD_IMAGES.
4. Prepare AWS credentials that can read the source table. The tool supports AK/SK, AWS profiles, Web Identity, ECS task roles, and EC2 instance roles.
5. Prepare a server and verify that it can access AWS DynamoDB and the Tair endpoint.

Run ddb-migrate

Extract the archive and edit ddb-migrate.toml. The following compact configuration migrates AWS DynamoDB data to Tair:

[source]
access_key_id = "<aws-access-key-id>"
secret_access_key = "<aws-secret-access-key>"
region = "us-east-1"
table = "source_table"
checkpoint_enabled = true
checkpoint_file_path = "checkpoint.json"

[target]
access_key_id = "username:password"
secret_access_key = "dummy"
endpoint_url = "http://your-tair-endpoint:80"
table = "target_table"

Replace the source credentials, Region, table names, Tair account and password, and endpoint. The source uses the official AWS endpoints for its Region; set the target endpoint to the DynamoDB-compatible endpoint of the Tair instance. See ddb-migrate configuration for AWS authentication methods, target endpoint behavior, checkpoints, and concurrency options.

Start the migration:

./ddb-migrate ddb-migrate.toml

Scan in the log means that existing data is being copied. Stream means that incremental changes are being synchronized.

Validate data

Wait until ddb-migrate is stable in the Stream phase and its synchronization QPS is close to the source application write QPS. Then edit ddb-data-check.toml:

check_mode = "source"

[source]
access_key_id = "<aws-access-key-id>"
secret_access_key = "<aws-secret-access-key>"
region = "us-east-1"
endpoint_url = ""
table = "source_table"

[target]
access_key_id = "username:password"
secret_access_key = "dummy"
endpoint_url = "http://your-tair-endpoint:80"
table = "target_table"

Both files can reuse the source credentials, Region, and table name, along with the Tair account and password, endpoint, and table name. See ddb-data-check configuration for check directions and credential settings.

Run the check and generate an HTML report:

./ddb-data-check ddb-data-check.toml
python3 analyze_report.py ddb-data-check-diff/report.json

check_mode = "source" verifies that each source item exists and matches on the target. After it finishes, change the value to target and run the command again to detect target-only items.

Switch applications

1. Stop all source writes during the planned cutover window.
2. Continue watching the ddb-migrate log until Stream synchronization QPS drops to and remains at zero.
3. Point the applications to the new Tair instance and restart them.

Troubleshooting

Stream or checkpoint errors at startup

Before moving data, the program reads the source table definition and verifies that DynamoDB Streams is enabled with NEW_IMAGE or NEW_AND_OLD_IMAGES. If the log says that Streams is missing or disabled, enable it in the AWS console. A newly enabled Stream can take several minutes to appear in the table definition. A wrong view type must be changed; waiting does not change that error.

The checkpoint records the source table name, Stream ARN, save time, Scan progress, and each shard position. The old checkpoint cannot be used when:

• It belongs to another source table or Stream.
• Its last save is older than the 23-hour safety window.
• The log reports a sequence number outside the 24-hour retention period, trimmed data, or a parent shard that disappeared before completion.

These errors mean that incremental continuity can no longer be verified. Stop the process, confirm that a new full migration is required, delete checkpoint.json, and start again with the same configuration. Deleting the file discards all existing Scan and Stream progress.

Target connection, schema, or write failures

At startup, the program calls DescribeTable on the target. If this step fails, check in order:

1. endpoint_url includes the correct scheme, host, and port.
2. Tair access_key_id is account:password and secret_access_key is dummy.
3. The server IP is on the target allowlist and the endpoint is reachable from the server.
4. The target table exists and its partition key and optional sort key have the same names and data types as the source.

When destination throttling appears during migration, the writer keeps the pending batch and retries with backoff for up to 30 minutes. Increasing workers adds more target pressure; raise target write capacity or lower workers. A BatchWriteItem ... failed after AWS SDK retries message is a non-throttling failure. Follow the permission, request, or validation error included in that log entry.

After fixing the target, restart with the same configuration and checkpoint to resume. Delete the checkpoint only when the log explicitly requires a new full migration.

Full Scan is slow

The status log reports the Scan read rate, Write rate, and backlog together:

• backlog stays near zero while the Scan rate is low: the source read side is limiting throughput. After confirming that source RCU is available, raise scan_segments. Scan uses strongly consistent reads, so more parallelism increases source read requests and RCU consumption.
• backlog keeps growing: target writes are slower than source reads. Raise workers only when there is no throttling log. When destination throttling appears, raise target write capacity or lower workers.

Changing scan_segments prevents an incomplete old Scan checkpoint from being resumed, so the source table is scanned from the beginning. Change it only when the current full-scan progress can be discarded. When the full phase approaches the Stream retention period, inspect source read capacity, target write capacity, and backlog together; raising read parallelism alone can move the delay into the writer.