How it works

ddb-migrate has a full Scan phase and an incremental Stream phase. Both phases use the same target writer and checkpoint.

Startup checks

The command reads the source and target definitions and validates the names and data types of their partition keys and optional sort keys. DynamoDB Streams must be enabled on the source with a view type that includes NewImage.

Connecting the full and incremental phases

On a fresh run, the command records the current Stream shard positions before starting the full Scan. It continues monitoring shards during the Scan so the incremental starting position remains usable.

Record Stream starting positions
        ↓
Scan the source table in parallel
        ↓
Wait for every full-copy write acknowledgment
        ↓
Consume DynamoDB Streams from the saved positions

Full Scan

The source table is scanned in scan_segments parallel segments with strongly consistent reads. Each item is sent to the writer.

A segment checkpoint advances only after every item in the current Scan page has been written and acknowledged. A restart before the checkpoint update reads that page again.

Target writes

The router sends the same primary key to the same worker to preserve per-key write order. A worker sends up to 25 operations through BatchWriteItem.

INSERT, MODIFY, and Scan items become PutRequest; REMOVE becomes DeleteRequest. The writer retries target throttling with backoff and exits on non-throttling failures.

Incremental synchronization

After the full copy, the command consumes DynamoDB Streams by shard. A child shard waits for its parent to finish.

The sequence checkpoint advances only after every record in the current batch has been written and acknowledged. A failed record leaves the batch position unchanged.

Validation and application cutover

1. Wait until the command is stable in the Stream phase and its synchronization QPS is close to the source write QPS.
2. Run ddb-data-check for a full consistency check.
3. Stop all source writes during the cutover window.
4. Wait until Stream synchronization QPS drops to and remains at zero.
5. Point the applications to the new Tair instance and restart them.