10.42Scheduler + watermark
Typical delta sync:
Scheduler
↓
read watermark
↓
query changed records
↓
process
↓
update watermark
Remember Module 5 concerns:
crash halfway
equal timestamps
deleted records
overlap
Often you intentionally query with overlap and rely on idempotent upserts.