5.18Be careful with watermarks

Suppose your job runs:

02:00

retrieves records through:

02:07

and then crashes.

If you blindly save:

02:07

as the watermark before completing, you could skip records.

So:

query range
↓
process successfully
↓
only then advance checkpoint

Or use overlapping windows plus idempotent upsert.

QA thinking helps enormously here.