How I choose the cadence of an incremental pipeline

Most pipeline debates start in the wrong place. Someone asks for “near real time,” a dashboard looks empty without today’s numbers, and the schedule becomes */15 * * * *. Fifteen minutes feels responsible. It is often just expensive.

Cadence is not an orchestration preference. It is a statement about when a decision would actually change. If nobody acts on the table between runs, a faster job is mostly a cost and a source of flakiness.

Start from the decision, not the warehouse

I write down one sentence: who uses this, to do what, by when? A collections supervisor checking promised payments at 9:00 is a different consumer than a model that scores conversations as they close. The first can live on a morning batch. The second may need minutes — but only for the tables that feeding the score, not for every mart in the warehouse.

That sentence also names the grain. If the decision is “which accounts to call today,” the grain is the account for the operating day, not every event in the chat log. Event-level freshness is wasted if the product thinks in daily accounts.

Freshness has a bill

Each run pays for scan, shuffle, slot time, and — if you are not careful — full refreshes disguised as increments. On warehouses billed by bytes scanned, a 15-minute job that reads a wide raw table 96 times a day will dominate a well-partitioned daily build even when “only a little” data arrived.

I ask three cost questions before touching the scheduler:

1. What is the incremental predicate (updated_at, _partitiontime, CDC timestamp)? If it is missing, you do not have an incremental pipeline. You have a frequent full load.
2. How wide is the read? Partitioning and clustering only help if the filter matches how files are laid out.
3. What fails when a run is late? If the answer is “the dashboard looks stale until 10:00,” that is not an incident.

Measure before optimizing: log bytes processed, duration, and rows inserted for two weeks on a conservative cadence. Then argue for speed with numbers, not anxiety.

A simple ladder

I rarely jump from daily to sub-hourly. The ladder looks like this:

Daily. Default for finance-ish marts, executive reporting, and anything that is reconciled to an operating day. Idempotent, cheap to replay, easy to explain.

A few times per day. Useful when operations have a midday correction (late files, a second extract from a vendor). Still batch-shaped. Easier than true micro-batch.

Hourly. Reasonable when the product has an hourly loop — staffing, bid adjustments, same-day outreach — and the increment is cheap. Not reasonable because “hourly sounds modern.”

Every 15 minutes (or less). Reserved for a thin serving path: the tables a live system actually queries, with a tight incremental key, bounded lookback, and an owner who will get paged. Everything else stays slower.

Streaming exists, but it is a different contract: watermarks, late data, exactly-once or at-least-once semantics. I do not use a streaming platform to make a batch mart feel fashionable.

Late data is the hidden cadence problem

A 15-minute job that only reads “the last 15 minutes” will silently drop late events. Collections and messaging data are full of them: a webhook retry, a mobile client that syncs at night, a partner file that lands after the SLA.

If you need frequent updates, you also need a lookback window (replay the last N hours each run) or a true CDC cursor that never assumes arrival time equals event time. That window is part of the cadence decision. A 15-minute schedule with a 24-hour lookback is often more honest — and more expensive — than people admit. Sometimes a hourly run with a 36-hour lookback is cheaper and more correct.

Reliability beats a clever schedule

Cleverness here looks like: dynamic intervals, “smart” backfills at noon, a DAG that fans out into twenty micro-jobs so each one is small. Those designs fail in ways that are hard to explain on a Monday morning.

I prefer:

• One clear increment key.
• A documented lookback.
• A full-refresh path that is allowed to be slow.
• Alerts on data volume and freshness SLOs that match the decision, not on “job duration > 4 minutes.”

If a downstream consumer needs something faster, I would rather serve a smaller table on a faster cadence than accelerate the entire graph. Systems beat tools: Airflow, Composer, and a cron job are interchangeable compared with a wrong increment key.

A checklist I actually use

Before I set schedule_interval:

• Name the decision and the person who owns it.
• Name the grain of the output table.
• Prove the incremental filter with a query, not a comment.
• Size the lookback for late data.
• Estimate daily scans at the proposed frequency.
• Write the SLO in business time (“ready by 08:30 local,” not “runs every 15 minutes”).

If those lines are empty, the pipeline should not run every 15 minutes. It should run when we can defend it — usually slower than the first request, and much easier to trust.