Memory hook: Transform work; orchestrate dependencies.
Must remember
- Dataflow executes Beam batch/stream pipelines; Dataproc runs Spark/Hadoop workloads; Data Fusion provides visual integration; Dataform manages SQL transformation dependencies and assertions in BigQuery.
- An orchestrator coordinates work rather than replacing every processing engine. Cloud Composer provides managed Airflow; Workflows coordinates service/API steps; scheduled queries fit simple recurring SQL; Dataproc workflow templates coordinate supported cluster jobs.
- Cloud Scheduler supplies time-based triggers. Eventarc routes matching events to destinations; Pub/Sub decouples producers and consumers. A Pub/Sub BigQuery subscription can deliver supported messages without a custom transformation service.
- Make retries safe with idempotent writes, stable event identifiers and checkpoints. Route malformed records to a review path rather than silently losing them or blocking all progress.
- Monitor freshness, failed runs, input/output counts, backlog and processing latency. Dataflow job views expose pipeline progress; Cloud Logging supplies event details and Cloud Monitoring alerts on actionable symptoms.
- Separate development and production identities, configuration and data. Backfills must handle historical partitions without corrupting current output.
Choose under exam pressure
| Requirement | Choice and reason |
|---|---|
| One daily SQL transformation | A scheduled BigQuery query may be sufficient. |
| Many dependent jobs with backfills | Composer/Airflow or an appropriate orchestration workflow. |
Traps
- Cloud Scheduler does not guarantee that downstream business processing happened exactly once.
- A green job can still produce stale or empty data.
Active recall
1. When choose Dataform?
For versioned SQL transformations, dependencies and data assertions in BigQuery.
2. What makes a retry safe?
An operation that avoids duplicate side effects, often using deduplication or upsert keys.
3. How diagnose rising stream backlog?
Compare arrival and processing rates, worker utilization, errors and downstream limits.
4. What is an event-driven pipeline?
Processing initiated by an event such as object creation rather than only a fixed schedule.
5. Why monitor output counts?
To detect silent data loss or duplication that job-success status misses.