Memory hook: DAG orders work; monitoring proves it completed correctly.
Must remember
Cloud Composer runs managed Apache Airflow for dependency-driven data workflows. A DAG describes dependencies, scheduling and retries; the actual heavy processing belongs in services such as Dataflow, BigQuery or Spark. Workflows coordinates API calls with managed execution and branching when a full Airflow environment is unnecessary.
Parameterize environments and keep code/configuration in version control. Test transformations against representative fixtures, validate schema contracts, and separate unit, integration and reconciliation tests. Use CI/CD service identities with limited access and approval appropriate to production risk. Secrets belong in managed secret storage, not DAG source or task logs.
Retries must be safe. Write to staging then publish atomically where supported, use deterministic partition targets or transaction keys, and distinguish a retry from intentional backfill. Backfills can overwhelm quotas or overwrite historical data if date boundaries are wrong. Track run IDs, source offsets and output partitions.
Monitor job success, freshness, completeness, lag, rejected records and resource use. A green task can still publish an empty or stale table. Alert on user-facing data SLOs and actionable causes. Correlate logs with workflow and pipeline IDs; restrict sensitive payloads and retention.
For cost, terminate idle clusters, select suitable worker sizes/autoscaling, tune query scans, and isolate reservations or priorities where necessary. Compare steady capacity to per-job processing, including operator effort and recovery time. Quotas, regional capacity and dependencies are part of scheduling, not afterthoughts.
Choose under exam pressure
| Requirement | Choice and reason |
|---|---|
| Complex recurring Airflow DAGs | Composer. |
| Coordinate a modest sequence of service APIs | Workflows. |
| Successful job but missing sales rows | Reconciliation/quality checks, not only process exit status. |
Traps
- The orchestrator should not carry a large dataset through task metadata.
- Blind retries can duplicate output or overwrite valid partitions.
Active recall
1. What does a DAG express?
Task dependencies and execution workflow.
2. Why keep processing outside the scheduler?
To use scalable execution engines and avoid burdening orchestration components.
3. What is a backfill?
An intentional historical reprocessing run over a defined range.
4. Name a data SLO beyond job success.
Freshness, completeness, validity or availability to consumers.
5. How make a retry safe?
Use idempotent writes, staging/atomic publication or deduplication with tracked run keys.