Memory hook: Govern the object; checkpoint the load; inspect the stage.
Reviewed 10 October 2026. Read this once, then answer the last-pass checks without looking.
Scope/version: The official English skills update begins 19 October 2026. These notes use that published upcoming outline; check your booked exam date and the service’s current naming.
Must remember by domain
| Domain | Rapid revision |
|---|---|
| Environment | Job compute suits automated runs; interactive compute supports development; SQL warehouses serve SQL/BI; serverless reduces supported infrastructure management. Configure runtime, libraries, Photon, scale/termination and access mode deliberately. Photon does not fix arbitrary Python UDFs or severe data skew. |
| Unity Catalog | catalog.schema.object organizes governed assets. Tables hold/query data, volumes hold governed files, views store logic, materialized views persist supported results. Managed tables delegate storage lifecycle; external tables reference externally managed locations. Foreign catalogs expose supported remote systems; Genie instructions clarify trusted definitions, not permissions. |
| Security/governance | USE CATALOG and USE SCHEMA plus object privileges are needed for appropriate access. Compute permission is separate. Use groups/service principals, governed storage credentials/locations and managed identity. Row filters, column masks and ABAC policies constrain supported access. Catalog lineage/history/audit helps impact analysis; Delta Sharing governs recipients without recalling already exported copies. |
| Modeling/ingestion | Bronze retains raw; silver validates/conforms; gold serves business consumption. Define grain and SCD1 overwrite versus SCD2 history. Auto Loader discovers files incrementally with checkpoint/schema state; COPY INTO handles supported batch file loads; CDC applies ordered changes; Event Hubs supplies streams. Delta adds a transaction log to data files. |
| Transformation/quality | Joins enrich columns; union/append stacks rows; intersect/except compare sets. Check duplicate keys before MERGE. Schema enforcement rejects incompatible data; deliberate evolution accepts supported changes. Declarative Pipeline expectations can retain, drop or fail violating records according to policy. Liquid clustering and partition/Z-order choices are not blindly cumulative. |
| Delivery/orchestration | Lakeflow Jobs coordinates tasks, dependencies, parameters, schedules/events, retries and notifications. Declarative pipelines manage supported transformation dependencies. Git/reviews/tests plus Declarative Automation Bundles promote versioned definitions to environments. Repair only failed tasks when retained upstream results and side effects remain consistent. |
| Operations | Spark UI/query profile reveals stages, shuffle, skew, spill and cache effects. Broadcast only genuinely small data. OPTIMIZE reorganizes/compacts supported tables; VACUUM removes obsolete files and affects time travel/readers. Tune layout/joins/parallelism before increasing compute; monitor freshness and data quality with cost. |
Diagnostic sequence and traps
Job/task error → actual input/version → credential/location privileges → schema/quality → checkpoint/progress → Spark stage or query plan → capacity. Deleting checkpoints can cause replay. Dropping an external table definition does not imply its underlying files have been removed. Do not use aggressive VACUUM retention merely to free space during an active workload.
Last-pass self-check
1. Can a user attach to compute and therefore read every table?
No. Unity Catalog/data privileges still apply.
2. One Spark task dominates runtime: likely investigation?
Skew, partition size, shuffle/spill and the relevant join or aggregation stage.
3. Why deduplicate CDC before MERGE?
Multiple unordered changes for a key can produce ambiguous or incorrect updates.
4. Which operation threatens old time-travel data?
VACUUM removes obsolete physical files according to retention.
5. What proves a repaired job is correct?
Reconciled data and idempotent side effects, not just a successful final task.
Sources
- Official exam scope and version
- Product documentation
- Product documentation
- Product documentation
- Product documentation
- Product documentation
Every topic at a glance
Open any topic to revisit its essential facts, decisions and exam traps. Use the full topic for active recall and supporting references.
01 · Compute and Unity Catalog organization
Memory hook: Workspace runs work; catalog governs data.
Must remember
- Choose interactive/classic compute for suitable development, job compute for automated work, SQL warehouses for SQL/BI and supported serverless compute to reduce infrastructure management. Compare isolation, startup time, cost and feature support.
- Configure runtime/Spark version, node type/count, autoscaling, termination and pools to match the workload. Photon accelerates supported execution; it is not a cure for every skewed query or Python UDF bottleneck.
- Install compatible libraries and manage compute permissions separately from data privileges. A user able to attach to compute should not automatically read every catalog.
- Unity Catalog organizes catalog.schema.object names. Tables store/query data; volumes govern files; views define queries; materialized views persist supported results with refresh behavior.
- Managed tables delegate more storage lifecycle to the platform; external tables reference externally managed locations. Foreign catalogs expose supported external systems through governed connections.
- Use naming/environment boundaries, comments and ownership to make assets discoverable. AI/BI Genie instructions can explain trusted metrics and context, but do not replace correct permissions or semantic definitions.
Choose under exam pressure
| Requirement | Choice and reason |
|---|---|
| Scheduled production transformation | A job-oriented/serverless option with a pinned supported runtime and scoped identity. |
| Govern non-tabular files | Unity Catalog volumes with appropriate privileges and storage credentials. |
Traps
- A workspace is not the same security object as a catalog.
- An external table does not automatically mean unrestricted access to its storage.
02 · Security, lineage and sharing
Memory hook: Grant access to the object and its path.
Must remember
- Grant Unity Catalog privileges to groups or service principals at the required object scope. Ownership, USE CATALOG/SCHEMA and object privileges serve different purposes.
- Use row filters and column masks for supported fine-grained controls. ABAC tags and policies can centralize supported classification-based rules; test effective results for representative identities.
- Storage credentials/external locations and managed identities provide governed storage access. Service principals suit automation; use Key Vault-backed or supported secret handling without printing secret values.
- Capture table/column descriptions, lineage, history and dependencies in Catalog Explorer. Lineage helps impact analysis, but coverage depends on supported workloads and recorded operations.
- Audit administrative and data activity according to requirements, with controlled log access and retention. Data-retention policy must coordinate table history, underlying files and legal requirements.
- Delta Sharing provides governed data sharing across supported recipients. Scope shared objects, recipient authentication and revocation; a data export or recipient copy may outlive future access removal.
Choose under exam pressure
| Requirement | Choice and reason |
|---|---|
| Analysts may see only their region | A tested row filter with identity-aware policy. |
| Production jobs access Azure storage | A scoped workload identity and governed storage location, not an embedded account key. |
Traps
- Masking a column is not the same as deleting or encrypting the underlying value.
- Revoking future sharing does not erase copies already lawfully exported.
03 · Ingest and model Delta data
Memory hook: Raw to trusted, with replayable progress.
Must remember
- Use bronze/silver/gold as a useful layering pattern: retain raw evidence, validate/conform it, then publish business-ready data. Choose grain and SCD behavior from the required history.
- Delta adds transaction-log semantics to data files; Parquet is a columnar format; CSV/JSON are interchange formats; Iceberg is another table format with its own support considerations. Choose compatibility intentionally.
- Lakeflow Connect, notebooks, Data Factory, SQL COPY INTO/CTAS and supported connectors solve different ingestion needs. Batch suits bounded arrivals; Structured Streaming suits incremental unbounded processing.
- Auto Loader incrementally discovers supported files with checkpoints/schema handling. Event Hubs can feed streaming pipelines through supported interfaces; configure authentication, offsets and consumer behavior.
- CDC and MERGE support incremental updates when keys and ordering are correct. Deduplicate source changes and handle late/out-of-order records so one key is not matched ambiguously.
- Partitioning, Z-ordering and liquid clustering optimize different layouts; use supported combinations rather than piling every technique onto a small table. Managed versus external storage determines lifecycle ownership.
Choose under exam pressure
| Requirement | Choice and reason |
|---|---|
| Files continuously arrive in cloud storage | Auto Loader with durable checkpoints and deliberate schema evolution. |
| Need to preserve changing dimension history | SCD Type 2 with correct effective periods and change ordering. |
Traps
- Streaming is not automatically lower cost than a frequent batch job.
- Partitioning by a very high-cardinality field can create excessive small files.
04 · Transform, validate and orchestrate
Memory hook: Quality gates before dependent jobs.
Must remember
- Profile distributions, nulls, duplicates and types before transformations. Join, union, intersect and except have different semantics; validate row counts after joins to detect accidental multiplication.
- Use filters, groups, aggregates, pivots/unpivots and denormalization according to the target grain. Choose merge/insert/append based on update semantics rather than convenience.
- Schema enforcement rejects incompatible writes; schema evolution accepts supported planned changes. Uncontrolled drift can silently change analytics meaning even when the pipeline runs.
- Lakeflow Spark Declarative Pipelines express managed transformations and expectations. Configure whether failed expectations are retained, dropped or fail processing according to supported behavior and business policy.
- Lakeflow Jobs orchestrates tasks with dependencies, parameters, triggers/schedules, retries and notifications. A notebook task and a declarative pipeline have different lifecycle and dependency management models.
- Design failure handling, checkpoints and repair strategy before production. Retrying only the failed task is safe only when upstream/downstream side effects and data versions remain consistent.
Choose under exam pressure
| Requirement | Choice and reason |
|---|---|
| Invalid records should be inspected without blocking valid ones | A deliberate quarantine/expectation policy with visible quality metrics. |
| Several dependent transformations run nightly | Lakeflow Jobs with explicit dependencies and failure notifications. |
Traps
- A successful Spark job can still duplicate every business measure through a bad join.
- Automatically accepting every new column is not a governance strategy.
05 · Deploy and optimize production workloads
Memory hook: Version the job; inspect the expensive stage.
Must remember
- Use Git branches, reviews and tests for notebooks/code/configuration. Unit, integration, end-to-end and user-acceptance tests validate different layers; include data correctness and access boundaries.
- Declarative Automation Bundles package supported jobs/pipelines and environment targets for repeatable deployment. Validate, deploy and run through supported CLI/API workflows with scoped service identities.
- Inspect Spark UI, DAG/stage metrics and query profiles to distinguish shuffle, skew, spill, cache misses and insufficient parallelism. One slow skewed partition can dominate an otherwise idle cluster.
- Tune joins, data layout, partition count and compute only after measuring. Broadcast joins suit genuinely small inputs; excessive caching can consume memory and increase spill.
- OPTIMIZE compacts/reorganizes supported Delta layouts; VACUUM removes obsolete files according to retention and safety constraints. Aggressive retention can break time travel or running readers.
- Monitor cluster consumption, job duration, failures and cost. Use supported Log Analytics integration and Azure Monitor alerts; repair/restart jobs only with an understanding of checkpoints and side effects.
Choose under exam pressure
| Requirement | Choice and reason |
|---|---|
| One Spark task runs much longer than others | Investigate skew and partition distribution. |
| Many tiny Delta files slow reads | Evaluate OPTIMIZE and better write patterns before adding nodes. |
Traps
- VACUUM is not a generic performance command with no recovery consequences.
- Restarting a cluster does not fix a deterministic bad join or data-quality error.