Memory hook: Time boundaries and late events decide the answer.
Must remember
Eventstream ingests, transforms and routes supported real-time events. Eventhouse provides KQL-oriented storage/query capabilities. Spark Structured Streaming offers code-based stateful processing. Select an engine from latency, state, transformation complexity, connectors and team skills.
Native Eventhouse tables ingest data into the engine; OneLake shortcuts reference external lake data. Query acceleration for supported OneLake shortcuts improves suitable query access through additional acceleration behavior and cost/refresh trade-offs. Choose after checking freshness, supported formats and performance needs; it is not identical to ordinary shortcut access.
Event time reflects when an event occurred; processing time reflects when it is handled. Tumbling windows do not overlap, hopping/sliding patterns may overlap, and session windows group activity by inactivity gaps where supported. Watermarks and late-data policies control how long state/results remain open to delayed events. Exact syntax and behavior differ between KQL, Eventstream and Spark.
KQL pipelines use operators such as where, project, extend, summarize and joins to shape event data. Apply selective filters early and choose appropriate time bins. Streaming checkpoints track progress/state for supported recovery, but external side effects still need idempotence. Monitor lag, dropped/late records and poison events rather than only whether the stream is running.
Choose under exam pressure
| Requirement | Choice and reason |
|---|---|
| Visual event routing and supported transforms | Eventstream. |
| Fast event/time-series exploration | Eventhouse and KQL. |
| Complex stateful code transformations | Spark Structured Streaming with checkpoint/recovery design. |
Traps
- A running stream can be far behind real time.
- A window result can be incomplete if late data is discarded.
Active recall
1. Event time versus processing time?
Occurrence timestamp versus handling timestamp.
2. What does a tumbling window do?
Groups events into fixed non-overlapping intervals.
3. What does a checkpoint preserve?
Supported streaming progress/state for recovery.
4. Why monitor late records?
They can materially change the accuracy of windowed results.
5. Shortcut versus native ingestion?
Reference existing external data versus ingest into the engine’s own tables.