Spark 4.3 adds stateful processing to Real-Time Mode

Apache Spark 4.3 adds stateful processing to Real-Time Mode, allowing streaming applications to retain context across events. Databricks reports P99 end-to-end latency of 136 milliseconds for deduplication, 139 milliseconds for window aggregation and 141 milliseconds for custom stateful processing in three benchmarks. The release lets developers use existing Structured Streaming APIs for these workloads. Concurrent stage scheduling and incremental data transfers allow producing and consuming stages to run together, while existing state stores and checkpoints support recovery. For supported workloads, this can remove the need for a separate streaming engine to achieve millisecond-scale latency. The benchmark results do not guarantee latency for every pipeline, and each concurrently scheduled group requires enough task slots for all its tasks.