Databricks makes Real-Time Mode generally available in Spark Declarative Pipelines

Databricks announced general availability of Real-Time Mode in Spark Declarative Pipelines, extending its managed pipeline framework to operational workloads such as fraud detection and live personalization. The company says the mode processes streaming data with end-to-end latency in the millisecond range. It continuously streams data, runs query stages simultaneously and passes results between stages as they are produced. Pipelines retain automatic orchestration and failure recovery, and can write to operational stores such as Lakebase and Redis through custom writers. Real-Time Mode was already available in Spark Structured Streaming. Using it in the pipeline framework requires continuous pipeline mode and a real-time update flow.