What Is Data Stream Processing?

Data stream processing, commonly called stream processing, is a computing paradigm in which records are processed continuously as they arrive rather than accumulated into finite batches and processed later. The input is treated as an unbounded sequence with no defined end, so a query does not run once over a fixed dataset but stays resident and emits results incrementally as new data flows through it. This inverts the assumption behind traditional database systems, where data sits still and queries move over it.

The paradigm grew out of database research on continuous queries in the early 2000s and out of practical demands from financial trading, telecommunications monitoring, and sensor networks. A survey of the evolution of stream processing systems in the VLDB Journal traces this lineage from the first academic prototypes through the scale-out systems now in production use, and identifies the recurring design axes of state, time, and fault tolerance that separate one generation from the next.

Dataflow Graphs and Windowing

A streaming computation is expressed as a directed graph in which vertices are operators and edges are streams. Operators fall into stateless kinds, such as filters and projections, and stateful kinds, such as aggregations and joins, which must retain information between records. Because an unbounded stream has no end, aggregate operations require a way to bound the input, and that mechanism is the window.

Tumbling windows partition the stream into fixed non-overlapping intervals, sliding windows overlap by a defined step, and session windows group records separated by less than a specified gap of inactivity. Windows may be defined over time or over record counts. Choosing a window type is a modeling decision about what question is being asked, since a five-minute tumbling count and a five-minute sliding count of the same stream answer different questions.

Event Time and Out-of-Order Arrival

The hardest problem in stream processing is that records do not arrive in the order they occurred. Network delays, mobile devices reconnecting, and partitioned sources all produce late data. Systems therefore distinguish event time, the timestamp at which something happened, from processing time, the moment the system observed it. The Dataflow Model, published in the Proceedings of the VLDB Endowment in 2015, set out the abstraction now used across the field: watermarks that assert progress in event time, triggers that decide when to emit a result for a window, and accumulation modes that specify how a later revision relates to an earlier one.

The model's premise is that a system can never know that all data for a window has arrived, only that more may still come. Rather than pretending otherwise, it makes the tradeoff between completeness and latency an explicit parameter, allowing an early speculative result followed by corrections as late records land.

State, Fault Tolerance, and Delivery Guarantees

Stateful operators must survive machine failure without losing or double-counting records. Production engines take periodic distributed snapshots of operator state and stream positions, then restore from the last snapshot and replay the log on recovery, which yields effectively exactly-once results even though each record may physically be delivered more than once. Guarantees are usually stated as at-most-once, at-least-once, or exactly-once, and an analysis of end-to-end processing guarantees in streaming systems shows how far a claimed guarantee depends on the behavior of sources and sinks outside the engine. Reference architectures such as the NIST Big Data Interoperability Framework situate these components within the broader data flow from collection to consumption.

Applications

Data stream processing is used across a range of fields, including:

• Fraud detection and real-time risk scoring in payments and banking
• Network and infrastructure monitoring, including anomaly and intrusion detection
• Industrial IoT telemetry and predictive maintenance
• Algorithmic trading and market data analytics
• Clickstream analysis and online personalization
• Smart grid load monitoring and vehicle fleet telematics