Disclaimer: I’m an Apache Flink committer and PMC member and only familiar with Storm’s high-level design, not its internals.
Apache Flink is a framework for unified stream and batch processing. Flink’s runtime natively supports both domains due to pipelined data transfers between parallel tasks which includes pipelined shuffles. Records are immediately shipped from producing tasks to receiving tasks (after being collected in a buffer for network transfer). Batch jobs can be optionally executed using blocking data transfers.
Apache Spark is a framework that also supports batch and stream processing. Flink’s batch API looks quite similar and addresses similar use cases as Spark but differs in the internals. For streaming, both systems follow very different approaches (mini-batches vs. streaming) which makes them suitable for different kinds of applications. I would say comparing Spark and Flink is valid and useful, however, Spark is not the most similar stream processing engine to Flink.
Coming to the original question, Apache Storm is a data stream processor without batch capabilities. In fact, Flink’s pipelined engine internally looks a bit similar to Storm, i.e., the interfaces of Flink’s parallel tasks are similar to Storm’s bolts. Storm and Flink have in common that they aim for low latency stream processing by pipelined data transfers. However, Flink offers a more high-level API compared to Storm. Instead of implementing the functionality of a bolts with one or more readers and collectors, Flink’s DataStream API provides functions such as Map, GroupBy, Window, and Join. A lot of this functionality must be manually implemented when using Storm. Another difference are processing semantics. Storm guarantees at-least-once processing while Flink provides exactly-once. The implementations which give these processing guarantees differ quite a bit. While Storm uses record-level acknowledgments, Flink uses a variant of the Chandy-Lamport algorithm. In a nutshell, data sources periodically inject markers into the data stream. Whenever an operator receives such a marker, it checkpoints its internal state. When a marker was received by all data sinks, the marker (and all records which have been processed before) are committed. In case of a failure, all sources operators are reset to their state when they saw the last committed marker and processing is continued. This marker-checkpoint approach is more lightweight than Storm’s record-level acknowledgments. This slide set and the corresponding talk discuss Flink’s streaming processing approach including fault tolerance, checkpointing, and state handling.
Storm also offers an exactly-once, high-level API called Trident. However, Trident is based on mini-batches and hence more similar to Spark than Flink.
Flink’s adjustable latency refers to the way that Flink sends records from one task to the other. I said before, that Flink uses pipelined data transfers and forwards records as soon as they are produced. For efficiency, these records are collected in a buffer which is sent over the network once it is full or a certain time threshold is met. This threshold controls the latency of records because it specifies the maximum amount of time that a record will stay in a buffer without being sent to the next task. However, it cannot be used to give hard guarantees about the time it takes for a record from entering to leaving a program because this also depends on the processing time within tasks and the number of network transfers among other things.