Stream Processing at Scale: Apache Flink 2.0 vs. RisingWave vs. Kafka Streams Architecture & Benchmarks

# Stream Processing Apache Flink vs RisingWave: Architecture, Benchmarks, and Failover at Scale

Summary: Evaluating stream processing apache flink vs risingwave and Kafka Streams highlights an architectural shift from local-disk state engines to disaggregated cloud storage. Flink 2.0 introduces ForSt to decouple RocksDB state into remote object storage with async execution. RisingWave eliminates JVM overhead via Rust and its Hummock engine, maintaining real-time materialized views queryable over PostgreSQL wire protocol. Kafka Streams retains embedded partition-local state, excelling for lightweight microservices but facing long MTTR during heavy state rebuilds.

---

The Disaggregated State Paradigm Shift in Modern Stream Processing

Real-time stream processing architectures have reached an inflection point. For over a decade, the standard distributed streaming blueprint paired stream compute workers with local storage engines. Apache Flink paired TaskManagers with embedded RocksDB instances writing to local NVMe drives. Kafka Streams embedded RocksDB directly inside Java Virtual Machine (JVM) application instances, replicating state updates to Apache Kafka changelog topics.

While this partition-local storage design delivers low single-digit microsecond read and write access during steady-state processing, it introduces severe operational bottlenecks when state scales beyond hundreds of gigabytes:

1. Coupled Compute and Storage Scaling: To expand storage capacity for large stateful join windows, operators must provision additional compute workers (TaskManagers or Kafka Streams instances) even when CPU utilization remains low.

2. Prolonged Failover and Rescaling MTTR: When a compute worker dies or a cluster autoscales, the replacement node must transfer tens or hundreds of gigabytes of state over the network before processing can resume. Mean Time to Recovery (MTTR) regularly stretches from minutes to hours.

3. Local I/O and Garbage Collection Spikes: Background LSM-tree compactions inside local RocksDB instances contend directly with stream execution threads for CPU cycles, memory allocations, and NVMe disk bandwidth, creating erratic tail latencies.

To resolve these structural bottlenecks, the streaming ecosystem has bifurcated into two primary architectural evolutions: disaggregated state engines for established compute platforms (exemplified by Apache Flink 2.0 and its ForSt storage engine) and Rust-native streaming databases that treat cloud object storage as primary durable storage (exemplified by RisingWave).

Understanding the trade-offs between stream processing apache flink vs risingwave and Kafka Streams requires analyzing their internal storage engines, stateful join latencies under heavy load, failover recovery dynamics, and query-serving boundaries.

---

Architectural Deep Dive: Three Storage and Execution Models

The fundamental differentiator between Flink 2.0, RisingWave, and Kafka Streams lies in how each engine manages state ownership, data distribution, and fault tolerance boundaries.

Architectural Comparison Topology

The following diagram contrasts how state flows through compute workers, local caches, and durable storage across the three architectures:

flowchart TD
    subgraph Flink2["Apache Flink 2.0 (ForSt Engine)"]
        subgraph Flink_TM["TaskManager Pod"]
            AEC["Asynchronous Execution Controller"]
            TaskWorker["Stream Operator DAG"]
            ForStMem["ForSt MemTable (RAM)"]
            LocalCache["Local NVMe Read Cache"]
            AEC --> TaskWorker
            TaskWorker <--> ForStMem
            TaskWorker <--> LocalCache
        end
        UFS["Unified File System (UFS)"]
        Flink_S3["Remote Object Storage (S3 / MinIO)\nPrimary State & Checkpoints"]
        ForStMem -.->|Async Flush / Compaction| UFS
        UFS <--> Flink_S3
        LocalCache -.->|Cache Miss Fetch| UFS
    end

    subgraph RW["RisingWave (Rust-Native Streaming Database)"]
        subgraph RW_Compute["Stateless Compute Node"]
            RW_Actor["Streaming Actors (Graph)"]
            RW_SharedBuf["Shared MemTable Buffer"]
            FoyerCache["Foyer Tiered Cache (RAM + NVMe)"]
            RW_Actor <--> RW_SharedBuf
            RW_Actor <--> FoyerCache
        end
        subgraph RW_Compactor["Dedicated Compactor Nodes"]
            CompactorService["Compactor Service"]
        end
        RW_S3["Cloud Object Storage (S3 / GCS)\nHummock LSM SSTables"]
        RW_Meta["Meta Service (Raft / Epochs)"]
        
        RW_SharedBuf -.->|Epoch Barrier Flush| RW_S3
        CompactorService <-->|Background Merging| RW_S3
        FoyerCache -.->|Cache Miss Fetch| RW_S3
        RW_Meta -.->|Epoch Commit / Fencing| RW_Actor
    end

    subgraph KStreams["Apache Kafka Streams (Embedded Engine)"]
        subgraph JVM_App["JVM Application Container"]
            StreamThread["StreamThread / Processor Node"]
            LocalRocksDB["Embedded RocksDB (Local Disk)"]
            StreamThread <--> LocalRocksDB
        end
        subgraph KafkaBroker["Kafka Cluster"]
            ChangelogTopic["Changelog Topic (Compacted)"]
            DataTopic["Source / Sink Topics"]
        end
        StreamThread <--> DataTopic
        LocalRocksDB -.->|Changelog Producer Writes| ChangelogTopic
    end

---

Apache Flink 2.0: Disaggregated State Storage with ForSt

For more than a decade, Apache Flink relied on the EmbeddedRocksDBStateBackend. Flink 2.0 introduces disaggregated state storage forst flink architecture (standardized under FLIP-427 and FLIP-388), redesigning Flink's storage layer for disaggregated cloud environments.

The Unified File System (UFS) Layer:

In Flink 1.x, TaskManagers wrote SSTable files to local directories and copied them asynchronously to durable checkpoint storage (e.g., S3 or HDFS) during checkpoint barriers. This created duplicate file I/O and forced recovery routines to pull entire SSTables back down to local disks.

ForSt introduces the Unified File System (UFS). In this model:

• The remote distributed file system is the primary home for durable state data.
• Working state files and checkpoint snapshots exist within the same storage namespace.
• Checkpointing no longer requires copying data across network boundaries; it registers logical file links and reference counts directly with the JobManager.
• Local NVMe storage operates strictly as a read-through secondary cache.

Asynchronous Execution Controller (AEC):

Accessing remote cloud object storage (such as Amazon S3, Google Cloud Storage, or MinIO) via standard HTTPS REST APIs inherently incurs network round-trip latencies—typically ranging from 10 ms to 50 ms for small block requests depending on payload size, TLS negotiation overhead, and physical availability zone routing—compared to sub-millisecond access times for local NVMe SSD lookups. If Flink retained its traditional synchronous state access model, operator throughput would collapse.

Flink 2.0 resolves this through an Asynchronous Execution Controller (AEC). When a stream record requires a state lookup:

1. The operator issues an asynchronous state call (asyncState.get()).

2. The AEC yields execution to other incoming events while the I/O request is in flight.

3. The AEC maintains an internal dependency graph to guarantee per-key event ordering, watermark progression, and deterministic record execution without thread blocking.

# Production flink-conf.yaml for Flink 2.0 ForSt Engine
state.backend.type: forst
state.backend.forst.primary-dir: s3://telemetry-flink-checkpoints/states/
execution.async-state.enabled: true
execution.async-state.max-in-flight-records: 8192

# Tiered local caching configuration
state.backend.forst.cache.dir: /mnt/nvme0n1/flink-forst-cache
state.backend.forst.cache.size-ratio: 0.65
state.backend.forst.cache.reserve-disk-size: 10GB

# Checkpoint intervals enabled by instant metadata linking
execution.checkpointing.interval: 2000
execution.checkpointing.mode: EXACTLY_ONCE
execution.checkpointing.min-pause: 1000
execution.checkpointing.unaligned.enabled: true

---

RisingWave: Rust-Native Streaming DAG and the Hummock LSM Engine

RisingWave approaches stream processing not as an iterative framework on top of a JVM runtime, but as a cloud-native streaming database engineered entirely in Rust.

Decoupled Compute, Compaction, and Metadata:

RisingWave explicitly separates cluster responsibilities into three isolated tiers:

• Compute Nodes: Stateless workers executing dataflow graphs (actors). Compute nodes buffer writes in a shared in-memory buffer and serve point-lookups and range scans.
• Compactor Nodes: Independent worker processes that pull SSTables from object storage, execute LSM compaction routines (merging levels, purging expired TTL tombstones), and push compacted SSTables back to object storage. This ensures compaction I/O never steals CPU cycles from active streaming join operators.
• Meta Service: A fault-tolerant Raft-backed coordinator managing table schemas, data distribution, and barrier injection.

The Hummock Storage Engine:

Instead of embedding a general-purpose key-value store, RisingWave features Hummock, a storage engine built specifically for cloud object storage.

• Epoch Barrier Injection: Every 1 to 2 seconds, the meta service injects barrier events into source connectors (Kafka, Kinesis, Pulsar).
• Zero-Copy Checkpointing: When compute actors encounter a barrier, they flush their in-memory write buffers as immutable SSTables directly to cloud object storage (S3/GCS/MinIO) and send a commit message with SSTable metadata back to the meta service.
• Foyer Hybrid Caching: To minimize object storage read latency during complex stateful joins, RisingWave utilizes the Foyer caching library. Foyer manages an in-memory block cache coupled with an asynchronous, direct-I/O NVMe disk cache. Read requests hit RAM first, fall back to local NVMe, and fetch from remote object storage only on a total cache miss.

---

Kafka Streams: Embedded Partition-Local RocksDB with Changelog Backing

Kafka Streams operates as an embedded client library rather than an independent distributed cluster manager. It executes directly inside application processes, typically deployed on Kubernetes pods or virtual machines.

Embedded Execution Model:

Each Kafka Streams instance creates one or more StreamThread instances based on input topic partition assignments:

• Partition Coupling: Processing parallelism is strictly bounded by the partition count of the source topics.
• Local Storage: In standard configurations, every stateful operator (such as a windowed aggregate or stream-table join) initializes an embedded C++ instance of the kafka streams state store rocksdb engine via the Java Native Interface (JNI) on the host container's local file system.
• Changelog Topic Synchronization: To survive container restarts, every modification to a local RocksDB state store is converted into a change record and produced to an internal, compacted Kafka topic (e.g., app-id-store-changelog).

The Standby Replica Tax:

Because Kafka Streams lacks a native remote object storage backend, recovering a failed node without hot standbys requires replaying the entire changelog topic from Kafka brokers to rebuild the local RocksDB database. To reduce this downtime, Kafka Streams provides standby tasks (num.standby.replicas):

• Standby tasks continuously consume changelog topics on healthy instances to maintain hot backup copies of the RocksDB state.
• Resource Overhead: Configuring standby replicas multiplies the cluster's storage footprint and RAM consumption by $1 + N$ (where $N$ is the standby count), substantially increasing operational infrastructure costs.

---

Standardized Workload Analysis: Evaluating NEXMARK Query Patterns and Scaling Dynamics

Evaluating stream processing performance across complex event workflows requires standardized workload patterns that isolate state management bottlenecks, join amplification, and compaction contention. The industry-standard nexmark stream processing benchmark 2026 suite provides a battle-tested framework for analyzing how streaming engines behave under diverse stateful topologies—from continuous stream-table enrichments to high-cardinality multi-stream temporal joins.

NEXMARK Query Topologies and Architectural Stress Vectors

To understand performance trade-offs, streaming architects evaluate four key NEXMARK query patterns:

1. NEXMARK Q3 (Stateful Stream-Table Join):

- Topology: Joins an unbounded stream of bids with a continuously updated customer account table.

- Stress Vector: Stresses point-lookup latency and read-cache hit rates under dynamic key updates.

2. NEXMARK Q7 (Highest-Bid Tumbling Window):

- Topology: Computes the maximum bid over a fixed 60-second window across high-volume auction records.

- Stress Vector: Tests bulk state eviction and checkpoint barrier alignment during window retirement.

3. NEXMARK Q8 (Author-Bid Sliding Window):

- Topology: Evaluates bids within a 300-second window advancing every 10 seconds, correlating new persons with incoming bids.

- Stress Vector: Generates high state churn, memory buffer pressure, and frequent LSM-tree write amplification.

4. Extended Multi-Stream Temporal Joins (10-Stream Co-partitioned Join):

- Topology: Continuously correlates 10 distinct telemetry event streams on a common entity key across independent time windows.

- Stress Vector: Tests multi-state store coordination, off-heap memory management, and LSM compaction thread isolation under sustained write pressure.

---

Architectural Workload Evaluation Matrix (NEXMARK Patterns)

The following matrix contrasts how Apache Flink 2.0 (ForSt), RisingWave (Hummock), and Apache Kafka Streams (RocksDB) execute these standardized query workloads:

---

Deep-Dive Analysis: Stateful Stream Joins Latency and Resource Contention

The structural differences between these three engines manifest most prominently in stateful stream joins latency when joining multiple high-velocity streams:

1. RisingWave's Vectorized Rust Execution and Compactor Isolation:

In high-cardinality temporal joins, RisingWave achieves superior tail-latency stability primarily due to its process isolation model. In traditional embedded architectures, when streaming joins generate millions of state records, the underlying LSM-tree triggers multi-level compactions. These compactions saturate local disk bandwidth and consume CPU cores, starving active stream processing threads. RisingWave eliminates this contention by routing all compaction workloads to dedicated Compactor nodes. Compute nodes remain dedicated to streaming actors and in-memory view updates. Furthermore, because RisingWave is written in Rust, it completely avoids JVM garbage collection pauses, which frequently disrupt real-time streaming pipelines.

2. Flink 2.0 ForSt Tail Latency Stabilization via FLIP-427 and FLIP-388:

In legacy Flink 1.x architectures utilizing EmbeddedRocksDBStateBackend, checkpointing required synchronous memtable flushing and local SSTable generation before initiating asynchronous upload. Under high-throughput stateful joins, this synchronous phase frequently caused checkpoint barrier alignment stalls, leading to pronounced P99 latency degradation. Flink 2.0 with ForSt addresses this directly: by decoupling durable state to remote object storage and utilizing the Asynchronous Execution Controller (AEC, detailed in FLIP-388 and FLIP-427), state writes and checkpoint references occur non-blockingly. This architectural separation stabilizes tail latencies and mitigates the severe P99 latency spikes historically caused by local snapshot synchronization.

3. Kafka Streams Contention Under Complex Multi-Way Joins:

Kafka Streams executes stateful operations by embedding RocksDB inside the client application JVM. When an application performs multi-stream temporal joins across multiple state stores, each store allocates its own memtables and block caches. This design encounters two structural bottlenecks under high event rates:

- JNI Serialization Overhead: Every read and write to state requires crossing the Java Native Interface boundary between the JVM heap and the native C++ RocksDB runtime, incurring CPU serialization overhead.

- Local I/O and Cache Contention: As multiple RocksDB instances compete for host OS page cache and disk I/O during concurrent window expirations, disk write amplification can cause write stalls, inflating tail latencies.

---

State Recovery and Failover Dynamics: Stream Processing MTTR Failover Analysis

Mean Time to Recovery (MTTR) is a critical operational SLA in distributed streaming systems. When a worker node experiences hardware failure or Kubernetes pod termination under a 100 GB+ state workload, the duration of pipeline downtime directly determines data lag and recovery overhead. Understanding stream processing mttr failover requires analyzing the distinct recovery stages each engine traverses.

Failover Recovery Architecture and Stage-by-Stage Mechanics

When an active worker node crashes, recovery proceeds through five sequential phases:

Failover Recovery Architecture: Disaggregated Storage vs. Embedded Changelog

[ Phase 1: Detection ]
Coordinator detects node failure (Heartbeat timeout / Raft lease expiration)
      │
      ▼
[ Phase 2: Task Reassignment ]
Coordinator reassigns partitions / actors to surviving or newly provisioned workers
      │
      ▼
[ Phase 3: State Acquisition ]
  ├─ Disaggregated (RisingWave / Flink ForSt): 0 bytes copied; link remote SST pointers via UFS/Hummock
  └─ Embedded Changelog (Kafka Streams): Stream entire state from Kafka brokers over network to disk
      │
      ▼
[ Phase 4: State Verification & Opening ]
Verify LSM manifests, open SST file handles, initialize in-memory indexes
      │
      ▼
[ Phase 5: Stream Catch-Up ]
Consume backlog of messages accumulated during downtime; achieve steady-state processing

---

Failover Timeline and Recovery Mechanics Breakdown

The table below details the architectural mechanics and typical recovery durations governing each phase:

---

Architectural MTTR Evaluation: Remote Object Store vs. Changelog Replay

The fundamental divergence in recovery performance originates in how state data is acquired by the replacement worker:

1. RisingWave Sub-10 Second Resumption:

In RisingWave, compute nodes operate statelessly. Because all committed streaming state is persisted as immutable SSTables in cloud object storage, the Meta Service simply reassigns the unassigned actor shards to healthy compute nodes. The new compute worker does not pull state files down from object storage during initialization. Instead, it registers the current epoch manifest and begins processing immediately. Blocks are fetched asynchronously into the Foyer cache on demand.

2. Flink 2.0 ForSt Elimination of Bulk Downloads:

In Flink 1.x, recovering a 100 GB+ stateful operator required the replacement TaskManager to download the entire snapshot file from remote storage (S3 or HDFS) to local NVMe, decompress it, and open RocksDB. Transferring 100 GB over a congested network frequently required several minutes. With ForSt, the new TaskManager uses UFS to link directly to remote SSTable references. Processing resumes as soon as the metadata registration completes (typically within 15 to 45 seconds), while local NVMe caches populate organically during runtime.

3. Kafka Streams Cold Replay Mathematical Model:

When an application instance running Kafka Streams suffers an unrecoverable failure without standby tasks enabled, the replacement container must rebuild its embedded RocksDB state store from scratch by replaying the partition's changelog topic from Kafka brokers.

Although internal datacenter networks often provide 10 Gbps or higher bandwidth, end-to-end changelog replay throughput is strictly bounded by broker egress I/O, network deserialization, JNI interface crossings, and RocksDB write pipeline processing (including memtable insertion, WAL logging, and background SST flushes). In production environments, sustained single-task changelog restoration typically operates at 200 MB/s to 300 MB/s. Rebuilding a 120 GB state store under these physical constraints requires an extended downtime:

$$\text{Restoration Time} \approx \frac{120{,}000\text{ MB}}{250\text{ MB/s}} = 480\text{ seconds } (\sim 8\text{ minutes})$$

During this multi-minute restoration window, stream processing on that partition is entirely stalled, and downstream consumers experience escalating data lag. While configuring num.standby.replicas: 1 brings failover time down to the range of 20 to 60 seconds, it doubles the cluster's storage, memory, and CPU overhead sitewide.

---

Unified Serving: Real-Time Materialized Views Over PostgreSQL Wire Protocol

Architectural complexity in real-time streaming architectures often concentrates at the boundary between the stream processor and the user-facing serving database.

Eliminating the Dual-Engine Architecture

In traditional stream processing architectures, stream compute engines (Flink or Kafka Streams) evaluate window aggregations and write materialized results to an intermediary sink topic or connector, which is subsequently ingested into an external analytical database (such as ClickHouse, Apache Pinot, or PostgreSQL).

[Kafka Ingress] 
      │
      ▼
[Flink / Kafka Streams Engine] (Window aggregations & stateful joins)
      │
      ▼
[Kafka Sink / Debezium CDC Connector]
      │
      ▼
[Downstream Serving DB: ClickHouse / PostgreSQL / Redis]
      │
      ▼
[Backend Application API Service]

This dual-engine topology requires maintaining separate distributed clusters, synchronizing schema changes across boundaries, and tolerating cumulative end-to-end data freshness latencies. This end-to-end delay compounds from stream processor commit intervals (e.g., exactly-once two-phase commit transaction boundaries), connector polling intervals, intermediate network hops, and downstream database indexing and block-merging cycles—frequently resulting in end-to-end data lag ranging from several hundred milliseconds to multiple seconds before updates become visible to end-user queries.

RisingWave eliminates this downstream boundary by unifying stream computation with native real time materialized views postgres wire compatibility. External client applications query live materialized views directly using standard SQL.

[Kafka Ingress]
      │
      ▼
[RisingWave Streaming Engine] (Continuous Materialized Views via Hummock)
      │
      ▼ (PostgreSQL Wire Protocol / port 4566)
[Backend Application API Service] (Sub-10ms point queries & analytical filters)

---

Runnable Streaming Materialized View DDL with Kafka CDC Ingestion

The following runnable SQL script demonstrates continuous CDC ingestion from an upstream PostgreSQL ordering system via Kafka, joining it with a clickstream event topic to maintain a real-time analytics view.

-- Step 1: Connect to RisingWave via psql or standard Postgres client
-- psql -h localhost -p 4566 -d dev -U root

-- Step 2: Create a streaming source for PostgreSQL CDC orders via Debezium
CREATE SOURCE orders_cdc_source (
    order_id BIGINT,
    customer_id BIGINT,
    order_amount NUMERIC,
    order_status VARCHAR,
    created_at TIMESTAMP WITH TIME ZONE
) WITH (
    connector = 'kafka',
    topic = 'postgres-cdc.inventory.orders',
    properties.bootstrap.server = 'kafka-broker-1:9092,kafka-broker-2:9092',
    scan.startup.mode = 'latest'
) FORMAT PLAIN ENCODE JSON;

-- Step 3: Create a real-time table for web clickstream telemetry
CREATE TABLE clickstream_events (
    session_id VARCHAR,
    customer_id BIGINT,
    page_url VARCHAR,
    device_type VARCHAR,
    event_timestamp TIMESTAMP WITH TIME ZONE,
    WATERMARK FOR event_timestamp AS event_timestamp - INTERVAL '5' SECOND
) WITH (
    connector = 'kafka',
    topic = 'telemetry.web.clickstream',
    properties.bootstrap.server = 'kafka-broker-1:9092,kafka-broker-2:9092'
) FORMAT PLAIN ENCODE JSON;

-- Step 4: Create a continuous streaming Materialized View
-- Performs a 10-minute temporal sliding window join between orders and clickstream
CREATE MATERIALIZED VIEW mv_customer_conversion_realtime AS
SELECT
    o.customer_id,
    o.order_id,
    o.order_amount,
    c.session_id,
    c.device_type,
    c.page_url AS landing_page,
    window_start,
    window_end,
    COUNT(*) OVER (PARTITION BY o.customer_id) AS total_recent_conversions
FROM
    TUMBLE(orders_cdc_source, created_at, INTERVAL '10' MINUTE) o
JOIN
    clickstream_events c
ON
    o.customer_id = c.customer_id
    AND c.event_timestamp BETWEEN o.created_at - INTERVAL '15' MINUTE AND o.created_at
WHERE
    o.order_status = 'COMPLETED';

-- Step 5: Create a secondary index for sub-millisecond point lookups
CREATE INDEX idx_mv_customer_lookup 
ON mv_customer_conversion_realtime (customer_id, order_id);

---

Client Implementation: Querying Streaming Views via Node.js / TypeScript

Because RisingWave implements the PostgreSQL wire protocol, any standard PostgreSQL client driver (such as pg, Prisma, or TypeORM) can query continuous streaming state with zero custom SDK dependencies.

Engineering teams constructing distributed transactional systems and analytical backends often require end-to-end architectural rigor when integrating streaming storage engines into production environments. When building high-throughput event processing pipelines, exploring Wise Hustlers custom software engineering services helps teams design resilient distributed data infrastructure tailored to strict latency and availability requirements.

Below is an enterprise TypeScript service querying the live streaming materialized view:

import { Pool, PoolConfig } from "pg";

// Interface mapping the RisingWave streaming materialized view schema
export interface CustomerConversionMetric {
  customerId: string;
  orderId: string;
  orderAmount: number;
  sessionId: string;
  deviceType: string;
  landingPage: string;
  windowStart: Date;
  windowEnd: Date;
  totalRecentConversions: number;
}

export class StreamingAnalyticsClient {
  private pool: Pool;

  constructor() {
    const config: PoolConfig = {
      host: process.env.RISINGWAVE_HOST || "localhost",
      port: parseInt(process.env.RISINGWAVE_PORT || "4566", 10),
      database: process.env.RISINGWAVE_DB || "dev",
      user: process.env.RISINGWAVE_USER || "root",
      password: process.env.RISINGWAVE_PASSWORD || "",
      max: 20, // Connection pool size
      idleTimeoutMillis: 30000,
      connectionTimeoutMillis: 2000,
    };

    this.pool = new Pool(config);

    this.pool.on("error", (err: Error) => {
      console.error("[RisingWave-Client] Unexpected connection pool error:", err.message);
    });
  }

  /**
   * Retrieves real-time conversion metrics directly from the live materialized view.
   * Utilizes the secondary index on (customer_id, order_id) for sub-5ms lookups.
   */
  public async getConversionMetrics(customerId: string): Promise<CustomerConversionMetric[]> {
    const query = `
      SELECT 
        customer_id AS "customerId",
        order_id AS "orderId",
        order_amount AS "orderAmount",
        session_id AS "sessionId",
        device_type AS "deviceType",
        landing_page AS "landingPage",
        window_start AS "windowStart",
        window_end AS "windowEnd",
        total_recent_conversions AS "totalRecentConversions"
      FROM mv_customer_conversion_realtime
      WHERE customer_id = $1
      ORDER BY window_end DESC
      LIMIT 10;
    `;

    const start = performance.now();
    const client = await this.pool.connect();

    try {
      const res = await client.query(query, [customerId]);
      const durationMs = (performance.now() - start).toFixed(2);
      
      console.log(`[Query Profile] Fetched ${res.rowCount} streaming rows in ${durationMs}ms`);
      return res.rows as CustomerConversionMetric[];
    } finally {
      client.release();
    }
  }

  public async close(): Promise<void> {
    await this.pool.end();
  }
}

---

Architectural Decision Framework & Production Trade-Offs

Choosing between Apache Flink 2.0, RisingWave, and Kafka Streams involves balancing query complexity, deployment topology, hardware costs, and operational maturity.

Multi-Dimensional Comparison Matrix

---

When to Select Apache Flink 2.0

• Complex Event Processing (CEP): Your application relies heavily on complex temporal pattern recognition across unordered streams using MATCH_RECOGNIZE.
• Custom Low-Level State APIs: You need granular access to KeyedProcessFunction, broadcast state, or custom timer services via the Java/Scala DataStream API.
• Pre-Existing Flink Infrastructure: Your organization maintains established Flink Kubernetes operators, checkpoint management pipelines, and monitoring infrastructure.

When to Select RisingWave

• Unified Stream-to-Serving Analytics: You want to eliminate the operational cost of maintaining separate analytical databases (ClickHouse, Pinot, Redis) downstream from your stream processor.
• SQL-First Development: Your pipelines can be expressed cleanly as streaming SQL queries, windowed joins, and real-time materialized views.
• Strict MTTR and Autoscaling Requirements: Your production service demands failover recovery and node rescaling within single-digit seconds without paying for idle standby nodes.
• PostgreSQL Ecosystem Integration: You want to query streaming state directly from standard business intelligence tools, ORMs, and application code via standard PostgreSQL drivers.

When to Select Kafka Streams

• Self-Contained Microservice Architectures: You are building lightweight, event-driven microservices that run within your existing application deployment lifecycle.
• Zero Dedicated Cluster Overhead: You do not have the operational capacity to manage dedicated Flink or RisingWave cluster infrastructure and prefer running stateful logic inside application containers.
• Small-to-Moderate State Sizes: Your state stores remain comfortably under 20 GB per partition, keeping changelog replay times and standby replica costs within reasonable bounds.

---

Frequently Asked Questions

How does Flink 2.0 ForSt handle object storage read latency compared to local RocksDB?

Flink 2.0 mitigates remote storage latency through two mechanisms: tiered local NVMe caching and the Asynchronous Execution Controller (AEC). The local NVMe cache serves frequently accessed working state blocks directly. When a cache miss occurs, the AEC releases the current execution thread to process other incoming records while asynchronously fetching the required block from remote object storage. This ensures overall stream throughput remains high even though individual remote read operations carry a higher latency penalty than local disk lookups.

Can RisingWave completely replace downstream analytical databases like ClickHouse?

RisingWave effectively replaces analytical databases for workloads centered on real-time point lookups, indexed filters, and continuous aggregated views over streaming data. However, if your use case involves ad-hoc, brute-force OLAP scans over petabytes of historical cold data without pre-aggregated materialized views, a dedicated columnar store like ClickHouse or Apache Doris remains complementary. Many enterprises deploy RisingWave for low-latency streaming aggregations (sub-second fresh state) and export batch historical snapshots to object-storage data lakes.

What happens in Kafka Streams if a worker fails when `num.standby.replicas` is zero?

When num.standby.replicas is configured to zero, Kafka Streams must initiate a full cold state restoration. The newly assigned instance creates an empty local RocksDB database and replays the entire partition changelog topic from the Kafka brokers from offset zero (or the latest local checkpoint). For state stores exceeding 100 GB, this reconstruction process can take 5 to 15+ minutes, during which event processing for that specific partition is completely blocked.

Why does RisingWave exhibit lower memory consumption than Flink 2.0 during multi-stream joins?

RisingWave is implemented in Rust, which operates without a garbage collection runtime. It controls memory layout at the byte level, avoiding the substantial JVM object header overhead and memory fragmentation inherent in Java-based stream processors. Additionally, RisingWave's Hummock engine uses a shared memory buffer across all streaming actors within a compute node, whereas Flink historically allocates separate write buffers and block caches across distinct operator instances.

---

Sources

• Apache Flink 2.0 Documentation & Architecture
• FLIP-427: Disaggregated State Storage with ForSt
• FLIP-388: Asynchronous State Execution in Flink
• RisingWave: Cloud-Native Streaming Database Architecture
• RisingWave Hummock Storage Engine Overview
• Apache Kafka Streams Architecture & State Stores
• NEXMARK Benchmark Suite for Streaming Systems