Wise Hustlers — Digital Product & App Development Studio Logo
Get Consultation
By Wise Hustler Admin•11/19/2026•23 min read

Apache DataFusion vs DuckDB: Embedded Analytical Query Engine Benchmarks on Parquet and Apache Iceberg

Apache DataFusion vs DuckDB: Embedded Analytical Query Engine Benchmarks on Parquet and Apache Iceberg

# Apache DataFusion vs DuckDB Benchmarks: Embedded Analytical Query Engines on Parquet and Apache Iceberg

Summary: Apache DataFusion and DuckDB represent two distinct embedded OLAP query engine architectures. In standardized ClickBench queries on Parquet, DuckDB achieves high single-node raw throughput on local NVMe storage using its morsel-driven push-based pipeline and integrated C++ buffer pool. Apache DataFusion excels across memory-constrained environments, concurrent Tokio asynchronous tasks, and remote object storage (AWS S3) by utilizing Apache Arrow record batches, modular memory pools, and native Rust-based Iceberg catalog integration.

---

The Rise of Embedded OLAP: Architectural Divergence

For more than two decades, embedded data storage was dominated by SQLite. SQLite provided a transactional, single-file relational database embedded directly inside host application processes. However, modern data engineering platforms, edge telemetry processors, metric aggregators, and lakehouse query layers face fundamentally different workloads. They require analytical query processing (OLAP) directly against immutable, columnar data formats such as Apache Parquet and Apache Iceberg, bypassing the overhead of standalone database clusters.

This architectural shift has given rise to embedded columnar analytical query engines. Rather than managing separate database daemons, network serialization boundaries, and distributed coordinators, software engineers can now embed vector execution engines directly into applications.

Two engines have emerged at the forefront of this space: DuckDB and Apache DataFusion. While both execute analytical SQL queries over columnar data at hardware limits, their design philosophies, internal execution paradigms, and extensibility profiles diverge sharply:

  • DuckDB is authored in C++11 and operates as a complete, autonomous analytical database management system (DBMS). It provides a full SQL parser, planner, cost-based optimizer, transactional storage engine, vector execution runtime, and rich client ecosystem (Python, Node.js, Go, R, Rust). DuckDB is engineered as a standalone drop-in analytical engine that requires zero external configuration.
  • Apache DataFusion is an extensible, modular query engine framework written in Rust. It was built from the ground up on the Apache Arrow in-memory columnar standard and the Tokio asynchronous runtime. Rather than positioning itself solely as an end-user database, DataFusion acts as an extensible engine framework. It provides foundational building blocks—logical plans, physical plans, expression trees, optimizers, and catalog providers—that engineers use to assemble custom query engines, distributed runtimes (such as Apache Arrow Ballista), and execution accelerators (such as Apache Comet for Apache Spark).

Understanding the technical trade-offs between DuckDB's monolithic, push-based execution and DataFusion's modular, pull-based asynchronous architecture is essential for selecting the correct engine for Parquet and Iceberg workloads.

---

Core Execution Paradigms: Push-Based Pipelines vs. Pull-Based Arrow Streams

The performance divergence between DuckDB and Apache DataFusion originates in their core execution models: DuckDB implements morsel-driven push-based pipelining, whereas Apache DataFusion implements pull-based vectorized batch streaming over Apache Arrow memory buffers.

DuckDB: Morsel-Driven Push-Based Vectorized Execution

DuckDB adapts the morsel-driven parallelism model originally designed for the Hyper database system (Leis et al., 2014). In this model:

1. Pipeline Construction: The physical query planner breaks the execution plan into discrete execution pipelines bounded by pipeline breakers. A pipeline breaker is an operator that cannot produce output tuples until all input tuples have been consumed (such as the build side of a hash join, a global sort, or a materializing aggregation).

2. Push-Based Processing: Within each pipeline, data is pushed from the source operator through intermediate operators (projections, filters, hash probes) directly into a sink operator. Push-based processing provides high CPU instruction-cache locality: once an intermediate vector is loaded into CPU L1/L2 cache, all linear operators in the pipeline execute consecutively on that vector before evicting it.

3. Dynamic Morsel Scheduling: Input datasets are partitioned dynamically into "morsels"—chunks containing approximately 100,000 to 120,000 tuples. Worker threads pull morsels from the source operator and iterate over them in standard DuckDB vectors of 2,048 tuples. If one worker thread encounters a dense, uncompressed Parquet row group while another encounters a dictionary-compressed row group, the dynamic scheduler balances the workload across threads without idle worker stalls.

4. Hardware Alignment: DuckDB vectors are aligned to cache lines and execute tightly inlined C++ loops with explicit SIMD vectorization (AVX2, AVX-512, and ARM NEON), minimizing function pointer indirection and virtual call dispatch.

Apache DataFusion: Apache Arrow Pull-Based Record Batch Streaming

Apache DataFusion utilizes a pull-based Volcano iterator model modernized for asynchronous stream processing:

1. Arrow Native Representation: DataFusion does not maintain an internal proprietary vector format. Instead, all tabular data exists as Apache Arrow RecordBatches. Each RecordBatch is a collection of contiguous, immutable, memory-aligned Arrow arrays (primitive arrays, dictionary arrays, string offsets, and null validity bitmaps).

2. Sendable RecordBatch Streams: Physical operators implement the ExecutionPlan trait. Calling execute(partition, context) returns a SendableRecordBatchStream (Pin<Box<dyn RecordBatchStream + Send>>). Downstream consumer operators asynchronously call poll_next() to request the next RecordBatch.

3. Tokio Concurrency: DataFusion delegates thread scheduling to Rust's multi-threaded tokio runtime. The physical planner splits execution into a configured number of partitions (typically matching CPU core counts). Each partition runs as a distinct Tokio green thread. When an operator must wait for disk I/O, network packets from AWS S3, or an upstream channel, the Tokio worker thread yields immediately to process other partitions.

4. Zero-Copy Memory Semantics: Because DataFusion adheres to the Arrow memory format, operations such as column projection, slicing, and array concatenations do not copy underlying memory buffers. Memory buffers are managed through atomic reference counters (Arc<Buffer>), enabling zero-copy data exchange with other Arrow-native systems without serialization costs.

---

Architectural Comparison: Push vs. Pull Vectorization

The following architectural diagram illustrates the structural differences between DuckDB's morsel-driven push pipeline and Apache DataFusion's asynchronous pull stream:

flowchart TD
  subgraph DuckDB["DuckDB: Morsel-Driven Push Pipeline (C++)"]
    direction TB
    D_Source["Parquet Storage Source"] --> D_Morsel["Dynamic Morsel Scheduler\n(~120K Tuples/Morsel)"]
    D_Morsel --> D_W1["Worker Thread 1\n(Vector: 2048 Tuples)"]
    D_Morsel --> D_W2["Worker Thread 2\n(Vector: 2048 Tuples)"]
    D_Morsel --> D_W3["Worker Thread N\n(Vector: 2048 Tuples)"]
    D_W1 --> D_Pipe["Push: In-Cache Filter -> Project -> Probe"]
    D_W2 --> D_Pipe
    D_W3 --> D_Pipe
    D_Pipe --> D_Sink["Pipeline Breaker / Sink\n(Hash Table Build / Global Sort)"]
  end

  subgraph DataFusion["Apache DataFusion: Async Arrow Pull Stream (Rust)"]
    direction TB
    DF_Sink["Query Root / Result Collector"] -->|poll_next| DF_Stream["SendableRecordBatchStream\n(Async Pull Interface)"]
    DF_Stream --> DF_Plan["Physical ExecutionPlan Tree\n(HashJoinExec / AggregateExec)"]
    DF_Plan --> DF_Tokio["Tokio Task Scheduler\n(Partitioned Worker Pool)"]
    DF_Tokio --> DF_T1["Tokio Task (Partition 0)"]
    DF_Tokio --> DF_T2["Tokio Task (Partition 1)"]
    DF_Tokio --> DF_TN["Tokio Task (Partition N)"]
    DF_T1 --> DF_Arrow["Apache Arrow RecordBatches\n(Zero-Copy Buffers / Jemalloc)"]
    DF_T2 --> DF_Arrow
    DF_TN --> DF_Arrow
    DF_Arrow --> DF_Source["ObjectStore / Parquet Exec"]
  end

DuckDB pushes fixed-size vectors upward through compiled C++ pipelines until hitting a pipeline breaker, whereas DataFusion pulls RecordBatch streams downward through Tokio tasks while maintaining Apache Arrow buffer references.

---

Memory Management Breakdown: C++ Buffer Pool vs. Rust Arenas and Jemalloc

Predictable memory management under heavy analytical workloads determines whether an embedded engine executes smoothly or crashes the host process via out-of-memory (OOM) killer signals.

DuckDB Buffer Management and Disk Spilling

DuckDB features a centralized, tightly integrated C++ BufferManager:

  • Configured Memory Ceiling: The host application can configure an explicit memory ceiling via SET max_memory='16GB';. The BufferManager enforces this ceiling globally across all worker threads.
  • Standardized Block Allocation: In-memory data and cached file structures are broken into uniform 256 KB memory blocks. The buffer manager maintains an eviction queue based on an adaptive Least Recently Used (LRU) algorithm.
  • Disk Spilling Mechanism: When intermediate query states (such as hash join tables or sorting buffers) push total memory consumption past the ceiling, the BufferManager serializes temporary blocks into an on-disk swap directory (.tmp.duckdb). Uncompressed query blocks are compressed on the fly before writing to disk, maintaining query stability even when processing datasets larger than available RAM.
  • Buffer Recycling: DuckDB recycles allocated vector buffers within execution pipelines to minimize standard system allocator calls (malloc/free), avoiding operating system heap fragmentation during long-running analytical sessions.

DataFusion MemoryPool and Arrow Buffer Allocations

Apache DataFusion approaches memory management through a decoupled, trait-driven architecture:

  • The `MemoryPool` Trait: Memory accounting in DataFusion is separated from raw memory allocation. The engine provides three primary memory pool implementations:

1. GreedyMemoryPool: Allocates memory aggressively on demand. It fails immediately with an error if a designated threshold is exceeded.

2. FairSpillPool: Tracks memory across concurrent queries and tasks. If a task requests memory beyond its fair share or the global pool ceiling, the pool invokes disk spilling on the requesting operator.

3. UnboundedMemoryPool: Tracks memory consumption for metrics and profiling without enforcing an upper boundary.

  • Memory Reservations: Operators that accumulate state (such as ExternalSorter, HashJoinExec, and AggregateExec) must construct a MemoryReservation handle. As hash tables grow, the operator calls reservation.try_grow(bytes). If the pool denies the request, the operator invokes its spilling routine, writing intermediate Arrow batches to disk in Arrow IPC streaming format.
  • Jemalloc Integration: DataFusion relies on the host system's global allocator. In production Rust services, DataFusion is typically compiled with tikv-jemallocator or mimalloc. Jemalloc organizes allocations into thread-specific arenas, which mitigates lock contention across dozens of Tokio worker tasks allocating Arrow arrays simultaneously.
  • Zero-Copy Slicing: Arrow arrays use Arc<Buffer>. Slicing a column of 1,000,000 integers to process a 2,048-row batch merely increments an atomic reference count and offsets a pointer. No memory is cloned or moved across CPU registers.

---

Standardized ClickBench Benchmarks: Parquet on Local NVMe vs. AWS S3

To evaluate single-node analytical execution on Parquet, the data engineering community relies on ClickBench, the open-source benchmark suite developed by ClickHouse. ClickBench standardizes performance evaluation across columnar database engines using a real-world web analytics dataset containing 100 million rows (hits.parquet, 14.8 GB compressed Snappy, 99 columns) across 43 analytical SQL queries.

It is critical to distinguish between raw columnar formats and table formats: ClickBench evaluates raw Parquet files and native DBMS storage structures; it does not evaluate open table format metadata layers such as Apache Iceberg or Delta Lake. Table formats introduce snapshot manifests and delete reconciliations that require separate architectural consideration.

Benchmark Environment and Methodology

Reproducible evaluation of embedded OLAP engines requires rigorous specification of compute, storage subsystems, and caching protocols. Benchmarking on AWS requires storage-optimized instance types equipped with physical NVMe drives:

  • Compute Instance: AWS EC2 c6id.8xlarge (32 vCPUs, Intel Xeon Platinum 8375C @ 2.90 GHz, 64 GB DDR4 RAM, up to 12.5 Gbps network interface). Note that local NVMe instance storage is available on c6id instances, whereas standard c6i instances are EBS-only. Reference results also evaluate against the community standard ClickBench platform (c6a.4xlarge, 16 vCPUs, AMD EPYC 7R13, 32 GB RAM).
  • Operating System: Ubuntu 24.04 LTS (Linux Kernel 6.8.0).
  • Storage Tiers:

1. Local NVMe: Direct PCIe 4.0 NVMe SSD instance storage formatted with ext4 (delivering up to 3.2 GB/s sequential read throughput).

2. Cloud Object Storage: AWS S3 Standard bucket in the same availability zone (us-east-1), queried directly over HTTPS endpoints without caching middleware.

  • Engine Versions: DuckDB v1.1+ (compiled with default release flags, threads=32), Apache DataFusion v43+ (compiled with Rust 1.82+, opt-level = 3, target-cpu = native, 32 partition threads).
  • Execution Protocol:

Cold NVMe*: Operating system page caches dropped prior to query execution (echo 3 > /proc/sys/vm/drop_caches) to measure raw I/O throughput, metadata parsing, and column decompression.

Warm NVMe*: Executed consecutively over 5 iterations after pre-warming the OS page cache; measures pure CPU vector evaluation and memory bandwidth.

Cold S3*: Query executed directly against AWS S3 with no cached local metadata or local byte buffers, testing remote range request overhead.

Warm S3*: Consecutive execution against S3 utilizing in-engine metadata reuse and persistent HTTP connection pools.

ClickBench Performance Comparison Table

The table below maps 6 canonical ClickBench queries representing key OLAP access patterns—filtering, exact distinct counts, low-cardinality aggregation, string sorting, high-cardinality grouping, and multi-key aggregations—analyzing operator bottlenecks and comparative execution behavior based on public ClickBench benchmarks (benchmark.clickhouse.com).

Query IDOfficial ClickBench SQL PatternWorkload Profile & Primary BottleneckDuckDB Execution DynamicsDataFusion Execution DynamicsObserved Performance & Resource Characteristics
Q02SELECT COUNT(*) FROM hits WHERE AdvEngineID <> 0;Point filter & scan; bounded by column decoding and predicate vector evaluation.Evaluates selection vector in C++ loops; skips non-qualifying rows without materializing unprojected columns.Arrow compute kernels evaluate boolean mask; Tokio partitions scan Parquet row groups concurrently.Sub-second warm execution in both engines. DuckDB minimizes function call overhead on NVMe; DataFusion scales effectively across Tokio threads on cold S3 reads.
Q05SELECT COUNT(DISTINCT UserID) FROM hits;Exact distinct count; bounded by hash set allocation and cache misses on 100M keys.Builds thread-local hash sets in C++; merges sets into final accumulator during pipeline sink phase.Utilizes Arrow multi-column distinct grouping kernels; allocates hash tables within configured MemoryPool.Highly memory- and CPU-intensive. DuckDB's inlined hash tables achieve rapid set union; DataFusion provides strict memory accounting to prevent unexpected host OOM.
Q08SELECT AdvEngineID, COUNT(*) FROM hits WHERE AdvEngineID <> 0 GROUP BY AdvEngineID ORDER BY COUNT(*) DESC;Low-cardinality aggregation; bounded by hash aggregation update speed.Hash table fits entirely within CPU L2 cache; vectors push directly into accumulator array.Partitioned AggregateExec streams Arrow arrays into per-task hash tables, merging via final sort operator.Extremely fast warm execution (<100ms). DuckDB's push pipeline maintains strong cache locality; DataFusion displays near-identical throughput via vectorized Arrow accumulators.
Q13SELECT SearchPhrase, COUNT(*) AS c FROM hits WHERE SearchPhrase <> '' GROUP BY SearchPhrase ORDER BY c DESC LIMIT 10;String aggregation & Top-K; bounded by string hashing, heap allocations, and Top-K heap maintenance.Employs string dictionary vectors and compact string representations to avoid string cloning.Utilizes Apache Arrow StringView arrays, storing string prefixes inline (12 bytes) to bypass heap dereferences.StringView in modern DataFusion significantly closes string processing overhead; DuckDB delivers consistent Top-K pruning through its integrated min-heap sink.
Q17SELECT UserID, COUNT(*) FROM hits GROUP BY UserID ORDER BY COUNT(*) DESC LIMIT 10;High-cardinality aggregation; bounded by dynamic hash table resizing and random memory access.Dynamically resizes hash tables within the buffer pool; spills partition blocks to disk if memory limits are reached.Partitions data by hash key across Tokio tasks; invokes FairSpillPool when memory pressure requires spilling.Query runtime is dominated by CPU memory bus bandwidth. Both engines trade top spots depending on SIMD instruction availability and compiler optimizations.
Q31SELECT SearchEngineID, ClientIP, COUNT(*) AS c, SUM(IsRefresh), AVG(ResolutionWidth) FROM hits WHERE SearchPhrase <> '' GROUP BY SearchEngineID, ClientIP ORDER BY c DESC LIMIT 10;Multi-key grouping & multi-aggregate; bounded by composite key hashing and memory bandwidth.Computes composite hashes over 2,048-tuple chunks, pushing results into vectorized hash accumulator.Pulls multi-column RecordBatch streams into vectorized aggregation kernels, evaluating multiple accumulators per batch.Multi-column key hashing tests compiler loop unrolling. DuckDB yields low overhead on local memory; DataFusion masks S3 latency through non-blocking HTTP fetch pipelines.

In-Depth Benchmark Analysis

1. Warm NVMe and In-Memory Performance (DuckDB Push-Based Efficiency):

When data resides entirely within the operating system page cache or fast NVMe storage, DuckDB's morsel-driven push pipeline provides exceptional CPU instruction-cache locality. In queries like Q02 and Q08, DuckDB pushes 2,048-tuple vectors directly through inlined filter and aggregation kernels, keeping accumulators inside CPU L1/L2 cache lines across consecutive iterations. In contrast, DataFusion's pull-based Volcano streams historically incurred minor function call and polling overhead across partition boundaries. However, in recent releases featuring Apache Arrow StringView representations, SIMD-vectorized grouping kernels, and optimized row formats, DataFusion has significantly narrowed this delta, allowing both engines to trade leading scores on the public ClickBench Parquet leaderboard.

2. Remote Object Storage over AWS S3 (DataFusion Asynchronous Advantage):

When executing queries directly against remote cloud object storage (AWS S3) without local caching, the execution bottleneck transitions from CPU compute limits to network I/O latency and HTTP round-trip time (RTT). In this environment, Apache DataFusion demonstrates distinct architectural advantages. Built natively on Rust's asynchronous Tokio runtime and the object_store crate, DataFusion decomposes Parquet scans into concurrent asynchronous byte-range requests. Tokio worker tasks pipeline the retrieval of subsequent row groups concurrently while Arrow CPU compute kernels process currently loaded batches, effectively masking network round-trip latency. In contrast, DuckDB's httpfs extension issues range requests synchronously across its thread pool, which can lead to worker thread stalls while awaiting network buffers.

3. Memory Footprint and Allocation Models:

Memory utilization patterns diverge significantly between the two engines due to their underlying buffering designs. Apache DataFusion operates as a pull-based stream of discrete Arrow RecordBatch instances, deallocating batches immediately once consumed downstream (unless entering a pipeline-breaker sink like a global sort or hash build). DataFusion's FairSpillPool and Jemalloc arena allocation keep memory increments tight and predictable. In contrast, DuckDB's dynamic morsel scheduler allocates thread-local morsel buffers and pre-sized hash sink tables within its centralized BufferManager. While this optimizes CPU cache residency and throughput, it results in higher baseline peak RSS under high thread concurrency before spilling thresholds are met.

---

Querying Apache Iceberg Without a Database Server

Apache Iceberg has become the open table format standard for modern analytical lakehouses. Traditionally, executing SQL against Iceberg required heavy distributed query engines like Trino, Starburst, or Apache Spark. Both DuckDB and Apache DataFusion now enable querying Apache Iceberg tables in-process without running any database server.

An Apache Iceberg table consists of three architectural layers:

1. Iceberg Catalog: Tracks the current metadata pointer (metadata.json).

2. Metadata Layer: Contains snapshot logs, manifest lists (snap-*.avro), and manifest files (*.avro) containing column-level metrics and file partition paths.

3. Data Layer: Stores data in Apache Parquet files, accompanied by positional or equality delete files.

Embedded Iceberg Execution with DuckDB

DuckDB queries Iceberg tables directly using its official iceberg extension. The extension parses Iceberg metadata JSON files and manifest Avro files, applies partition pruning, and delegates columnar reads to DuckDB's native Parquet reader.

The following Python script demonstrates executing embedded SQL on an Iceberg table without a server daemon:

    import duckdb

    # 1. Initialize an in-process, embedded DuckDB instance
    con = duckdb.connect(database=":memory:")

    # 2. Install and load the official Iceberg extension
    con.execute("INSTALL iceberg;")
    con.execute("LOAD iceberg;")

    # 3. Configure AWS S3 credentials for direct object store access
    con.execute("""
        SET s3_region='us-east-1';
        SET s3_access_key_id='AKIAEXAMPLEKEYID';
        SET s3_secret_access_key='wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY';
    """)

    # 4. Query Iceberg table metadata directly via the metadata JSON path
    iceberg_metadata_path = "s3://lakehouse-production-bucket/warehouse/events/metadata/v3.metadata.json"

    query = f"""
        SELECT 
            event_type,
            COUNT(*) AS total_events,
            ROUND(AVG(latency_ms), 2) AS avg_latency
        FROM iceberg_scan('{iceberg_metadata_path}')
        WHERE event_date >= '2026-11-01'
        GROUP BY event_type
        ORDER BY total_events DESC
        LIMIT 5;
    """

    result = con.execute(query).fetch_df()
    print(result)

Execution Breakdown:

  • iceberg_scan() reads the Iceberg v3.metadata.json file to identify the latest snapshot ID.
  • It reads the corresponding manifest list and evaluates partition filters (event_date >= '2026-11-01') against manifest Avro metadata, eliminating non-matching Parquet data files before reading any columnar data blocks.
  • Matching Parquet files are scanned using DuckDB's multi-threaded push engine.

Embedded Iceberg Execution with Apache DataFusion in Rust

In Apache DataFusion, Iceberg tables are queried natively in Rust by registering an IcebergTableProvider with a SessionContext. By integrating DataFusion with the official iceberg crate and the datafusion_iceberg provider, developers embed analytical lakehouse execution directly into high-performance microservices.

Here is a complete, runnable Rust implementation demonstrating proper catalog resolution and metadata pruning:

use std::sync::Arc;
use datafusion::prelude::*;
use datafusion_iceberg::table::IcebergTableProvider;
use iceberg::catalog::Catalog;
use iceberg_catalog_rest::{RestCatalog, RestCatalogConfig};
use url::Url;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    // 1. Initialize DataFusion execution context with multi-threaded Tokio runtime
    let ctx = SessionContext::new();

    // 2. Configure Iceberg REST Catalog client pointing to the lakehouse catalog endpoint
    let catalog_config = RestCatalogConfig::builder()
        .uri("https://catalog.lakehouse.production/v1".to_string())
        .build();
    let catalog = Arc::new(RestCatalog::new(catalog_config));

    // 3. Load the Iceberg table metadata using the official iceberg-rust catalog
    let table_ident = iceberg::TableIdent::from_strs(["warehouse", "events"])?;
    let iceberg_table = catalog.load_table(&table_ident).await?;

    // 4. Wrap the Iceberg table in DataFusion's TableProvider
    // This integration delegates partition pruning, manifest parsing,
    // and delete file handling directly to iceberg-rust before scanning Parquet
    let table_provider = Arc::new(IcebergTableProvider::try_new(iceberg_table).await?);
    
    // Register the Iceberg table into DataFusion's session catalog
    ctx.register_table("events_table", table_provider)?;

    // 5. Construct and execute the analytical query plan
    // Filter predicates and projections are pushed down into the Iceberg scanner
    let df = ctx.sql(
        "SELECT \
            event_type, \
            COUNT(*) AS total_events, \
            AVG(latency_ms) AS avg_latency \
         FROM events_table \
         WHERE event_date >= '2026-11-01' \
         GROUP BY event_type \
         ORDER BY total_events DESC \
         LIMIT 5"
    ).await?;

    // 6. Stream Apache Arrow RecordBatches to standard output
    df.show().await?;

    Ok(())
}

Execution Breakdown:

  • The SessionContext initializes the logical query planner, physical optimizer, and runtime environment.
  • The RestCatalog from iceberg-rust connects to the catalog service (e.g., Iceberg REST, AWS Glue, or Polaris) to resolve the active metadata pointer (v3.metadata.json) without scanning storage directories.
  • IcebergTableProvider implements DataFusion's TableProvider trait. It exposes schema definitions and pushes SQL predicates (WHERE event_date >= '2026-11-01') down to the Iceberg metadata reader.
  • The iceberg-rust engine evaluates the manifest list (snap-*.avro), prunes unneeded manifest files using partition boundaries, evaluates column-level min/max statistics to skip non-qualifying Parquet data files, and reconciles equality and position delete files.
  • Surviving Parquet row groups are streamed directly into DataFusion's physical execution plan as Apache Arrow RecordBatch items across Tokio worker tasks, preserving zero-copy memory semantics.

---

Engineering Decision Matrix: When to Deploy DataFusion vs. DuckDB

Selecting between DuckDB and Apache DataFusion depends on whether your project requires an end-user analytical database system or an embeddable query engine framework.

Architectural Feature Comparison

Architectural DimensionDuckDB (v1.1+)Apache DataFusion (v43+)
Primary LanguageC++11Rust
Core AbstractionSelf-contained relational databaseExtensible analytical engine toolkit
Vector FormatDuckDB proprietary vectors (2,048 tuples)Standard Apache Arrow RecordBatches
Execution ModelMorsel-driven push-based pipelineAsynchronous pull-based batch stream
Concurrency EngineDedicated worker thread pool & schedulerRust Tokio multi-threaded asynchronous runtime
Memory ManagementGlobal C++ BufferManager with disk swapModular MemoryPool (FairSpillPool) + Jemalloc
Extensibility PointsC++ Extensions, Table/Scalar UDFsPluggable Logical Plans, Physical Exec Plans, Catalogs
Language EcosystemPython, Node.js, Go, Rust, Java, R, CLINative Rust, Python (datafusion-python), C ABI
Lakehouse IntegrationOfficial iceberg and delta extensionsNative iceberg-rust, delta-rs, lance ecosystem
Wasm / Edge ExecutionDuckDB-Wasm (browser-compiled)Fully supported via wasm32-unknown-unknown
Cloud Object Store I/Ohttpfs extension (synchronous range pulls)Native object_store crate (async concurrent pulls)

When to Choose DuckDB

1. Data Science and Analytical Notebooks: DuckDB is unmatched for local analytical processing in Python, R, and SQL scripts. It integrates with Pandas, Polars, and Apache Arrow without data copying.

2. Embedded Database Replacement for SQLite: If your microservice requires an in-process database with SQL compliance, joins, window functions, and an analytical focus, DuckDB delivers instant out-of-the-box productivity.

3. Local Fast NVMe Analytics: For workloads where datasets reside locally on fast SSDs and raw query execution speed is the primary optimization goal, DuckDB's inlined C++ push engine provides low latency and strong single-node saturation.

4. Zero-Configuration Desktop Applications: DuckDB compiles cleanly into standalone executables or desktop tools without requiring a Rust toolchain or custom query planners.

When to Choose Apache DataFusion

1. Building Custom Query Engines and Databases: If your team is designing a domain-specific analytical database, time-series store, or lakehouse engine, DataFusion provides a mature, production-tested foundation. You retain full control over logical planning, physical execution rules, cost-based optimizers, and catalog bindings.

2. High-Concurrency Async Rust Microservices: If your service architecture is built on Rust and Tokio, DataFusion fits directly into your async stack. Queries execute as non-blocking Tokio futures without stalling OS threads.

3. Direct Cloud Lakehouse Querying (AWS S3 / GCS / Azure): When analytical queries run against remote cloud storage, DataFusion's async object_store range fetching delivers superior latency masking and lower memory overhead compared to DuckDB.

4. Strict Memory-Constrained Environments: For services operating inside tight container memory limits (such as Kubernetes pods with 512 MB to 2 GB RAM limits), DataFusion's FairSpillPool and streaming Arrow records minimize memory spikes and prevent OOM restarts.

Strategic Perspectives on Custom Analytical Engines

When organizations design high-scale analytical platforms or embedded query engines that process massive Parquet files across cloud storage, selecting the right execution primitives is critical. Teams designing bespoke data infrastructure can partner with specialized systems architects through custom software development to implement tailored, production-grade analytical runtimes.

---

Frequently Asked Questions

How do DuckDB and Apache DataFusion handle Parquet late materialization?

Both engines support Parquet late materialization, but they implement it differently. DuckDB evaluates filter predicates on the specific columns referenced in the WHERE clause first. It builds a selection vector containing only matching tuple row indices and fetches payload columns solely for qualifying rows. Apache DataFusion implements late materialization within its ParquetExec physical operator using Arrow page-level index filtering. It scans predicate columns, evaluates filter expressions at the Parquet data page level, and requests byte ranges for non-filter columns only when matching records exist within that page.

Can DuckDB and DataFusion spill to disk when processing datasets larger than RAM?

Yes. DuckDB includes an integrated C++ BufferManager that monitors a configured memory ceiling (max_memory). When memory limits are reached, DuckDB serializes temporary unpinned blocks to disk using its internal block format. In Apache DataFusion, disk spilling is managed through the FairSpillPool and implemented operator by operator. Operators with unbounded state accumulation (such as ExternalSorter and HashJoinExec) spill sorted runs and partition hash partitions to disk as Arrow IPC streaming files, merging them back as streams once upstream operators complete.

How does Apache DataFusion achieve memory safety without garbage collection overhead?

DataFusion is authored entirely in Rust and benefits from Rust's compile-time ownership and borrow checker mechanics. It eliminates runtime garbage collection pauses and dangling pointer risks. Tabular memory buffers are held in Apache Arrow arrays backed by reference-counted byte containers (Arc<Buffer>). Multiple operators can read, slice, and project columns concurrently without copying underlying memory or acquiring thread locks.

Which engine is better suited for serverless AWS Lambda or Cloudflare Workers?

Both engines can run inside serverless environments, but the choice depends on runtime constraints. DuckDB provides precompiled binaries that run inside containerized AWS Lambda functions, offering fast execution for single queries. However, for memory-restricted serverless containers (such as 256 MB or 512 MB Lambda functions) or edge runtimes targeting WebAssembly (Wasm), Apache DataFusion compiled to wasm32-unknown-unknown provides a smaller binary footprint, lower baseline memory consumption, and non-blocking asynchronous I/O over HTTP range requests.

---

Sources

Related articles