Wise Hustlers — Digital Product & App Development Studio Logo
Get Consultation
By Wise Hustler Admin•11/16/2026•23 min read

Cloudflare Pingora vs Envoy vs NGINX: High-Throughput Edge Proxy Benchmarks & Architecture

Cloudflare Pingora vs Envoy vs NGINX: High-Throughput Edge Proxy Benchmarks & Architecture

# Cloudflare Pingora vs Envoy vs NGINX: High-Throughput Edge Proxy Benchmarks & Architecture

Direct Answer: In high-concurrency edge proxy evaluations, Cloudflare Pingora outperforms Envoy and NGINX under multiplexed HTTP/2 and HTTP/3 traffic by replacing thread-per-core and multi-process isolation with Tokio's multi-threaded work-stealing runtime. While NGINX and Envoy partition upstream connection pools and suffer core utilization skew under persistent connections, Pingora's unified connection pool and cross-core task stealing reduce connection overhead and eliminate tail-latency head-of-line blocking, cutting CPU consumption by 70% and memory by 67% in production.

---

Concurrency Primitives: Work-Stealing vs Thread-per-Core vs Master-Worker

Edge reverse proxies operate under extreme constraints: they terminate TLS handshakes, parse protocol state machines (HTTP/1.1, HTTP/2 multiplexing, and HTTP/3 QUIC streams), route requests based on dynamic policies, and maintain persistent connections to upstream origins. When traffic scales to hundreds of thousands of concurrent client sessions across high-core-count servers, the underlying operating system concurrency model dictates throughput limits, memory efficiency, and P99 tail latency.

Choosing between Cloudflare Pingora, Envoy Proxy, and NGINX requires an architectural understanding of three distinct concurrency models: the multi-process model, the thread-per-core model, and the multi-threaded work-stealing model.

flowchart TD
    subgraph NGINX["NGINX: Multi-Process Master-Worker"]
        direction TB
        N_Master["Master Process (Control Plane)"] --> N_W1["Worker Process 1 (Core 0, epoll)"]
        N_Master --> N_W2["Worker Process 2 (Core 1, epoll)"]
        N_W1 --> N_Pool1["Isolated Upstream Pool 1"]
        N_W2 --> N_Pool2["Isolated Upstream Pool 2"]
    end

    subgraph Envoy["Envoy: C++ Thread-per-Core"]
        direction TB
        E_Main["Main Thread (Init, Admin, xDS)"] --> E_T1["Worker Thread 1 (Core 0, libevent)"]
        E_Main --> E_T2["Worker Thread 2 (Core 1, libevent)"]
        E_T1 --> E_TL1["ThreadLocal Upstream Pool 1"]
        E_T2 --> E_TL2["ThreadLocal Upstream Pool 2"]
        E_T1 -. "Cross-Thread Post (Atomic Queue)" .-> E_T2
    end

    subgraph Pingora["Cloudflare Pingora: Multi-Threaded Tokio Work-Stealing"]
        direction TB
        P_Runtime["Tokio Runtime Scheduler"] --> P_W1["Worker Thread 1 (Core 0)"]
        P_Runtime --> P_W2["Worker Thread 2 (Core 1)"]
        P_W1 <== "Work Stealing (Local Task Queue)" ==> P_W2
        P_W1 --> P_Shared["Global Shared Upstream Pool (Arc & Lock-Free)"]
        P_W2 --> P_Shared
    end

NGINX: Multi-Process Isolation and Keepalive Fragmentation

NGINX employs an event-driven master-worker architecture. A privileged master process reads configuration files, binds listener sockets, and forks unprivileged worker processes—typically pinned 1:1 to physical CPU cores via worker_cpu_affinity. Each worker process executes an independent single-threaded event loop driven by Linux epoll(7) or BSD kqueue(2).

This multi-process design guarantees process isolation: if a worker process crashes due to a memory corruption bug, other workers continue servicing traffic while the master forks a replacement. However, process isolation imposes severe architectural penalties in modern high-throughput environments:

1. Upstream Connection Pool Fragmentation: Worker processes cannot share memory without explicit POSIX shared-memory segments (shm). Consequently, NGINX upstream keepalive connection pools are partitioned per worker process. If an edge proxy runs 64 workers and targets an upstream cluster with keepalive 100;, the system opens up to 6,400 upstream TCP connections. When traffic is sparse or irregularly distributed, connections idle and time out prematurely, triggering repetitive upstream TLS handshakes and ephemeral port exhaustion.

2. Core Imbalance Under Multiplexing: With the reuseport socket flag, the Linux kernel distributes incoming TCP SYN packets using a hash of the IP/port 4-tuple. Under HTTP/1.1, short-lived connections frequently re-hash across workers. Under HTTP/2 and HTTP/3, however, tens of thousands of logical requests are multiplexed over a single long-lived TCP connection or QUIC UDP session. If a subset of persistent connections carries disproportionately heavy payload traffic, the pinned worker saturates at 100% CPU while neighboring worker cores sit idle. NGINX cannot migrate an active file descriptor from one worker process to another.

3. C Codebase Memory Risks: Memory management in NGINX is manual. Despite pool allocators (ngx_pool_t), vulnerability classes like buffer overruns, use-after-free, and pointer misalignment remain persistent operational liabilities.

Envoy Proxy: C++ Thread-per-Core Event Loops and Filter Chains

Envoy Proxy was built in C++14/C++20 by Lyft to overcome the dynamic routing and configuration limitations of traditional proxies. Envoy uses a multi-threaded, thread-per-core architecture managed by a single control-plane main thread.

Envoy spawns one worker thread per physical core (--concurrency N). Each worker thread runs an independent libevent event loop processing I/O events on listener sockets. Key architectural characteristics include:

1. Shared Address Space with Thread-Local Isolation: Envoy threads operate within a single virtual address space. However, to avoid global mutex contention on high-core-count machines, Envoy implements a strict shared-nothing pattern across worker threads. State management relies on ThreadLocal storage.

2. Thread-Local Connection Pools: Similar to NGINX, Envoy worker threads maintain independent, thread-local connection pools to upstream clusters. When worker thread 2 requires an upstream connection, it acquires a connection from its local pool. If none is idle, it initiates a new upstream TCP/TLS handshake. While this design avoids cross-thread locking overhead, it perpetuates upstream connection pool fragmentation across threads. Cross-thread request handoffs require explicit message posting via thread dispatchers (Event::Dispatcher::post()), which incurs context-switch latency and queue contention.

3. Complex L7 Filter Pipelines: Envoy directs requests through a chain of downstream and upstream filters. Each filter dynamically allocates state machines on the heap. Under 100,000 concurrent sessions, this deep object graph inflates resident set size (RSS) memory consumption significantly.

Cloudflare Pingora: Rust Tokio Work-Stealing and Shared Upstream Pooling

Cloudflare engineered Pingora in Rust to replace their NGINX infrastructure, which was processing over 1 trillion requests per day. The architectural goal was eliminating upstream connection fragmentation, removing core imbalance bottlenecks, and enforcing memory safety without garbage collection pauses.

Pingora abandons the thread-per-core isolation model in favor of an asynchronous multi-threaded runtime powered by Tokio. Its architectural differentiators include:

1. Work-Stealing Task Scheduling: Tokio maintains a local run queue for each worker thread alongside a global injector queue. When worker thread 0 finishes processing its local event queue, it does not sleep in an epoll wait loop while other cores are overloaded. Instead, it accesses the tail of worker thread 1's local queue and steals half of its pending asynchronous tasks. When CPU-intensive operations occur—such as HPACK header decompression, Brotli streaming decompression, or TLS 1.3 session negotiations—work is automatically redistributed across all active CPU cores.

2. Unified, Globally Shared Upstream Connection Pool: Because Rust guarantees memory safety and data-race prevention via the type system (Send and Sync traits), Pingora implements a unified upstream connection pool shared across all worker threads. Thread-safe reference counting (Arc) combined with lock-free concurrent hash maps allows any worker thread to borrow an established, idle keepalive socket to an origin. In Cloudflare's published engineering retrospective ("How we built Pingora, the proxy that connects Cloudflare to the internet"), transitioning from NGINX's partitioned pools to Pingora's unified pool reduced new connection establishments to one-third of previous volumes (a ~67% reduction in connection churn). This dramatic reuse cut CPU cycles spent on TLS handshakes, lowering overall median Time to First Byte (TTFB) by 5 ms and 95th percentile TTFB by 80 ms worldwide while saving 70% CPU and 67% memory compared to their previous NGINX service under identical traffic loads.

3. Memory Safety and Zero-Copy Parsing: Written in safe Rust at the application layer, Pingora mitigates memory vulnerability classes (dangling pointers, double frees, buffer overflows, data races). Network packet processing utilizes zero-copy slicing via the bytes crate and Jemalloc, ensuring that payload buffering avoids unnecessary heap reallocation.

---

Concurrency Architecture Matrix

The following table contrasts the low-level operating characteristics of Pingora, Envoy, and NGINX:

Architectural DimensionCloudflare PingoraEnvoy ProxyNGINX
Primary LanguageRust (safe application code)C++ (C++14 / C++20)C (C99 with custom primitives)
Concurrency ModelMulti-threaded work-stealing (Tokio)Multi-threaded thread-per-core (libevent)Multi-process master-worker (epoll)
Upstream Pool ScopeGlobal: Shared across all worker threadsThread-Local: Partitioned per worker threadProcess-Local: Partitioned per worker process
Memory IsolationSingle address space, compile-time borrow checkSingle address space, manual synchronizationSeparate address spaces (fork), shared shm
Core Skew MitigationDynamic work-stealing across run queuesNone (pinned to thread-local event loop)None (pinned to worker-process event loop)
HTTP/3 QUIC IntegrationNative Rust (quiche), zero-copy buffersC++ Google QUICHE libraryNGINX QUIC module (merged mainline)
Configuration ModelProgrammatic Rust code / compiled binaryDynamic via xDS gRPC APIs or static YAMLStatic declarative text files / reload signals
Crash Blast RadiusCaught via catch_unwind task boundariesSingle-thread crash takes down processWorker crash restarts isolated process
Hot Reload MechanismFile descriptor passing via UNIX domain socketHot restart via shared memory socket passingMaster process forks new workers, drains old

---

High-Concurrency Multi-Core Benchmarks (100,000 Connections)

Evaluating how edge proxies scale under extreme concurrency requires analyzing both published global production telemetry—such as Cloudflare's fleetwide transition data—and testing concurrency limits under reproducible multi-core stress conditions with 100,000 concurrent multiplexed connections.

Benchmark Rig Hardware and Kernel Tuning Specification

To evaluate reverse proxy behavior under reproducible high-concurrency conditions, engineering teams deploy a standardized multi-socket server architecture and optimized Linux kernel socket profile. This reference testing setup isolates concurrency bottlenecks by saturating 100GbE network fabrics without bottlenecking on operating system socket limits:

  • Reference Edge Proxy Server Under Test (SUT):

Processor Architecture*: Dual AMD EPYC 9654 (192 physical cores, 384 threads @ 2.4 GHz base / 3.7 GHz boost, 768 MB L3 cache).

Memory*: 384 GB DDR5-4800 ECC Registered RAM across 24 memory channels.

Network Interface*: Dual-port 100GbE Mellanox ConnectX-6 Dx (PCIe 4.0 x16), with SR-IOV and hardware packet steering.

Operating System*: Ubuntu 24.04 LTS, Linux Kernel 6.8.0.

  • Load Generation Architecture:

* Distributed client cluster (8 worker nodes) driving open-loop HTTP/2 traffic via calibrated wrk2 and multiplexed HTTP/3 streams via h2load.

  • Upstream Mock Origin Services:

* Dedicated low-latency backend nodes running lightweight asynchronous HTTP/2 and HTTP/3 response targets (sub-millisecond origin response time).

The SUT kernel networking parameters are tuned to eliminate OS-level socket bottlenecks:

; Maximum open files and file descriptors
fs.file-max = 2097152

; Socket backlog and network core limits
net.core.somaxconn = 65535
net.core.netdev_max_backlog = 100000
net.core.rmem_max = 67108864
net.core.wmem_max = 67108864

; TCP buffer configurations (min, default, max in bytes)
net.ipv4.tcp_rmem = 4096 87380 33554432
net.ipv4.tcp_wmem = 4096 65536 33554432
net.ipv4.tcp_max_syn_backlog = 65535
net.ipv4.tcp_fin_timeout = 15
net.ipv4.tcp_tw_reuse = 1
net.ipv4.ip_local_port_range = 1024 65535

; Epoll and UDP configurations for HTTP/3 QUIC
net.core.optmem_max = 20485760
net.ipv4.udp_rmem_min = 16384
net.ipv4.udp_wmem_min = 16384

The workload profile applied across the cluster consists of:

  • Concurrent Connections: 100,000 persistent client connections established across the generator cluster.
  • Protocol Distribution: 70% HTTP/2 (multiplexed TCP with TLS 1.3), 30% HTTP/3 (QUIC over UDP with TLS 1.3).
  • Request Pipeline: 10 active concurrent streams per connection (1,000,000 inflight HTTP requests at peak load).
  • Payload Characteristics: 85% small API payloads (1 KB JSON request and response), 15% streaming payloads (64 KB gzipped binary objects).
  • Test Duration: 30 minutes sustained load following a 5-minute pre-warm phase.

Multi-Core Benchmark Results and Tail Latency Profiles

Comparing Pingora, Envoy, and NGINX across these dimensions reveals how fundamental architectural decisions dictate high-concurrency performance:

Performance & Concurrency DimensionCloudflare Pingora (v0.3+)Envoy Proxy (v1.30+)NGINX (v1.26+)
Concurrency Scheduling ModelMulti-threaded work-stealing (Tokio runtime)Thread-per-core event loops (libevent)Multi-process master-worker (epoll)
Upstream Keepalive PoolingUnified global pool: Thread-safe (Arc), shared across all worker threadsThread-local pools: Partitioned per worker thread; cross-thread handoffs require message dispatchProcess-local pools: Partitioned per worker process; no cross-process socket sharing
Core Utilization Under MultiplexingUniform: Dynamic task stealing redistributes async tasks from saturated worker queuesCore skew prone: Heavy streams pin to individual worker threads without task migrationSevere core skew: SO_REUSEPORT 4-tuple hashing binds long-lived connections to single workers
Tail Latency Vulnerability (P99/P99.9)Lowest: Idle threads steal pending lightweight tasks, avoiding head-of-line queue blockingModerate: Long-running filter operations or heavy frames delay pending requests on that threadHighest: Head-of-line blocking inside worker processes under asymmetric stream loads
Memory Footprint Scaling (100k Conns)Lowest: Zero-copy buffer slicing (bytes crate) + Jemalloc; minimal per-connection heap bloatHigher: Complex L7 filter graphs, dynamic buffer watermarks, and per-thread stats trackersModerate: Duplicated process page tables and per-worker upstream buffer structures
Verified Production Telemetry (Cloudflare Data)~70% less CPU, ~67% less memory, 5 ms lower median TTFB, 80 ms lower P95 TTFB, 67% fewer new connectionsBaseline service mesh proxy; higher per-connection memory overhead due to filter pipelinesReplaced in production due to upstream pool fragmentation and socket imbalance under multiplexing

Telemetry Analysis: Tail Latency, Memory RSS, and Core Imbalance

Examining these metrics reveals the operational impact of the underlying concurrency primitives:

1. Tail Latency Divergence (P99 and P99.9): Under high concurrency, tail latency is driven by head-of-line blocking inside worker execution queues. In NGINX and Envoy, each worker thread or process executes an isolated event loop. When a client initiates a series of compute-heavy 64 KB streaming downloads across an HTTP/2 or HTTP/3 connection assigned to a specific worker core, that core's event loop becomes saturated running TLS symmetric decryption, HPACK decompression, and chunked transfer framing. Other lightweight 1 KB JSON API streams assigned to that same worker are queued behind the heavy frames. Because neither NGINX nor Envoy allows idle workers to steal tasks from another core's event loop, latency degrades on that saturated core. Cloudflare's published production measurements demonstrated that moving to Pingora's Tokio work-stealing scheduler reduced 95th percentile TTFB by 80 ms worldwide, as idle threads dynamically dequeue and execute pending async tasks from overloaded queues.

2. Resident Set Size (RSS) Memory Footprint: In Cloudflare's production fleet, Pingora demonstrated a ~67% reduction in memory usage compared to NGINX under identical traffic loads. Pingora achieves this memory efficiency by leveraging Rust's bytes::Bytes abstraction with memory managed via Jemalloc, allowing zero-copy packet buffer slicing and immediate deallocation upon frame transmission. In contrast, Envoy's resident memory footprint expands under high concurrent connection counts due to the heap allocations required by its deep L7 filter chain state machines, per-thread statistics tracking, and downstream/upstream buffer watermarks. NGINX incurs memory overhead through duplicated process-level page tables across its worker processes and partitioned upstream buffer chains.

3. CPU Core Saturation and Kernel Socket Hashing Skew: Load distribution across physical CPU cores highlights a fundamental limitation of static socket hashing. With Linux SO_REUSEPORT, incoming connections are assigned to worker processes or threads based on a 4-tuple IP/port hash. Under HTTP/1.1, short-lived connections frequently re-hashed across workers. However, with long-lived multiplexed HTTP/2 and HTTP/3 connections, a single persistent connection carries thousands of interleaved requests. When high-volume sessions land on a subset of workers, those cores can hit 100% CPU capacity while adjacent cores remain underutilized. NGINX cannot migrate open file descriptors between processes, and Envoy does not migrate connections between thread-local loops without expensive inter-thread synchronization. Pingora decouples connection acceptance from request execution: Tokio worker threads collaboratively steal and execute pending tasks, smoothing CPU utilization across all physical cores. In Cloudflare's global deployment, this architectural shift contributed to an overall 70% reduction in CPU consumption compared to their previous NGINX infrastructure.

---

Production Pingora Implementation: Custom Reverse Proxy with Upstream Connection Pooling

To demonstrate Pingora's architecture, we implement a production reverse proxy using pingora-core and pingora-proxy. This implementation features:

1. Dynamic round-robin upstream peer selection.

2. Global upstream keepalive connection pooling with idle timeouts.

3. Zero-copy header manipulation (injecting distributed trace headers, stripping hop-by-hop headers).

4. Automatic upstream failover logic.

Rust Pingora Service Implementation

The following complete Rust implementation builds a high-throughput edge proxy with global connection pooling:

use async_trait::async_trait;
use pingora_core::prelude::*;
use pingora_http::ResponseHeader;
use pingora_proxy::{ProxyHttp, Session};
use std::sync::atomic::{AtomicUsize, Ordering};
use std::time::Duration;

/// Custom Context tracking per-request lifecycle telemetry
pub struct RequestContext {
    pub client_ip: String,
    pub request_start_instant: std::time::Instant,
    pub retries: usize,
}

/// Dynamic Load Balancer with global upstream connection pooling
pub struct ProductionGateway {
    upstream_endpoints: Vec<String>,
    round_robin_counter: AtomicUsize,
    tls_sni_name: String,
}

impl ProductionGateway {
    pub fn new(upstreams: Vec<String>, sni: String) -> Self {
        Self {
            upstream_endpoints: upstreams,
            round_robin_counter: AtomicUsize::new(0),
            tls_sni_name: sni,
        }
    }

    /// Select upstream endpoint using atomic round-robin
    fn select_upstream(&self) -> &str {
        let index = self.round_robin_counter.fetch_add(1, Ordering::Relaxed);
        &self.upstream_endpoints[index % self.upstream_endpoints.len()]
    }
}

#[async_trait]
impl ProxyHttp for ProductionGateway {
    type CTX = RequestContext;

    fn new_ctx(&self) -> Self::CTX {
        RequestContext {
            client_ip: String::new(),
            request_start_instant: std::time::Instant::now(),
            retries: 0,
        }
    }

    /// Select and configure the upstream peer connection
    async fn upstream_peer(
        &self,
        session: &mut Session,
        ctx: &mut Self::CTX,
    ) -> Result<Box<HttpPeer>> {
        let target_host = self.select_upstream();
        
        // Extract client IP for telemetry
        if let Some(client_addr) = session.client_addr() {
            ctx.client_ip = client_addr.to_string();
        }

        // Configure HttpPeer with TLS, SNI, and upstream pooling
        let mut peer = Box::new(HttpPeer::new(
            target_host,
            true, // Enable TLS
            self.tls_sni_name.clone(),
        ));

        // Upstream socket and keepalive connection pooling settings
        peer.options.connection_timeout = Some(Duration::from_millis(500));
        peer.options.total_connection_timeout = Some(Duration::from_secs(2));
        peer.options.read_timeout = Some(Duration::from_secs(10));
        peer.options.write_timeout = Some(Duration::from_secs(10));
        peer.options.idle_timeout = Some(Duration::from_secs(60));

        // Enforce HTTP/2 ALPN negotiation with upstream
        peer.options.alpn = pingora_core::protocols::ALPN::H2;

        Ok(peer)
    }

    /// Filter downstream request before forwarding to upstream
    async fn request_filter(
        &self,
        session: &mut Session,
        _ctx: &mut Self::CTX,
    ) -> Result<bool> {
        // Strip hop-by-hop headers
        let req_header = session.req_header_mut();
        req_header.remove_header("Proxy-Connection");
        req_header.remove_header("Keep-Alive");

        // Inject distributed tracing headers
        let trace_id = format!("wh-{}", fastrand::u128(..));
        req_header.insert_header("X-Request-Trace-ID", &trace_id)?;
        req_header.insert_header("X-Edge-Proxy", "Pingora-Tokio-v0.3")?;

        // Return false to indicate request processing should continue
        Ok(false)
    }

    /// Filter upstream response before writing back to client
    async fn upstream_response_filter(
        &self,
        _session: &mut Session,
        upstream_response: &mut ResponseHeader,
        ctx: &mut Self::CTX,
    ) -> Result<()> {
        // Add security and telemetry response headers
        let duration_ms = ctx.request_start_instant.elapsed().as_millis();
        upstream_response.insert_header("X-Upstream-Latency-MS", duration_ms.to_string())?;
        upstream_response.insert_header(
            "Strict-Transport-Security",
            "max-age=63072000; includeSubDomains; preload",
        )?;
        upstream_response.insert_header("X-Content-Type-Options", "nosniff")?;
        Ok(())
    }

    /// Failover logic when upstream fails to respond or disconnects
    fn fail_to_connect(
        &self,
        _session: &mut Session,
        _peer: &HttpPeer,
        ctx: &mut Self::CTX,
        mut e: Box<Error>,
    ) -> Box<Error> {
        ctx.retries += 1;
        if ctx.retries < 2 {
            log::warn!("Connection failure to upstream. Triggering failover retry #{}", ctx.retries);
            e.set_retry(true);
        } else {
            log::error!("Max retries exceeded for upstream request from {}", ctx.client_ip);
        }
        e
    }
}

fn main() {
    env_logger::init();

    // Initialize Pingora Server runtime with custom worker configurations
    let mut server = Server::new(None).expect("Failed to create Pingora Server");
    server.bootstrap();

    let upstreams = vec![
        "10.0.0.101:443".to_string(),
        "10.0.0.102:443".to_string(),
        "10.0.0.103:443".to_string(),
        "10.0.0.104:443".to_string(),
    ];

    let gateway = ProductionGateway::new(upstreams, "origin.internal.net".to_string());
    let mut proxy_service = pingora_proxy::http_proxy_service(&server.configuration, gateway);

    // Bind dual-stack listening ports with TLS termination
    proxy_service.add_tcp("0.0.0.0:8443");
    
    server.add_service(proxy_service);
    log::info!("Starting Pingora edge proxy daemon on 0.0.0.0:8443...");
    server.run_forever();
}

Comparative Configurations: Envoy and NGINX

To contrast Pingora's programmatic Rust implementation, the equivalent high-concurrency configurations for Envoy and NGINX are detailed below.

Envoy Proxy Configuration (`envoy.yaml`)

static_resources:
  listeners:
  - name: edge_listener_h2
    address:
      socket_address:
        protocol: TCP
        address: 0.0.0.0
        port_value: 8443
    listener_filters:
    - name: envoy.filters.listener.tls_inspector
      typed_config:
        "@type": type.googleapis.com/envoy.extensions.filters.listener.tls_inspector.v3.TlsInspector
    per_connection_buffer_limit_bytes: 32768
    filter_chains:
    - filter_chain_match:
        transport_protocol: "tls"
      transport_socket:
        name: envoy.transport_sockets.tls
        typed_config:
          "@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.DownstreamTlsContext
          common_tls_context:
            alpn_protocols: ["h2", "http/1.1"]
            tls_certificates:
            - certificate_chain: { filename: "/etc/ssl/certs/edge.crt" }
              private_key: { filename: "/etc/ssl/private/edge.key" }
      filters:
      - name: envoy.filters.network.http_connection_manager
        typed_config:
          "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
          stat_prefix: ingress_http
          http2_protocol_options:
            max_concurrent_streams: 100
            initial_stream_window_size: 65536
            initial_connection_window_size: 1048576
          route_config:
            name: local_route
            virtual_hosts:
            - name: backend_services
              domains: ["*"]
              routes:
              - match: { prefix: "/" }
                route:
                  cluster: upstream_service
                  timeout: 10s
                  retry_policy:
                    retry_on: "5xx,connect-failure,reset"
                    num_retries: 2
          http_filters:
          - name: envoy.filters.http.router
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router

  clusters:
  - name: upstream_service
    connect_timeout: 0.50s
    type: STRICT_DNS
    dns_lookup_family: V4_ONLY
    lb_policy: ROUND_ROBIN
    circuit_breakers:
      thresholds:
      - priority: DEFAULT
        max_connections: 50000
        max_pending_requests: 10000
        max_requests: 100000
    typed_extension_protocol_options:
      envoy.extensions.upstreams.http.v3.HttpProtocolOptions:
        "@type": type.googleapis.com/envoy.extensions.upstreams.http.v3.HttpProtocolOptions
        explicit_http_config:
          http2_protocol_options: {}
    load_assignment:
      cluster_name: upstream_service
      endpoints:
      - lb_endpoints:
        - endpoint: { address: { socket_address: { address: 10.0.0.101, port_value: 443 } } }
        - endpoint: { address: { socket_address: { address: 10.0.0.102, port_value: 443 } } }
        - endpoint: { address: { socket_address: { address: 10.0.0.103, port_value: 443 } } }
        - endpoint: { address: { socket_address: { address: 10.0.0.104, port_value: 443 } } }

NGINX Multi-Core Configuration (`nginx.conf`)

user nginx;
worker_processes auto;
worker_cpu_affinity auto;
worker_rlimit_nofile 262144;
pid /var/run/nginx.pid;

events {
    worker_connections 65536;
    use epoll;
    multi_accept on;
}

http {
    include /etc/nginx/mime.types;
    default_type application/octet-stream;

    client_body_buffer_size 16k;
    client_header_buffer_size 2k;
    large_client_header_buffers 4 8k;
    client_max_body_size 10m;

    sendfile on;
    tcp_nopush on;
    tcp_nodelay on;
    keepalive_timeout 65s;
    keepalive_requests 10000;

    upstream backend_nodes {
        zone backend_nodes_shm 64m;
        server 10.0.0.101:443 max_fails=2 fail_timeout=5s;
        server 10.0.0.102:443 max_fails=2 fail_timeout=5s;
        server 10.0.0.103:443 max_fails=2 fail_timeout=5s;
        server 10.0.0.104:443 max_fails=2 fail_timeout=5s;
        
        # Partitioned per worker process: 192 workers * 256 = 49,152 sockets
        keepalive 256;
        keepalive_time 1h;
    }

    server {
        listen 8443 ssl reuseport;
        http2 on;
        server_name edge.internal.net;

        ssl_certificate /etc/ssl/certs/edge.crt;
        ssl_certificate_key /etc/ssl/private/edge.key;
        ssl_session_timeout 1d;
        ssl_session_cache shared:SSL:100m;
        ssl_protocols TLSv1.2 TLSv1.3;

        location / {
            proxy_pass https://backend_nodes;
            proxy_http_version 1.1;
            
            proxy_set_header Connection "";
            proxy_set_header Host $host;
            proxy_set_header X-Forwarded-For $remote_addr;

            proxy_connect_timeout 500ms;
            proxy_read_timeout 10s;
            proxy_send_timeout 10s;

            proxy_buffering on;
            proxy_buffer_size 8k;
            proxy_buffers 8 16k;
        }
    }
}

Reproducing High-Concurrency Load: Reference Test Harness

  # Execute h2load against target edge proxy under 12,500 connections per node (8 nodes = 100k)
  TARGET_HOST="10.0.0.1"
  TARGET_PORT="8443"
  CONCURRENT_CONNS=12500
  MAX_STREAMS=10
  TEST_DURATION="1800s"

  echo "Spawning distributed HTTP/2 load generation..."
  h2load \
    --alpn-list=h2 \
    --clients=${CONCURRENT_CONNS} \
    --max-concurrent-streams=${MAX_STREAMS} \
    --duration=${TEST_DURATION} \
    --warm-up-time=30s \
    --connect-to=${TARGET_HOST}:${TARGET_PORT} \
    "https://${TARGET_HOST}:${TARGET_PORT}/api/v1/telemetry"

---

Architectural Trade-Offs and Migration Decision Matrix

Selecting an edge proxy architecture requires aligning your organization's operational model, scale, and deployment environment against the strengths and constraints of each engine.

When to Deploy Cloudflare Pingora

1. High Concurrency with Tail Latency Sensitivity: If your systems maintain over 50,000 persistent concurrent HTTP/2 or HTTP/3 sessions where tail latency spikes directly violate SLAs, Pingora's Tokio work-stealing scheduler prevents the thread starvation that occurs in thread-per-core architectures.

2. Upstream Connection Pool Efficiency: When fronting distributed microservice topologies where thousands of upstream servers exist, Pingora's unified, thread-shared connection pool reduces connection churn, avoids upstream SYN floods, and saves significant CPU cycles spent on repeated TLS handshakes.

3. Custom Protocol Logic and Programmatic Routing: If your infrastructure requires deep programmatic control—such as custom caching tiers, real-time cryptographic signature verification, dynamic header mutations, or proprietary RPC protocols—writing type-safe Rust directly against pingora-proxy provides stronger compile-time safety and higher performance than compiling C modules for NGINX or writing C++ filters for Envoy.

4. Memory-Safe Infrastructure Standards: Enterprise edge platforms and zero-trust perimeter proxies adhering to industry memory safety advisories (such as CISA, NSA, and White House ONCD software security recommendations) benefit from Rust's compile-time memory guarantees. By eliminating spatial and temporal memory safety bugs—such as buffer overflows, dangling pointers, and use-after-free conditions—without runtime garbage collection, Pingora substantially reduces core attack surfaces compared to C/C++ proxies.

When to Deploy Envoy Proxy

1. Kubernetes-Native Ingress and Service Mesh: Envoy is the foundational data plane for Kubernetes service meshes (Istio, Cilium Service Mesh, Linkerd). Its dynamic xDS APIs (LDS, RDS, CDS, EDS) allow real-time route updates without reloading processes or terminating connections.

2. Standard Enterprise Observability: Envoy provides mature, out-of-the-box telemetry integrations, including distributed tracing (OpenTelemetry, Jaeger, Zipkin), native Prometheus metrics, and access logging without writing custom code.

3. Advanced L7 Traffic Shifting: Envoy's built-in feature set includes canary deployments, shadow traffic mirroring, circuit breaking, local/distributed rate limiting, and gRPC-JSON transcoding out of the box.

When to Deploy NGINX

1. Static File Delivery and Traditional Web Hosting: NGINX remains unmatched in static asset caching and direct OS zero-copy file transmission via sendfile(2) and epoll.

2. Mature Monolithic Stacks: For standard monolithic architectures (WordPress, PHP-FPM, Python WSGI, Ruby Rack) with moderate concurrency (under 20,000 active connections), NGINX provides a straightforward, low-maintenance deployment model.

3. Declarative Configuration Simplicity: When infrastructure teams prefer declarative configuration files over compiled binaries or complex dynamic control planes, NGINX offers rapid configuration iteration without requiring software compilation toolchains.

For enterprises re-architecting edge routing or transitioning high-volume workloads from legacy monoliths to distributed Rust or Go microservices, partnering with seasoned infrastructure specialists for custom software development ensures that low-level networking primitives, connection reuse pools, and kernel socket tunings are rigorously validated prior to production cutover.

---

Frequently Asked Questions

Why did Cloudflare develop Pingora in Rust instead of adopting Envoy?

Cloudflare operated one of the world's largest NGINX footprints, processing over 1 trillion requests per day. While Envoy solved dynamic configuration issues via xDS APIs, its thread-per-core concurrency model still partitions upstream connection pools per worker thread. At Cloudflare's scale, this pool fragmentation caused millions of redundant upstream TLS handshakes and TCP connection overhead. Furthermore, Cloudflare required deep programmatic customization directly in the proxy pipeline. Rust gave Cloudflare compile-time memory safety without a garbage collector, while Tokio's work-stealing runtime enabled a unified, globally shared upstream connection pool across all CPU cores.

How does Tokio work-stealing eliminate P99 tail latency in Pingora?

In traditional thread-per-core proxies like Envoy and multi-process proxies like NGINX, each thread or process processes an isolated event queue. If a worker core encounters an intensive task—such as large HTTP/2 frame decompression or heavy TLS crypto—that core's event loop blocks, causing all subsequent requests queued on that core to experience severe tail latency. Tokio's work-stealing scheduler solves this by maintaining per-thread local run queues alongside a global queue. When an idle worker core empties its local queue, it steals half the pending async tasks from the back of an overloaded core's queue, balancing CPU load across all physical cores.

Can Pingora replace Envoy within a Kubernetes Service Mesh?

Not directly as an out-of-the-box drop-in replacement today. Envoy includes a native, mature implementation of the dynamic xDS control-plane protocol, allowing control planes like Istio to stream endpoint updates, route tables, and TLS certificates dynamically over gRPC without restarts. Pingora is an architectural framework and library rather than an out-of-the-box proxy daemon; deploying Pingora in Kubernetes requires building a custom binary and implementing your own dynamic configuration or xDS client layers in Rust.

How does HTTP/3 QUIC connection migration impact core affinity?

HTTP/3 runs over UDP via QUIC. In NGINX and Envoy, UDP packets are routed to worker threads based on the kernel's 4-tuple hash (SO_REUSEPORT). When a mobile client moves from Wi-Fi to cellular data, its IP address and port change, initiating QUIC Connection Migration using a persistent 64-bit Connection ID (CID). In NGINX and Envoy, the new UDP 4-tuple causes the Linux kernel to steer packets to a different worker core, forcing costly cross-thread synchronization or connection state migration. In Pingora, connection state is accessible across Tokio worker threads via thread-safe shared memory, allowing any worker thread to process migrated QUIC packets without inter-process IPC overhead.

What are the operational trade-offs of embedding custom Rust code directly into the proxy binary?

Embedding custom proxy logic directly into a compiled Rust binary with Pingora eliminates runtime scripting overhead (such as OpenResty Lua or Envoy Wasm plugins), yielding superior execution speed and zero-cost abstraction guarantees. However, it shifts operational complexity into the software release cycle: any routing rule change, header manipulation update, or upstream policy modification requires recompiling the binary, running test suites, and performing a zero-downtime rolling binary upgrade via file-descriptor passing (SO_REUSEPORT / socket handover).

---

Sources

Related articles