Overhead on nested combinators

[donporter]

Hi!

I was trying out Vest for a project, and noticed that with nested structures (think payloads with multiple wrapping layers of headers/footers like an onion), the combinator overheads in Vest are pretty high compared to a hand- (or preprocesor-)unrolled serializer. I can share a full tarball out of band, but here are two examples.

I suspect that Vest could do a similar trick without undermining the correctness proof.

Record (flat type, 10k iterations, ~180 bytes each)

Operation	Hand-unrolled	Vest combinators	Ratio
serialize	55µs	278µs	~5x
deserialize	170µs	164µs	~1.0x

Table (nested Vec<Entry>, 1k tables x 10 entries, ~1KB each)

Operation	Hand-unrolled	Vest combinators	Ratio
serialize	46µs	434µs	~9.4x
deserialize	171µs	271µs	~1.6x

!BIG_ENDIAN

record = {
    id: u64,
    key: [u8; 32],
    @payload_len: u32,
    payload: [u8; @payload_len],
}

entry = {
    key: [u8; 32],
    @value_len: u32,
    value: [u8; @value_len],
}

table = {
    id: u64,
    @entry_count: u32,
    entries: [entry; @entry_count],
}

Manual:

/// Vest Combinator vs Hand-Unrolled Serde Benchmark
///
/// Demonstrates the performance overhead of using Vest's verified combinator
/// framework compared to hand-unrolled serialization for the same wire format.
///
/// The wire format is identical — the difference is purely in how the code
/// traverses the combinator stack vs directly reading/writing bytes.

// ---- Vest-generated module (from record.vest) ----
#[allow(warnings, unused)]
pub mod vest_generated {
    include!("record_generated.rs");
}

// ---- Domain types ----

/// Simple flat record: u64 + [u8; 32] + length-prefixed payload
#[derive(Debug, Clone, PartialEq)]
pub struct Record {
    pub id: u64,
    pub key: [u8; 32],
    pub payload: Vec<u8>,
}

/// Entry with a 32-byte key and variable-length value
#[derive(Debug, Clone, PartialEq)]
pub struct Entry {
    pub key: [u8; 32],
    pub value: Vec<u8>,
}

/// Table with nested Vec<Entry> — exercises combinator stack depth
#[derive(Debug, Clone, PartialEq)]
pub struct Table {
    pub id: u64,
    pub entries: Vec<Entry>,
}

// ============================================================
// Record: flat type (u64 + [u8;32] + Vec<u8>)
// ============================================================

pub mod record_hand {
    use super::Record;

    pub fn size(r: &Record) -> usize {
        8 + 32 + 4 + r.payload.len()
    }

    pub fn serialize(r: &Record, buf: &mut Vec<u8>) -> usize {
        let len = size(r);
        buf.resize(len, 0);
        let mut pos = 0;
        buf[pos..pos + 8].copy_from_slice(&r.id.to_be_bytes());
        pos += 8;
        buf[pos..pos + 32].copy_from_slice(&r.key);
        pos += 32;
        buf[pos..pos + 4].copy_from_slice(&(r.payload.len() as u32).to_be_bytes());
        pos += 4;
        buf[pos..pos + r.payload.len()].copy_from_slice(&r.payload);
        pos += r.payload.len();
        pos
    }

    pub fn deserialize(buf: &[u8]) -> (usize, Record) {
        let mut pos = 0;
        let id = u64::from_be_bytes(buf[pos..pos + 8].try_into().unwrap());
        pos += 8;
        let key: [u8; 32] = buf[pos..pos + 32].try_into().unwrap();
        pos += 32;
        let payload_len = u32::from_be_bytes(buf[pos..pos + 4].try_into().unwrap()) as usize;
        pos += 4;
        let payload = buf[pos..pos + payload_len].to_vec();
        pos += payload_len;
        (pos, Record { id, key, payload })
    }
}

// ============================================================
// Table: nested type (u64 + Vec<Entry{[u8;32], Vec<u8>}>)
// ============================================================

pub mod table_hand {
    use super::{Table, Entry};

    pub fn size(t: &Table) -> usize {
        8 + 4 + t.entries.iter().map(|e| 32 + 4 + e.value.len()).sum::<usize>()
    }

    pub fn serialize(t: &Table, buf: &mut Vec<u8>) -> usize {
        let len = size(t);
        buf.resize(len, 0);
        let mut pos = 0;
        buf[pos..pos + 8].copy_from_slice(&t.id.to_be_bytes());
        pos += 8;
        buf[pos..pos + 4].copy_from_slice(&(t.entries.len() as u32).to_be_bytes());
        pos += 4;
        for e in &t.entries {
            buf[pos..pos + 32].copy_from_slice(&e.key);
            pos += 32;
            buf[pos..pos + 4].copy_from_slice(&(e.value.len() as u32).to_be_bytes());
            pos += 4;
            buf[pos..pos + e.value.len()].copy_from_slice(&e.value);
            pos += e.value.len();
        }
        pos
    }

    pub fn deserialize(buf: &[u8]) -> (usize, Table) {
        let mut pos = 0;
        let id = u64::from_be_bytes(buf[pos..pos + 8].try_into().unwrap());
        pos += 8;
        let entry_count = u32::from_be_bytes(buf[pos..pos + 4].try_into().unwrap()) as usize;
        pos += 4;
        let mut entries = Vec::with_capacity(entry_count);
        for _ in 0..entry_count {
            let key: [u8; 32] = buf[pos..pos + 32].try_into().unwrap();
            pos += 32;
            let value_len = u32::from_be_bytes(buf[pos..pos + 4].try_into().unwrap()) as usize;
            pos += 4;
            let value = buf[pos..pos + value_len].to_vec();
            pos += value_len;
            entries.push(Entry { key, value });
        }
        (pos, Table { id, entries })
    }
}

[y1ca1]

Thanks for giving Vest a try and the detailed results and suggestions! I didn't expect that the combinators would introduce any "real" overhead especially when aggressive inlining is enabled. I will look into this and hopefully make Vest's abstraction zero cost.