Address misleading performance benchmarks

[ottomated]

I was looking into migrating from Zod and was drawn by the validation benchmarks on both the homepage and blog post. My intuition, however, was that the expensive schema initialization step would make arktype a mistake for serverless environments such as Cloudflare workers where schemas are reinitialized on each request.

I verified it with a quick benchmark that shows a significant performance hit that unfortunately more than outweighs the gain on validation speed:

import { run, bench, boxplot, summary } from 'mitata';
import { z } from 'zod';
import { type } from 'arktype';

boxplot(() => {
  summary(() => {
    bench('zod', () => z.object({
      string: z.string(),
      stringRange: z.string().min(0).max(10),
      optional: z.array(z.number()).optional(),
    }));
    bench('arktype', () => type({
      string: 'string',
      stringRange: '0 < string <= 10',
      'optional?': 'number[]',
    }));
  });
});

await run();

I understand that the library is not focused on initialization performance, but it feels a bit misleading to advertise a 100x speedup when a significant portion of your users will actually experience the opposite. Would it be possible to add a note mentioning this to the relevant docs pages?

[gregg-cbs]

I agree on this. One of the reasons we moved from zod was the performance benefits. Although this does not stray us from arktype, it solves some problems zod could not - like good union types. In mongodb this is essential.

Good post OP.

[ssalbdivad]

Yeah, it would be valuable to give a more nuanced breakdown of the tradeoffs here. I will try to do that in an upcoming comparisons section I have planned for the docs.

That said, I wouldn't say the existing framing is going out of its way to be misleading. It just references the only community-maintained validation benchmarks that exist. 100% agree it would be valuable for those to cover other scenarios like initialization or complex unions.

However, for environments sensitive to initialization speed/bundle size, there will be an opt-in build-step to avoid these costs in an upcoming release:

#810

For now, you may be interested in jitless mode. There will still be some overhead to intializing the fully-reduced types, but it would help mitigate it at the expense of validation speed.