Decouple lodestar and @chainsafe/blst

[twoeths]

Problem description

Right now lodestar couples with PublicKey and Signature classes generated by napi-rs @chainsafe/blst. This is not great when we integrate lodestar-z where it uses blst-z there and we may end up 2 blst implementations in the same application

The reason we have to track those classes in lodestar is because rust binding of blst always want PublicKey or Signature instances. But in lodestar, the vast majority of signature sets are verified on worker thread, it means some inefficiencies here: serialize() pubkeys on the main thread, deserialize() pubkeys and signatures on worker threads on NodeJS, then pass to native

Solution description

Track pubkeys and signatures as Uint8Array in NodeJS so that we don't have to serialize() and deserialize() passing through worker boundary, native layers have to deserialize them. This is better for performance (will post later) and provides a way to use bls implementation when we switch to lodestar-z

Additional context

• for non-aggregated validatorpubkey, we can maintain a List of pubkeys in native layer so that we don't have to deserialize there
• for aggregated pubkey, we can use a Map<Uint8Array, PublicKey> at native layer to skip deserializing pubkeys there: cache them on the main thread and lookup/delete on the worker thread. This is optional though, on-the-flight deserialization is already faster than the current flow (see benchmark below)

[twoeths]

some rough benchmark on https://github.com/ChainSafe/blst-ts/blob/86c49590d37d4e1dd44b3b5ba604132f3b51d99d/src/lib.rs#L113 using the most used verifyMultipleAggregateSignatures() api

#[napi(object)]
pub struct SignatureSetSerialized {
  pub msg: Uint8Array,
  pub pk: Uint8Array,
  pub sig: Uint8Array,
}

#[napi]
pub fn verify_multiple_aggregate_signatures_serialized(
  sets: Vec<SignatureSetSerialized>,
  pks_validate: Option<bool>,
  sigs_groupcheck: Option<bool>,
) -> bool {
  // let (msgs, pks, sigs) = unzip_signature_sets_serialized(&sets)?;
  let (msgs, pks, sigs) = match unzip_signature_sets_serialized(&sets) {
    // Case 1: Success. Unwrap the tuple and proceed.
    Ok(result_tuple) => result_tuple,

    // Case 2: Error. Log the error (optional) and return false immediately.
    Err(_e) => {
        // Optional: Add logging here if you need to know why it failed.
        // eprintln!("Error during signature set deserialization: {:?}", _e);
        return false;
    }
  };

  let pks_refs: Vec<&min_pk::PublicKey> = pks.iter().collect();
  let sigs_refs: Vec<&min_pk::Signature> = sigs.iter().collect();
  let rands = create_rand_scalars(sets.len());
  min_pk::Signature::verify_multiple_aggregate_signatures(
    &msgs,
    &DST,
    pks_refs.as_slice(),
    pks_validate.unwrap_or(false),
    sigs_refs.as_slice(),
    sigs_groupcheck.unwrap_or(false),
    &rands,
    64,
  ) == BLST_ERROR::BLST_SUCCESS
}

/// Convert a list of tuples into a tuple of lists
fn unzip_signature_sets_serialized<'a>(
  sets: &'a [SignatureSetSerialized],
) -> Result<(
  Vec<&'a [u8]>,
  Vec<min_pk::PublicKey>,
  Vec<min_pk::Signature>,
)> {
  let len = sets.len();
  let mut msgs = Vec::with_capacity(len);
  let mut pks = Vec::with_capacity(len);
  let mut sigs = Vec::with_capacity(len);

  for set in sets {
    msgs.push(set.msg.as_ref());
    // pk group check is checked at consumer side if needed
    pks.push(min_pk::PublicKey::from_bytes(set.pk.as_ref()).map_err(from_blst_err)?);
    // signature validation is checked at consumer side if needed
    sigs.push(min_pk::Signature::from_bytes(set.sig.as_ref()).map_err(from_blst_err)?);
  }

  Ok((msgs, pks, sigs))
}

// this is what happens on lodestar bls worker thread
  describe("verifyMultipleAggregateSignatures - deserialize on nodejs", () => {
    for (const count of [1, 8, 32, 128, 256]) {
      itBench({
        id: `verifyMultipleAggregateSignatures deserialize on nodejs - ${count} sets`,
        beforeEach: () => arrayOfIndexes(0, count - 1).map((i) => getTestSet(i)).map((set) => ({
          msg: set.msg,
          pk: set.pk.toBytes(),
          sig: set.sig.toBytes(),
        })),
        fn: (sets) => {
          blst.verifyMultipleAggregateSignatures(sets.map((set) => ({
            msg: set.msg,
            pk: blst.PublicKey.fromBytes(set.pk),
            // true = validate signature
            sig: blst.Signature.fromBytes(set.sig, true),
          })));
        },
      });
    }
  });

  // what is supposed to change
  describe("verifyMultipleAggregateSignaturesSerialized - deserialize on native", () => {
    for (const count of [1, 8, 32, 128, 256]) {
      itBench({
        id: `verifyMultipleAggregateSignaturesSerialized - ${count} sets`,
        beforeEach: () => arrayOfIndexes(0, count - 1).map((i) => getTestSet(i)).map((set) => (
          {
            msg: set.msg,
            pk: set.pk.toBytes(),
            sig: set.sig.toBytes(),
          }
        )),
        fn: (sets) => {
          blst.verifyMultipleAggregateSignaturesSerialized(sets);
        },
      });
    }
  });

this is the result on my Mac, doing the deserialization on native is faster, and we also save a serialize() call on the main thread

    verifyMultipleAggregateSignatures - deserialize on nodejs
      ✓ verifyMultipleAggregateSignatures deserialize on nodejs - 1 sets    1506.823 ops/s    663.6480 us/op        -        305 runs  0.704 s
      ✓ verifyMultipleAggregateSignatures deserialize on nodejs - 8 sets    903.4852 ops/s    1.106825 ms/op        -        543 runs   1.11 s
      ✓ verifyMultipleAggregateSignatures deserialize on nodejs - 32 set    344.1522 ops/s    2.905691 ms/op        -        139 runs  0.916 s
      ✓ verifyMultipleAggregateSignatures deserialize on nodejs - 128 se    96.00295 ops/s    10.41635 ms/op        -         21 runs  0.733 s
      ✓ verifyMultipleAggregateSignatures deserialize on nodejs - 256 se    49.24324 ops/s    20.30735 ms/op        -         16 runs  0.837 s
    verifyMultipleAggregateSignaturesSerialized - deserialize on native
      ✓ verifyMultipleAggregateSignaturesSerialized - 1 sets                1586.143 ops/s    630.4600 us/op        -        480 runs  0.806 s
      ✓ verifyMultipleAggregateSignaturesSerialized - 8 sets                1132.611 ops/s    882.9160 us/op        -        453 runs  0.912 s
      ✓ verifyMultipleAggregateSignaturesSerialized - 32 sets               490.9918 ops/s    2.036694 ms/op        -        148 runs  0.817 s
      ✓ verifyMultipleAggregateSignaturesSerialized - 128 sets              141.8843 ops/s    7.047996 ms/op        -        119 runs   1.37 s
      ✓ verifyMultipleAggregateSignaturesSerialized - 256 sets              77.63135 ops/s    12.88139 ms/op        -         25 runs  0.854 s