Add store metrics

[canton-network-da]

[ This issue was auto-migrated from DA's internal repo (DACH-NY/canton-network-node#10879). Original author: @cocreature ]

A few obvious ones that come to mind:

• Number of Daml transactions inserted
• Number of events inserted
• Average time to process a Daml transaction.
• DACH-NY/canton-network-node#14128
• Ideally also some sort of saturation, maybe we can just slap a bounded queue in front of ingestion and measure the size of that to see whether we can keep up?

[canton-network-da]

[ from @cocreature ]: @OriolMunoz-da @rautenrieth-da do these metrics seem sensible? anything else that you think we should cover short term?

[canton-network-da]

[ from @OriolMunoz-da ]: Maybe also these:

• Number of ACS deletions
• Insertions/deletions from assign/reassign tables

And for inspiration, this is everything we track in the ingestion summary:

final case class IngestionSummary(
    updateId: Option[String],
    offset: Option[String],
    newAcsSize: Int,
    ingestedCreatedEvents: Vector[CreatedEvent],
    numFilteredCreatedEvents: Int,
    ingestedArchivedEvents: Vector[ExercisedEvent],
    numFilteredArchivedEvents: Int,
    updatedContractStates: Vector[ContractStateEvent],
    addedAssignEvents: Vector[(ContractId[?], ReassignmentId)],
    numFilteredAssignEvents: Int,
    removedAssignEvents: Vector[(ContractId[?], ReassignmentId)],
    addedUnassignEvents: Vector[(ContractId[?], ReassignmentId)],
    numFilteredUnassignEvents: Int,
    removedUnassignEvents: Vector[(ContractId[?], ReassignmentId)],
    prunedContracts: Vector[ContractId[_]],
    ingestedTxLogEntries: Seq[String],
)

[canton-network-da]

[ from @meiersi-da ]: > Ideally also some sort of saturation, maybe we can just slap a bounded queue in front of ingestion and measure the size of that to see whether we can keep up?

You should get a decent saturation metric from adding a counter measuring the time spent on ingesting transactions. You can then just take the rate of of that counter, and once it is at 1, you know that you are spending a 100% of your available time in the sequential insertion bottleneck.

The other metric we likely want is a gauge for record_time of the last tx received from the server. This will allow checking catchup speed and lag of the tx tree stream.

[canton-network-da]

[ from @meiersi-da ]: > The other metric we likely want is a gauge for record_time of the last tx received from the server. This will allow checking catchup speed and lag of the tx tree stream.

Thinking more about this: it might make sense to compute the lag on the app backend against it's own wall clock reading right upon ingestion. Otherwise, our measurements are no more precise than the Prometheus scraping interval. For latency investigations, I'd expect us to want less than 15s of noise/artificial clock jitter.

[canton-network-da]

[ from @cocreature ]: > You should get a decent saturation metric from adding a counter measuring the time spent on ingesting transactions. You can then just take the rate of of that counter, and once it is at 1, you know that you are spending a 100% of your available time in the sequential insertion bottleneck.

ah right, that sounds good.

Thinking more about this: it might make sense to compute the lag on the app backend against it's own wall clock reading right upon ingestion. Otherwise, our measurements are no more precise than the Prometheus scraping interval. For latency investigations, I'd expect us to want less than 15s of noise/artificial clock jitter.

yeah makes sense. It's a bit annoying because you can't distinguish between the participant lagging behind and our apps ingestion lagging behind because the timestamp comes from the sequencer. 15s might be too slow there (definitely too slow given current Canton performance).

[canton-network-da]

[ from @cocreature ]: > Number of ACS deletions

wouldn't we get that from ACS size?

Insertions/deletions from assign/reassign tables

makes sense but happy to do that post mainnet given that we're not focusing on multi-domain.

[canton-network-da]

[ from @cocreature ]: > wouldn't we get that from ACS size?

ah I guess you want to distinguish no atciviyt from lots of archive/create makes sense

[canton-network-da]

[ from @rautenrieth-da ]: Would the number of UpdateIngestionService instances running be useful? Or does the participant already expose the number of open update streams?

The other metric we likely want is a gauge for record_time of the last tx received from the server.

Also the same thing for the first ingested update, after we implement backfilling data or repairing the TxLog, but that's probably not "short term".

[canton-network-da]

[ from @cocreature ]: > Would the number of UpdateIngestionService instances running be useful? Or does the participant already expose the number of open update streams?

I would have just labeled them with the respective store key. That way we can aggregate in prometheus if needed but we can also drill down into individual stores to see, e.g., which user suddenly gets a lot of CC transactions.