optimized filtering

[nvdk]

builds on #20, but has some optimizations wrt to filtering:

• by grouping services on match pattern: this means we only do detection and filtering once for a given match pattern (takes the pattern and options.sendMatchesOnly into account as a grouping key)
• by doing pattern matching before origin filter, since the latter seems more costly

edit: can be tested using nvdk/mu-delta-notifier:feature-optimized-filtering. see https://github.com/lblod/app-lblod-harvester/pull/79/files for an example

[nvdk]

I kept this, but I don't think this situation can occur.

[x-m-el]

Some optimizations are possible. I added comments in adjusted code below, because the suggestions are interlinked with each other, so separate comments could become confusing:

async function informWatchers( changeSets, res, muCallIdTrail, muSessionId ){
  /* This code used to be inside the full loop, but this is not needed. 
   * If the entry has `sendMatchesOnly` set to true, using changedTriples can be avoided (see below).
   * If `sendMatchesOnly` is set to false, the code using `changedTriples` will always need all triples
   * (because there is no extra "filtering").
   * so there is no need to recalculate it every time.
   */
  let allInserts = [];
  let allDeletes = [];
  changeSets.forEach( (change) => {
    allInserts = [...allInserts, ...change.insert];
    allDeletes = [...allDeletes, ...change.delete];
  } );
  const changedTriples = [...allInserts, ...allDeletes];

  // Iterate over each unique match pattern
  for (const matchKey in groupedServices) {
    const firstEntry = groupedServices[matchKey][0];
    // can use first entry since it's part of grouping
    const sendMatchesOnly = firstEntry.options.sendMatchesOnly;
    let maybePatternFilteredChangesets = changeSets;
    if (sendMatchesOnly) {
      maybePatternFilteredChangesets = filterChangesetsOnPattern(changeSets, firstEntry);
      /* Currently, fully empty change sets are send too, but I don't think 
       * this is desired if `sendMatchesOnly` is set to true?
       * Note: can also do this filtering in `filterChangesetsOnPattern` directly
       * (by having an if-case or similar around `filteredChangesets.push(clonedChangeSet);`)
       */
      maybePatternFilteredChangesets = maybePatternFilteredChangesets.filter(
         (change) =>
           change.insert.length > 0 ||
           change.delete.length > 0 ||  change.effectiveInsert.length > 0 ||
           change.effectiveDelete.length > 0
       );
    }

    /* `filterChangesetsOnPattern` already does all the filtering for matches,
     * so if `sendMatchesOnly` is true, we can just check if there are still
     * non-empty change sets left to send. If all are empty (or none are left,
     * in case of filtering them away as stated above), there are no matches with the spec.
     */
    const someTripleMatchedSpec = sendMatchesOnly
      ? maybePatternFilteredChangesets.length > 0 /* makes the assumption that maybePatternFilteredChangesets has no empty change sets. */
      : changedTriples.some((triple) =>
          tripleMatchesSpec(triple, firstEntry.match)
        );
  
    /* The if can be taken out of the forEach loop, to avoid the loop all together
     * if no triples will be send out. Note that `DEBUG_TRIPLE_MATCHES_SPEC` should
     * be moved before the `if` case then. However, this does change the debug output:
     * before it would print a line per match (true/false), now it will do this
     * per service group. The difference is negligible, so this change could be skipped.
     */ 
  
    if( someTripleMatchedSpec ) {
    const matchingServices = groupedServices[matchKey];
      matchingServices.forEach( async (entry) => {
   /// rest of the code
   });
   }

[x-m-el]

unused code

Suggested change

const matchSpec = entry.match;