Field/node dependency graph

[DeMol-EE]

In certain cases (specifically: large award/result notices), our platform sometimes struggles with performance. Through profiling, we’ve established that the culprit is the execution of EFX (in our case: transpiled to XPath). The time to evaluate any one expression is small, but our implementation suffers from the problem that we re-evaluate all expressions of every field after any change to the form. That is of course largely unnecessary, as the vast majority of the form will not be impacted by a single field being edited, but the reason we re-evaluate everything is because we are missing a structured way to determine which fields are related to each other.

We figure we could derive a dependency graphs between the fields from the EFX expressions to optimize our platform, but we were curious whether OP has already considered building something similar (it presumably being easier starting from the base data that is used to also generate fields.json itself and the EFX expressions, rather than reverse engineering it from them).

The way we envisage it, it could take the form of a new property "related_fields" in each field in fields.json, having an array of field ids as value (which could be empty).

With such an addition, we presume we could drastically improve the efficiency (and thereby the performance) of our form as it would lower the algorithmic complexity of our engine from linear to constant.

To put this in numbers: we recently had a notice where each user interaction would trigger ~7000 EFX/XPath evaluations. If we’d have a dependency graph, then touching any field should trigger no more than a few dozen or so evaluations (assuming the dependency graph is sparse).

Should OP not be able to provide such a dependency graph, does it seem like a good approach? Has anyone considered something similar and if so, was it a success?

[rousso]

Well Robin @DeMol-EE,
The idea has crossed my mind many times, but I have nothing planned so far because it would require a change in fields.json which I was reluctant to introduce before SDK 2. But yes, the idea is good and now that I see that it bothers you too, I will certainly add this in SDK 2.

Instead of adding this to fields.json it might also be a better idea to consider a separate file for a dependency graph. Do you have any ideas on how such file would look like already?

Thanks for raising this,
Yannis

[DeMol-EE]

Thanks for getting back to me @rousso. A separate file would of course also work. I would try to keep the structure as simple as possible. Perhaps just a CSV/TSV file with 2 columns to represent fields that are related to one another?