Consider decoupling plugin ABI using something like capnproto

[rgleason]

Example: #400 (comment)

Quinton: It would seem preferable to have the return value distinguish between "yes", "no" and "don't know". When browsing the plugin API I noticed several methods like that which returned booleans where in fact more than two return values seemed very possible. I wonder whether we shouldn't review those holistically and try to improve the interfaces? I can imagine providing backwards compatibility by also supporting the old, lower resolution API as a shim on top of the newer higher resolution one.

Sebastien: Sure, that's why I have created OpenCPN/OpenCPN#4387. Right now there is only one API for land crossing. We need something much more extensive that covers bathymetry. For efficiency reasons, the new API should provide advanced capabilities such as the shortest path between two points.

Quinton: Above I was actually referring more to basic correctness of several existing API's not only related to collision avoidance. If an API function returns a Boolean, and can't throw exceptions, it's not possible to correctly handle various error-like situations. For example missing data, invalid parameters, or whatever.

Quinton: I was proposing adding new, correct API's for the future, and retaining the old ones for backwards compatibility.

---

Sebastien: That's my intent as well in the referred issue. The problem space is fairly complex because we have to extract features from multiple types of datasets: OSMSHP, GSHHG, ENC charts (which have multiple formats). Ideally we would even be able to extract relevant features from raster charts.

See section about "Extracting Depth Data from ENC Charts for Distance Field Implementation".

Plugins like the weather routing shouldn't have to understand the details of the polygon schemas, i.e. they should be able to query more abstract concepts like bathymetry, coastline, shortest path, etc.

I'm interested in contributions to define the apis with the right level of abstraction. I do think we should avoid a quick fix for the GSHHG land crossing API, as this would create yet another API for a niche use case, when in reality we have many more use cases to cover.

---

Quinton: I have quite a bit of experience designing and implementing API's. As an aside, I really think we should consider decoupling the ABI's better, for example with something like protobuf (or better still, capnproto). That would allow us to add fields to both the requests and responses, without breaking either clients or servers.

---

Sebastien: In principle yes, but consider this: weather routing needs GRIB data. Currently, because the code is structured as independent modules, there is no memory sharing of the GRIB data, which means the same GRIB data is duplicated in the GRIB plugin and the weather routing plugin. It would be good for performance reasons to have a shared memory model.

Quinton: I agree that shared memory would be very useful in several different contexts. Using something like protobuf does not at all preclude the use of shared memory. It only defines the interface through which one reads and writes chunks of data (which can be in shared memory, or a file, or a network stream).

---

Rick: The level of complexity to the full route checking routine is quite high. You both appear to want to solve the whole thing, which is admirable, but I think you will need any resource tools that make it easier to iterate as your chances of achieving the goal will be much better. ...someone is going to need to move this closer to a real proposition to the program team.
Protobuf
capnproto

Quinton: Don't worry Rick, we are still just tossing ideas around. Once we have identified the highest priority ideas, and the dependencies between them, we can compose a real set of proposals, that people can either shoot down, or carve up into logical pieces for implementation.

---

---

capnproto documentation: #400 (comment)

Sebastien: Are you proposing to run plugins as a separate processes? #400 (comment)

Quinton: What I am proposing is that we create a freedom to add optional fields to both requests and responses between plugins and OCPN-core over time. This could ensure, for example, that an old non-recompiled client plugin could read a new core data structure. It would of course only be able to read the old fields, but new plugins would be able to read the new fields. The same goes for request parameters. Additional optional parameters could be added. Old plugins would not specify the optional parameters, the server would receive substituted default values, and life is good.

That's what something like capnproto buys you. It's orthogonal to whether the client and server are in the same or different processes. And orthogonal to whether they are communication using files, shared memory or sockets.