Provide functions in ModelicaUtilities.h to external objects implemented as shared libraries

[t-sommer]

Currently it is not possible to access the functions in ModelicaUtilities.h from an ExternalObject that is implemented as a shared library in a generic way. Therefore I propose to introduce an ExternalObject

class ModelicaUtilityFunctions
  extends ExternalObject;

  function constructor
    output ModelicaUtilityFunctions functions;
  end constructor;

  function destructor
    input ModelicaUtilityFunctions functions;
  end destructor;

end ModelicaUtilityFunctions;

into the MSL that is implemented by the tools and returns a pointer to the following structure.

#include "ModelicaUtilities.h"

typedef struct {

	void  (*ModelicaMessage)(const char *string);
	void  (*ModelicaFormatMessage)(const char *string, ...);
	void  (*ModelicaVFormatMessage)(const char *string, va_list);
	void  (*ModelicaError)(const char *string);
	void  (*ModelicaFormatError)(const char *string, ...);
	void  (*ModelicaVFormatError)(const char *string, va_list);
	char* (*ModelicaAllocateString)(size_t len);
	char* (*ModelicaAllocateStringWithErrorReturn)(size_t len);
        // ...

} ModelicaUtilityFunctions_t;

This pointer can then be used by external objects to access these functions.

class MyExternalObject
  extends ExternalObject;

  function constructor
    input ModelicaUtilityFunctions callbacks;
    output MyExternalObject externalObject;
  external"C" externalObject =
        MyExternalObject_open(callbacks) annotation (
    Library="MyExternalObject");
  end constructor;

  function destructor
    input MyExternalObject externalObject;
  external"C" MyExternalObject_close(externalArduino) annotation (
    Library="MyExternalObject");
  end destructor;

end MyExternalObject;

[casella]

@t-sommer this is a good proposal to address a long-standing problem, see modelica/ModelicaSpecification#2191. We proposed an alternative solution there a few months ago, which just requires all Modelica tools and simulation runtimes to export the symbols of those functions, so they can be dynamically linked both at compile time and at run time. Unfortunately there was no feedback from tool vendors (other than OMC, that is) back then.

This already works in OpenModelica, we implemented and tested it in the ExternalMedia library. Unfortunately Dymola.exe does not export those symbols, so if some external functions are called during translation and they give an error, there is no way to properly report it. Hence, we conjured up a hack to pass those pointers to ExternalMedia static interface.

I'm not sure which of the two solutions is better. Our proposal requires to slighly change how you building the Modelica compiler and the simulation runtime, all tool vendors could do it right away. Yours is easier to understand for non-hackers like myself, but it requires an addition to ModelicaServices, so it could be released with 4.2.0 sometime next year.

For sure we should go ahead and implement one of these two ASAP.

BTW, this is yet another issue at the boundary between MAP-Lang and MAP-Lib.

@fedetftpolimi what do you think?

[fedetftpolimi]

From a quick check it should allow both static and dynamic libraries to access the pointers to the modelicaUtilities functions, so from a C/C++ perspective it seems good.

However, the example is geared to external libraries based on external objects, for this approach to be complete is should also allow libraries based solely on external functions (such as ExternalMedia) to get the pointers.

This could be as simple as allowing to declare an external function taking the parameter
input ModelicaUtilityFunctions callbacks;

[t-sommer]

This could be as simple as allowing to declare an external function taking the parameter
input ModelicaUtilityFunctions callbacks;

Passing instances of external objects to functions is already allowed today, so this should not require any changes to the Modelica language.

[fedetftpolimi]

@casella By the way, the two proposal are also not mutually exclusive. We may even support both allowing to retrieve the functions as symbols by the linker or pointers through the struct, even though there's no need to standardize both.

In any case, what we need is at least one solution implemented by all tool vendors asap (and one that isn't just "use static libraries" because that's a dead end when you account for dependencies).

[beutlich]

For the sake of completeness:

• ModelicaUtilityFunctions_t of @t-sommer misses the functions for ModelicaDuplicateString and ModelicaDuplicateStringWithError.
• Mo-2191 holds more than the one proposal mentioned by @casella, e.g., by providing a specific shared object (named ModelicaExternalC) at run-time (which is implemented in SimulationX).

[fedetftpolimi]

The proposal in modelica/ModelicaSpecification#2191 we were referring to is the one @casella posted on on May 22, 2024: exporting symbols to allow automatic lookup by the dynamic linker/loader.

The idea of providing a tool-specific shared object as far as I know is not portable, and it forces C/C++ library developers to provide different versions of their library differing only by each version being linked with the shared object of a particular vendor, which does not scale well.
E.g: the ExternalMedia build system does not provide versions of the library linked with the SimulationX shared object.

[casella]

• ModelicaUtilityFunctions_t of @t-sommer misses the functions for ModelicaDuplicateString and ModelicaDuplicateStringWithError.

You're right, see Section 12.9.6.2 they were added in Modelica 3.5.

@t-sommer you may edit your original proposal to keep it consistent to the latest standard.

[casella]

@HansOlsson the related issue modelica/ModelicaSpecification#2191 has been open for 7 years. I understand we now have two good proposals to fix it, so I would put it in the agenda of the MAP-Lang group. I have a slight preference for @t-sommers's proposal since it relies less on low-level C-code functionality (importing symbols in the external C functions) and uses explicitly defined Modelica functionality instead, but I'm not 100% sure it always works (see below).

Besides that, @t-sommer's proposal requires changes to the language spec (regarding the contents of ModelicaUtilities.h) and to the MSL (adding the ModelicaUtilityFunctions to ModelicaServices library), while mine only requires to change the language spec. But I don't see that as a big deal, with a modicum of coordination between the two groups 😃

[casella]

Passing instances of external objects to functions is already allowed today, so this should not require any changes to the Modelica language.

@t-sommer maybe @fedetftpolimi has a point. It is not clear to me where the external object should be instantiated in some cases. Consider for example this MWE (which is relevant for ExternalMedia):

package MWE
  partial function baseFunction
    input Real x;
    output Real y;
  end baseFunction;

  function function1
    extends baseFunction;
  algorithm
    y := 2*x;
  end function1;

  function function2
    extends baseFunction;
  external;
  end function2;

  model M
    replaceable function f = function1 constrainedby baseFunction;
    Real y = f(time);
  end M;
end MWE;

Now, we want the external function2() to be able to call ModelicaError. Where would we instantiate ModelicaUtilityFunctions so that we can pass the pointer to it to the function?

[casella]

One idea could be to amend the specification of the external function interface by defining a pointer to an implicitly defined default instance modelicaUtilityFunctions of the ModelicaUtilityFunctions pre-defined external object, so you could write something like

  function function2
    extends baseFunction;
  external "C"
    y = extFunction2(x, modelicaUtilityFunctions.functions);
  end function2;

double extFunction2(double x, void *callbacks)
{
  if (x > 0)
    return 2*x;
  else
  (ModelicaUtilityFunctions_t *)callbacks().ModelicaError("Negative input not allowed");
}

without the need of explicitly instantiating the modelicaUtilityFunctions external object. I'm not sure about the C syntax, but I guess the intent is clear, please correct it if necessary.

As I understand, that's the only way (besides exporting and importing symbols) to handle utility functions in purely function-based implementations, such as is the case of the setState_xx() functions in Modelica.Media. It would also be 100% backwards compatible with the existing external function interface.

[HansOlsson]

Please try to avoid using void*.

To me the current C-calling interface for Modelica was designed to interface existing C-code, and then people wrote "small" C-glue functions (or what-ever you call them) to handle other data-types etc. The ModelicaUtilities header works for the glue-functions, but not when building larger libraries specifically for Modelica-libraries.

Obviously it is still possible to create glue-functions for a Modelica-library, but it becomes cumbersome to use it every time:

function function2
    extends baseFunction;
  external;
  annotation(Library="...",, Include="
static void function2Print(const char*x) {
   ModelicaFormatMessage(\"From function2:%s\", x);
}
extern double function2External(double x, void (*f)(const char*));
double function2(double x) {
  return function2External(x,  function2Print);
}"
  end function2;

[HansOlsson]

Oh, I looked more and ExternalMedia is written in C++ and uses ModelicaError.

That C/C++ interfacing is a real problem, that requires a different good solution which is completely different from what is discussed here.