Module Creation

##Module Creation

Modules are the main concept in this platform. They are self contained, processing components which can do many operations, data conversion, data analysis, perform graph operations or machine learning tasks. A module is a self-contained unit of annotated code so that the platform can run it. The method of annotation is specified by the domain.

These modules will be treated as black boxes which can be composed from either existing Java or Python code or any other executable that can be wrapped as a module. Modules should accept input data, parameters, a reference to the platform context where it will be executed and will produce output data. For Java and Python we are using annotation to convert existing code base as module.

Module Domain

A domain describes a programming language and its associated information. An example of a domain is the Java domain. This is the collective name for the method of annotating Java code, the Java type system, the way raw values are converted to Java objects and so on. Some domains do not describe a traditional programming language, but just an execution environment (for instance, the command line domain).

####Java Domain

Module in Java domain is created by annotating existing Java Class. @Module annotation is used to denote the class which will be used as a module. Furthermore, one of the methods defined in this class will be considered as the @Main method to be executed, thus annotated as @Main. This @Main annotation can also refer to constructor of a Java class.

To identify inputs to a module, we can use @In annotation to mark a class field or a function parameter as an input of a module. Example of input annotations are as follows:

####Input Annotation on constructors

@Module(name="LibLINEARParms")
public class LibLINEARParmsModule {
    public LibLINEARParmsModule(
            @In(name="cs") List<Double> cs, 
            @In(name="nrFolds") int folds, 
            @In(name="target") List<Double> target
            ) {
        [...]
    }
}

####Input annotation on fields @Module(name="Adder", description="Adds two numbers together.") public class AdderModule { @In(name="first") public Integer first; @In(name="second") public Integer second }

Output is annotated in the similar manner as inputs. One or more method can be declared as an output of the module. In the following we can see sample of output annotations either on existing public methods or public fields of a class.

####Output annotation on public methods or fields

@Out(name="accuracy")
public double getAccuracy() {
    [...]
}

@Out(name="weights")
public double[] weights;

###Python Module

For Python modules, we are using Python decorator to annotate existing methods/function as a module. The decorator is defined in ducktape.py which needs to be imported when defining a module. In Python domain, we have only one decorator @ducktape.main(input_types, ..., output_type) to indicate that a Python function is a module.

####Example Python Module import ducktape from rdflib import Graph, URIRef

@ducktape.main("str", "int")
def countTriples(resourceURI):
    g = Graph()
    g.parse(resourceURI)
    return len(g)

Int the above function, countTriples is converted as a module accepting strings, as input type and integer as output type. The output name will automatically named countTriples the same name as the method name.

###Command Line Module

For command line domain, we assume an existing code runnable from command line. To convert this existing command line code as a module we need an additional configuration file which will describe a required input and output of the command line script.

Assuming an existing script which adds two integers, for example a configuration file will be a YAML file which indicates input and outputs name, type and descriptions.

####Command line configuration example. name: Simplest command line

descriptions:
    Basic arithmetic with two inputs from command line

inputs:
    - name: first
      type: integer
      description: this is the first input
    - name: second
      type: integer
      description: this is the second input
      
outputs:
    - name: result
      type: integer
      description: default result for this module which is now the same as produc
               
command:
    src/test/resources/commandLine/arith.sh