RDF-Lens

Look into a dataset with RDF-Lens. Here the term Lens, is the same concept of Haskell lenses available in this well known library.

A lens looks at a point in a dataset, for RDF-Lens this usually is looking at a Term inside a store. Lenses can be combined to from a point, look furthur inside the dataset, as long as the types allow for it.

Currently two lenses exist in RDF-Lens BasicLens<C, T> and BasicLensM<C, T> (M stands for Multiple: BasicLensM<C, T> is a subclass of BasicLens<C, T[]>). C stands for Container, a combination of the store and the current target. T stands for resulting type.

BasicLens<C, T> can be combined with BasicLens<T, D> and will result in BasicLens<C, D>.

Examples

Pred

A very common Lens is created with pred and takes in a predicate term and returns BasicLensM<Cont, Cont>. So it starts pointing inside the dataset and results in pointing to multiple things inside the dataset. Using the function .thenFlat another predicate can be chained to explore the dataset deper.

Extracting data

RDF-Lens tries to make it developer friendly and typed to extract data from a RDF store. This example shows how to extract a point with two coordinates from the data.

import { pred } from "rdf-lens";
import { DataFactory, Parser } from "n3";

const { namedNode } = DataFactory;

const extractX = pred(namedNode("x"))         // Follow predicate <x>
  .one()                                      // We expect only to find one term
  .map(({id}) => ({x: id.value}));            // Map that term to a Json Object
  
const extractY = pred(namedNode("y"))         // Follow predicate <y>
  .one()                                      // We expect only to find one term
  .map(({id}) => ({y: id.value}));            // Map that term to a Json Object
  
const pointLens = extractX.and(extractY)      // Combine both lenses
  .map(([{x}, {y}]) => ({x, y}));             // Map them together to a point object

const turtle = "<a> <x> 42; <y> 43.";
const quads = new Parser().parse(turtle);     // Parser quads 
const point = pointLens.execute({id: namedNode("a"), quads});  // Execute the lens over the dataset

Extracting RDF list items

Extracting point data is not very exciting, this example shows how to extract all items from a RDF List.

const RDFListElement = pred(RDF.first).one().and(pred(RDF.rest).one());

// RdfList is a Lens that takes in a Container pointing to a Term and returns a list of Terms 
const RdfList: BasicLens<Cont, Term[]> = new BasicLens((c) => {
  if (c.id.equals(RDF.nil)) {
    return [];
  }

  const [first, rest] = RDFListElement.execute(c);
  const els = RdfList.execute(rest);
  els.unshift(first.id);
  return els;
});

Extracting starting from shacl shapes

Shacl shapes are used widely to constrain rdf data to some shape. With rdf-lens you can extract data starting from a shape to a plain old javascript object. The field names are defined by the sh:name, that is part of the sh:property object.

This examples shows how to define and extract a point.

# The shacl shape for a point
[] a sh:NodeShape;
  sh:targetClass <Point>; # Derive a lens for js:Point
  sh:property [
    sh:name "x";             # Field x
    sh:path <x>;            # is found at path `js:x`
    sh:datatype xsd:integer; # and is an integer
    sh:maxCount 1;
    sh:minCount 1;
  ], [
    sh:datatype xsd:integer;
    sh:path <y>;
    sh:name "y";
    sh:maxCount 1;
    sh:minCount 1;
  ].

# Data that adheres to that shape
<MyPoint> a <Point>;
  <x> 5;
  <y> 8.

Let's use this data to extract a point.

const shapes = extractShapes(shapeQuads);
const quads = parseQuads(dataQuads);

const lens = shapes.lenses["Point"]; // The lens that extracts a point
const point = lens.execute({id: namedNode("MyPoint"), quads});

console.log(point); // { "x": 5, "y": 8 }

Deep objects are also supported, let's reuse the point shape to extract a line.

[] a sh:NodeShape;
  sh:targetClass <Line>;
  sh:property [
    sh:name "start";  // The start is a point
    sh:path <start>;
    sh:class <Point>;
    sh:maxCount 1;
    sh:minCount 1;
  ], [
    sh:name "end";    // The end is a point
    sh:path <end>;
    sh:class <Point>;
    sh:maxCount 1;
    sh:minCount 1;
  ].

Note: sh:datatype is used for literals, sh:class is used for objects, and sh:nodeKind is used to take a term as it is.

• sh:minCount tells rdf-lens that this property is required, and will fail to parse an object that does not adhere to the shape.
• sh:maxCount tells rdf-lens whether or not to expect multiple objects. If this is not set or is bigger than 1, the Javascript object will have an array as its value.
• sh:defaultValue tells rdf-lens what to use when the data has no value for this property. A default satisfies sh:minCount, so a required property with a default never fails to parse.

Default values are extracted the same way the data would have been, but from the shapes graph:

[] a sh:NodeShape;
  sh:targetClass <Me>;
  sh:property [
    sh:name "name";
    sh:path <name>;
    sh:datatype xsd:string;
    sh:defaultValue "ajuvercr";   # A literal is converted with sh:datatype
    sh:maxCount 1;
    sh:minCount 1;
  ], [
    sh:name "age";
    sh:path <age>;
    sh:datatype xsd:integer;
    sh:maxCount 1;
    sh:minCount 1;
  ].

<foobar> a <Me>;
  <age> 95.

extracts to { "name": "ajuvercr", "age": 95 }.

For a sh:class property the default is a node, and it is extracted through that class's shape. Its own properties are read from the shapes graph, and whatever it leaves out falls back to the defaults of that shape in turn:

[] a sh:NodeShape;
  sh:targetClass <Friend>;
  sh:property [
    sh:name "friend";
    sh:path <friend>;
    sh:class <Me>;
    sh:defaultValue [ <age> 95 ];   # <name> comes from the Me shape default
    sh:maxCount 1;
    sh:minCount 1;
  ].

<foobar> a <Friend>. extracts to { "friend": { "name": "ajuvercr", "age": 95 } }. The same happens one level down: a friend that is present in the data but has no <name> still gets "ajuvercr".

For a property that holds multiple values, the default may be a single value or an RDF list, and it is only used when the data has no values at all.

A default is held to sh:nodeKind and converted with sh:codeType like a value from the data (see below), so sh:nodeKind sh:IRI; sh:defaultValue "a literal" fails to parse when the default is used.

Extracting terms with sh:nodeKind and sh:codeType. Sometimes the term itself is what you want, not a converted literal or a nested object. sh:nodeKind says which kind of term a property holds, sh:codeType what code receives. sh:codeType comes from SHACL 1.2 Core: validation ignores it, so it can differ from the constraints. A property written as an IRI, so the parser resolves it against the base, can still be read as a string:

[] a sh:NodeShape;
  sh:targetClass <Document>;
  sh:property [
    sh:name "source";
    sh:path <source>;
    sh:nodeKind sh:IRI;        # written as <./data.ttl>, resolved by the parser
    sh:codeType xsd:string;    # read as "file:///.../data.ttl", leave out for a NamedNode
    sh:maxCount 1;
  ].

• All six node kinds are supported: sh:IRI, sh:BlankNode, sh:Literal, sh:BlankNodeOrIRI, sh:BlankNodeOrLiteral and sh:IRIOrLiteral. A value of another kind fails to parse, like a cardinality violation, with an error naming the property and the value. The kind is also checked next to sh:class.
• Either can be used alone: sh:nodeKind hands you the term, sh:codeType converts any term (rdf-lens warns when that is a blank node, which only has its label to convert).
• sh:codeType is one of xsd:string, xsd:integer, xsd:decimal, xsd:float, xsd:double, xsd:boolean, xsd:dateTime, xsd:anyURI or rdfl:Term. xsd:anyURI gives an IRI, from an IRI or from a string holding an absolute IRI, so sh:nodeKind sh:IRIOrLiteral; sh:codeType xsd:anyURI accepts <https://example.org> and "https://example.org" alike. A string needs a scheme of two or more characters and is not a host and port, so C:\data and localhost:8080 are no IRIs.
• sh:codeType rdfl:Term converts nothing: it hands back the term the parser produced, instead of rebuilding it with this library's data factory. rdfl:Term is an rdf-lens extension, not a datatype.
• Any other sh:codeType, a node kind, xsd:iri or a typo like xsd:strin, and an unknown sh:nodeKind, log a warning when the shape is read, and fail once a value is extracted.
• An rdfl:EnvVariable holds a string, so it becomes an IRI when the kind allows one and the value is an absolute IRI, a literal when the kind allows that, and an error otherwise.
• An rdfl:EnvVariable whose variable is not set and that has no rdfl:envDefault fails to parse, with or without a node kind. envReplace puts a term in its place: an IRI for rdfl:datatype xsd:anyURI or xsd:iri, a literal of its rdfl:datatype otherwise, and a plain string literal without one.

Which one applies. The first of these a property has decides how it is extracted:

1. sh:codeType: the term, checked against sh:nodeKind and converted with the code type. It goes before sh:class, as SHACL 1.2 asks tools to prefer it over the type they would derive from the constraints, so sh:class ex:Pipeline; sh:codeType xsd:anyURI gives the IRI of the pipeline instead of a nested object. sh:datatype describes the data, so it may differ: sh:datatype xsd:integer; sh:codeType xsd:string reads 4 as "4".
2. sh:class: a nested object, built with the shape of that class.
3. sh:nodeKind: the term, checked against the kind, and converted with sh:datatype when one is given (sh:datatype xsd:iri is read as xsd:anyURI here). This warns, suggesting sh:codeType instead: for sh:datatype xsd:iri when the shape is read, for a converting datatype such as xsd:string the first time it is given an IRI or a blank node.
4. sh:datatype: the value converted with that datatype.

Warning

sh:datatype describes literals, but rdf-lens used it on any term, so sh:datatype xsd:string turns IRIs and blank nodes into strings and sh:datatype xsd:iri (not an actual datatype) turns any term into an IRI. Both still work, but log a warning naming the property, its path and its shape, the first time they are given a term they should not describe, once per kind of term, as the replacement depends on it:

Old	Given	Replacement
sh:datatype xsd:iri	an IRI	sh:nodeKind sh:IRI, the same term
sh:datatype xsd:iri	a string	sh:nodeKind sh:IRIOrLiteral; sh:codeType xsd:anyURI, the same IRI for an absolute IRI
sh:datatype xsd:iri	a blank node	sh:nodeKind sh:BlankNodeOrIRI, the blank node itself instead of an IRI named after its label
sh:datatype xsd:string	an IRI	sh:nodeKind sh:IRIOrLiteral; sh:codeType xsd:string (or sh:IRI if only IRIs are expected), the same string
sh:datatype xsd:string	a blank node	sh:nodeKind sh:BlankNodeOrLiteral; sh:codeType xsd:string (or sh:BlankNode if only blank nodes are expected), the same label

Other datatypes rdf-lens converts, like xsd:integer, warn the same way as xsd:string, with their own code type. A datatype it does not convert, like rdf:langString, hands back the term and does not warn.

Warnings go to console.warn unless you route them elsewhere, a logger for example, with setWarningHandler((message) => logger.warn(message)). The handler also receives the property shape a warning is about, and setWarningHandler returns the handler it replaces, to put back afterwards. A handler that does nothing silences them, and setWarningHandler() restores console.warn.

Checking shapes. rdf-lens check reads shapes from Turtle or TriG files, and the files they owl:imports, and prints what rdf-lens warns about and what fails, without running anything:

npx rdf-lens check processor.ttl              # the shapes alone
npx rdf-lens check examples/pipeline.ttl      # data using them, with the shapes it imports

warning  property "url" (path <https://example.org/url>, shape <https://example.org/Fetch>) uses sh:datatype xsd:iri, which is deprecated: use sh:nodeKind sh:IRI instead, it extracts the same term.
         seen on <file:///.../examples/fetch>
error    <file:///.../examples/broken>: Field is not defined and required, in field "url"

Checked 1 shape on 2 nodes: 1 warning, 1 error.
Not checked, no shape describes them: <https://example.org/Writer>

• Without data it finds shapes that cannot be read, naming every property at fault, an unsupported sh:codeType or sh:nodeKind, and sh:datatype xsd:iri.
• With data it also extracts every node typed with a target class, or a subclass of one, and finds what depends on the data: sh:datatype given IRIs or blank nodes, the exact replacement for sh:datatype xsd:iri, and values that do not fit their shape. Pass example data, a pipeline using a processor for example, next to the shapes.
• A class used with sh:class that no shape describes, like one a framework provides a lens for, is taken as is and listed as not checked.
• It exits with 1 when it finds something, so it fits in CI, and with 2 when it cannot run. --no-imports reads only the given files.

The same check is available in code, to run in a test for example: checkShapes(quads) returns the findings, each with its level, message, property shape and the data node it was found on.

Special implemented classes Sometimes a plain old javascript objects is not enough, some special classes work out of the box. @prefix rdfs: <https://w3id.org/rdf-lens/ontology#>.

• rdfl:CBD: Provides a list of quads bounded by the cbd algorithm.
• rdfl:PathLens: Parses a shacl Path and returns a Lens that resolves this path.
• rdfl:Context: Provides a reference to the list of all data quads.
• rdfl:TypeExtract: Extracts according to the rdf:type object (including class hierarchy), by using the shape that corresponds to that type.