Skip to content

[OSSIE][DATABRICKS] Add Java Converter and Reformat #333

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
Haoranli503 wants to merge 4 commits into
base: main
Choose a base branch
from
Open

[OSSIE][DATABRICKS] Add Java Converter and Reformat #333

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
9182ff0
[OSSIE] Add Java converter and restructure converters/databricks
Haoranli503 Aug 19, 2026
7b4f765
[OSSIE] Mark the Python converter as deprecated in favor of Java
Haoranli503 Aug 19, 2026
5572106
[OSSIE][DATABRICKS] Fix Java converter review findings
jbonofre Aug 27, 2026
4fcc57a
Address internal bug bash feedbacks
Haoranli503 Aug 28, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 25 additions & 3 deletions .github/workflows/converter-databricks-ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,11 +32,14 @@ on:
- '.github/workflows/converter-databricks-ci.yml'

jobs:
build:
python:
runs-on: ubuntu-latest
strategy:
matrix:
python-version: ["3.11", "3.12", "3.13", "3.14"]
defaults:
run:
working-directory: converters/databricks/python

steps:
- name: Checkout project
Expand All @@ -53,11 +56,30 @@ jobs:
echo "${HOME}/.local/bin" >> "${GITHUB_PATH}"

- name: Sync dependencies
working-directory: converters/databricks
run: |
uv sync

- name: Unit Tests
working-directory: converters/databricks
run: |
uv run pytest

java:
runs-on: ubuntu-latest
defaults:
run:
working-directory: converters/databricks/java

steps:
- name: Checkout project
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1

- name: Set up JDK 21
uses: actions/setup-java@03ad4de0992f5dab5e18fcb136590ce7c4a0ac95 # v5.6.0
with:
java-version: '21'
distribution: 'temurin'
cache: 'maven'

- name: Build and test
run: |
mvn -B verify
114 changes: 13 additions & 101 deletions converters/databricks/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,108 +17,20 @@
under the License.
-->

# Apache Ossie Databricks Converter
Ossie <-> Metric View converters
=================================

Bidirectional, offline conversion between an [Apache Ossie](https://github.com/apache/ossie)
semantic model and a Databricks
[Unity Catalog Metric View](https://docs.databricks.com/aws/en/metric-views/) (YAML
`1.1`). No Databricks connection required.
A bidirectional converter between [Apache Ossie](https://github.com/apache/ossie) semantic models
and Databricks Unity Catalog Metric Views (YAML v1.1). Conversion is pure YAML text in, YAML text
out: it reads and writes the two formats as parsed maps and lists, independent of any engine.

- **Export** (`ossie-databricks export`): Apache Ossie -> Metric View (one fact
`source` with a nested `joins` tree and a flat `dimensions` list).
- **Import** (`ossie-databricks import`): Metric View -> Apache Ossie. Metric View features Apache Ossie has
no native field for are preserved in `custom_extensions[DATABRICKS]`, so
`MV -> Apache Ossie -> MV` is lossless.
Layout
------

On **export** (Apache Ossie -> Metric View), Apache Ossie features with no Metric View slot -- relationship
`ai_context`, `dimension.is_time`, non-`DATABRICKS`/`ANSI_SQL` dialects, foreign-vendor
`custom_extensions` -- are **dropped with a warning**. On **import** (Metric View -> Apache Ossie),
Metric View only features (filter, window, format, rely, ...) are instead **preserved** in
`custom_extensions[DATABRICKS]`, so `MV -> Apache Ossie -> MV` is lossless. Any input that breaks a
[requirement](#requirements) **raises a `ConversionError`** -- the converter never
silently drops a field or produces an invalid result.
| Path | Language | Role |
|------|----------|------|
| [`java/`](java/) | Java | The maintained implementation; also ships a command-line tool (`OssieDatabricksConverter`). |
| [`python/`](python/) | Python | The original reference implementation. To be deprecated. |

## Installation

```bash
pip install apache-ossie-databricks # once published to PyPI
# or, from a checkout of this directory:
pip install -e .
```

The only runtime dependency is `PyYAML`. Python 3.11+.

## Usage

### Command line

```bash
ossie-databricks export -i model.yaml -o view.yaml [--source orders] # Apache Ossie -> Metric View
ossie-databricks import -i view.yaml -o model.yaml [--name my_model] # Metric View -> Apache Ossie
```

With no `-o`, output goes to stdout. `--source` (export) picks the fact/grain (default:
the FK-sink dataset; naming a coarser-grain dataset produces `one_to_many` joins);
`--name` (import) sets the Apache Ossie model name (default: the source's last identifier).

### Python API

```python
from ossie_databricks import convert_ossie_to_metric_view, convert_metric_view_to_ossie

metric_view_yaml = convert_ossie_to_metric_view(ossie_yaml_str) # optionally choose the fact/grain, e.g. (ossie_yaml_str, source="orders")
ossie_yaml = convert_metric_view_to_ossie(metric_view_yaml_str, model_name="sales")
```

## Mapping

Each row maps in both directions; the **Notes** flag where a behavior is specific to
**export** (Apache Ossie -> Metric View) or **import** (Metric View -> Apache Ossie).

| Apache Ossie | Metric View (v1.1) | Notes |
|---|---|---|
| `semantic_model.description` | `comment` | Model-level description only. |
| root dataset | `source` | The fact/grain. |
| other `datasets` | nested `joins[]` | Export: the relationship graph is reassembled into the join tree; a dataset reached by two paths (a diamond) fans out into one aliased join per path. |
| `relationship` `from_columns`/`to_columns` | join `on` (differing names) / `using` (shared names) | Decomposed into columns on import; rebuilt into `on`/`using` on export. |
| `relationship.from`/`to` direction | join `cardinality` | Export: source on the many (`from`) side -> `many_to_one`; on the one (`to`) side -> `one_to_many`. |
| `dataset.primary_key` / `unique_keys` | join `rely.at_most_one_match` | Both directions: export sets `at_most_one_match` when a key covers the join columns; import recovers a `unique_keys` from it. |
| `dataset.fields[]` | `dimensions[]` | Export: fields flatten into one list and a joined column is qualified by its full join path (`customer.c_name`; `customer.region.r_name` when nested). |
| `field.expression.dialects[]` | `expr` | Export: prefer the `DATABRICKS` dialect, else `ANSI_SQL`. |
| `metrics[]` | `measures[]` | Export: fact columns are referenced bare (`SUM(amount)`). |
| `field.label` | `display_name` | |
| `field` / `metric` `description` | `comment` | |
| `ai_context.synonyms` | `synonyms` | |
| `custom_extensions[DATABRICKS]` | `filter`, `window`, `format`, `rely`, `materialization` | Import stashes Metric View only features here; export restores them -- keeping `MV -> Apache Ossie -> MV` lossless. |

## Requirements

Conversion raises a `ConversionError` (rather than guessing or emitting something
invalid) when an input breaks one of these:

- the Metric View `version` is not `1.1`;
- a `source` is not a 3-part `catalog.schema.table` name or a `SELECT`/`WITH` subquery;
- the relationship graph is not acyclic and resolvable to a single fact -- a cycle, or
multiple candidate facts without `--source`, is rejected (a diamond is allowed and
fanned out);
- a join has no condition (a cross join has no Apache Ossie relationship form);
- a join condition is non-equi or otherwise can't be decomposed into equi-join columns
(Apache Ossie relationships are equi-joins, so the join has no Apache Ossie representation);
- the input YAML is malformed.

## Development

```bash
pip install -e ".[dev]"
python3 -m pytest tests/
```

Example-based unit tests plus Hypothesis property-based round-trip tests
(`test_roundtrip_properties.py`, which skip if `hypothesis` is not installed).

## Future effort

Both the Apache Ossie specification and the Databricks Unity Catalog Metric View YAML are still
evolving. As either side adds or changes fields, this converter will be updated to track
them -- extending the mapping and coverage in both directions to keep the conversion
current and to support as much as each format allows over time.
See [`java/README.md`](java/README.md) and [`python/README.md`](python/README.md) for building and
using each implementation.
169 changes: 169 additions & 0 deletions converters/databricks/java/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,169 @@
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->

# Apache Ossie Databricks Converter

Bidirectional, offline conversion between an [Apache Ossie](https://github.com/apache/ossie)
semantic model and a Databricks
[Unity Catalog Metric View](https://docs.databricks.com/aws/en/metric-views/) (YAML `1.1`). Pure
YAML text in, YAML text out: it reads and writes the two formats as parsed maps and lists.

The directions are named from the Apache Ossie model's point of view, matching the
[Python converter](../python/README.md):

- **Export** (`OssieToMetricView`): Apache Ossie -> Metric View (one fact `source` with a nested
`joins` tree and a flat `dimensions` list).
- **Import** (`MetricViewToOssie`): Metric View -> Apache Ossie. Metric-View-only features Apache
Ossie has no native field for are preserved in `custom_extensions[DATABRICKS]`, so
`MV -> Apache Ossie -> MV` is lossless.

On **export** (Apache Ossie -> Metric View), Apache Ossie features with no Metric View slot --
relationship `ai_context`, the non-`synonyms` members of a field/metric `ai_context` object,
`dimension.is_time`, foreign-vendor `custom_extensions` -- are **dropped with a notice**. An
expression prefers the `DATABRICKS` dialect, then `ANSI_SQL`; the other dialect alternatives are
ignored (no notice) when a supported one is present, while a field or metric with no supported
dialect is dropped with a notice. On **import** (Metric View -> Apache Ossie), Metric-View-only
features (`filter`, `parameters`, `materialization`, per-column `format`, measure `window` /
`partition`) are instead **preserved** in `custom_extensions[DATABRICKS]`, so
`MV -> Apache Ossie -> MV` is lossless. Any input that breaks a [requirement](#requirements)
**raises a `ConversionException`** -- the converter never silently drops a field or produces an
invalid result.

## Requirements

- **Java 21+**
- **Maven 3.6+** -- required to build the jar

## Building

Build the self-contained executable jar from source:

```bash
mvn clean package
```

This produces `target/ossie-databricks-converter-0.1.0-SNAPSHOT.jar` with all dependencies
(Jackson and SnakeYAML) bundled.

## Usage

### Command line

```bash
# export: Apache Ossie -> Metric View
java -jar target/ossie-databricks-converter-0.1.0-SNAPSHOT.jar export model.yaml -o view.yaml

# import: Metric View -> Apache Ossie
java -jar target/ossie-databricks-converter-0.1.0-SNAPSHOT.jar import view.yaml -o model.yaml
```

With no `-o`, output goes to stdout. `--source` (export only) picks the fact/grain (default: the
FK-sink dataset; naming a coarser-grain dataset produces `one_to_many` joins); `--name` (import
only) sets the Apache Ossie model name (default: the source's last identifier). Passing a command
its counterpart's flag is an error rather than a silent reinterpretation. Conversion notices
(features dropped on export) are written to stderr; a non-convertible input exits non-zero.

### Java API

```java
import org.apache.ossie.converter.databricks.OssieConverter;

// export: Apache Ossie -> Metric View (optionally choose the fact/grain; default: the FK-sink
// dataset -- naming a coarser-grain dataset produces one_to_many joins)
OssieConverter.Result view = OssieConverter.convertOssieToMetricView(ossieYaml, "orders");

// import: Metric View -> Apache Ossie (optionally name the model; default: the source's last part)
OssieConverter.Result ossie = OssieConverter.convertMetricViewToOssie(metricViewYaml, "sales");
```

Each `Result` carries the output YAML (`result.yaml`) and any notices raised (`result.notices`,
the features dropped on export). A broken [requirement](#requirements) throws a
`ConversionException` instead.

## Mapping

Each row maps in both directions; the **Notes** flag where a behavior is specific to
**export** (Apache Ossie -> Metric View) or **import** (Metric View -> Apache Ossie).

| Apache Ossie | Metric View (v1.1) | Notes |
|---|---|---|
| `semantic_model.description` | `comment` | Model-level description only. |
| root dataset | `source` | The fact/grain. |
| other `datasets` | nested `joins[]` | Export: the relationship graph is reassembled into the join tree; a dataset reached by two paths (a diamond) fans out into one aliased join per path. |
| `relationship` `from_columns`/`to_columns` | join `on` (differing names) / `using` (shared names) | Decomposed into columns on import; rebuilt into `on`/`using` on export. |
| `relationship.from`/`to` direction | join `cardinality` | Export: source on the many (`from`) side -> `many_to_one`; on the one (`to`) side -> `one_to_many`. |
| `dataset.primary_key` / `unique_keys` | join `rely.at_most_one_match` | Both directions: export sets `at_most_one_match` when a key covers the join columns; import recovers a `unique_keys` from it. |
| `dataset.fields[]` | `dimensions[]` | Export: fields flatten into one list and a joined column is qualified by its full join path (`customer.c_name`; `customer.region.r_name` when nested). |
| `field.expression.dialects[]` | `expr` | Export: prefer the `DATABRICKS` dialect, else `ANSI_SQL`; other alternatives are ignored when a supported one is present, and a field with no supported dialect is dropped with a notice. |
| `metrics[]` | `measures[]` | Fact columns are referenced bare (`SUM(amount)`). A joined column is addressed by dataset name in Apache Ossie and by its full join path in the Metric View, so export expands `SUM(region.population)` to `SUM(customer.region.population)` and import maps it back. |
| `field.label` | `display_name` | |
| `field` / `metric` `description` | `comment` | |
| `ai_context.synonyms` | `synonyms` | Only `synonyms`: every other member of an `ai_context` object is dropped with a notice. |
| `custom_extensions[DATABRICKS]` | `filter`, `parameters`, `materialization`, per-column `format`, measure `window` / `partition` | Import stashes Metric-View-only features here; export restores them -- keeping `MV -> Apache Ossie -> MV` lossless. |

## Requirements

Conversion throws a `ConversionException` (rather than guessing or emitting something invalid) when
an input breaks one of these:

- the Metric View `version` is not `1.1`;
- a `source` is not a 3-part `catalog.schema.table` name or a `SELECT`/`WITH` subquery;
- the relationship graph is not acyclic and resolvable to a single fact -- a cycle, or multiple
candidate facts without a chosen source, is rejected (a diamond is allowed and fanned out);
- an Apache Ossie -> Metric View conversion has more than 200 distinct datasets or expands to more
than 200 join nodes;
- on the first semantic model, a consumed `custom_extensions` value is not a list of mappings, has
more than one `DATABRICKS` entry, or has non-empty `DATABRICKS` `data` that is not a string
containing one strict JSON object. Duplicate keys, trailing tokens, YAML syntax, and non-object
JSON roots are rejected; missing, null, or empty `data` is treated as an empty object.
Dataset-level `DATABRICKS` data has no Metric View counterpart and is not consumed;
- a join has no condition (a cross join has no Apache Ossie relationship form);
- a join condition is non-equi or otherwise can't be decomposed into equi-join columns (Apache
Ossie relationships are equi-joins, so the join has no Apache Ossie representation);
- the input YAML is malformed or contains duplicate mapping keys.

## Development

Run the test suite:

```bash
mvn test
```

JUnit 5 suites (unit, round-trip, CLI) live under `src/test/java/`, with the YAML fixtures in
`src/test/resources/`. The source layout:

```
src/main/java/org/apache/ossie/converter/databricks/
OssieConverter.java public facade: entry points + ConversionException/Notices/Result
OssieConverterCommon.java shared constants, YAML I/O, map accessors, the stash codec
OssieToMetricView.java export: Apache Ossie -> Metric View v1.1
MetricViewToOssie.java import: Metric View v1.1 -> Apache Ossie
OssieDatabricksConverter.java command-line entry point (export / import)
```

The authoritative contract is Metric View YAML v1.1 as Databricks defines it; the checked-in
fixtures under `src/test/resources/` pin the expected output of both directions.

## Future effort

Both the Apache Ossie specification and the Databricks Unity Catalog Metric View YAML are still
evolving. As either side adds or changes fields, this converter will be updated to track them --
extending the mapping and coverage in both directions to keep the conversion current and to support
as much as each format allows over time.
Loading