Dremio 24.2.6 support

STGeomFromGeoJSON Added

Author: sheinbergon

.idea/detekt.xml

@@ -9,14 +9,14 @@
 # Dremio Geo-Spatial Extensions
 
 ### What you get
+
 - Widespread OGC implementation for SQL (adheres to PostGIS standards)
-  - Supported input formats: `WKT`, `WKB (HEX or BINARY)`
-    - Supported output formats: `WKT`, `WKB`, `GeoJSON` 
-- Easily installable Maven-Central/Github artifacts shaded jar artifact  
-- Dremio CE version compatibility (new versions will be released with each community edition) 
+    - Supported input formats: `WKT`, `WKB (HEX or BINARY)`
+        - Supported output formats: `WKT`, `WKB`, `GeoJSON`
+- Easily installable Maven-Central/Github artifacts shaded jar artifact
+- Dremio CE version compatibility (new versions will be released with each community edition)
 - Up-2-date Proj4J & JTS geometry based implementation
 
-
 ### Sponsorship
 
 Enjoying my work? A show of support would be much obliged :grin:
@@ -28,6 +28,7 @@ Enjoying my work? A show of support would be much obliged :grin:
 </a>
 
 ### Installation
+
 - Take the shaded jar for the desired version and place inside your Dremio installation (`$DREMIO_HOME/jars/3rdparty`)
 - Restart your Dremio server(s)
 - Rejoice! (and see the [WIKI](https://github.com/sheinbergon/dremio-udf-gis/wiki) for detailed usage instructions)
@@ -36,30 +37,32 @@ Enjoying my work? A show of support would be much obliged :grin:
 
 | Library Version | Dremio Version | Status     |
 |-----------------|----------------|------------|
-| 0.2.x           | 20.1.x         | Legacy     |
-| 0.3.x           | 21.1.x         | Legacy     |
-| 0.4.x           | 21.2.x         | Legacy     |
-| 0.5.x           | 22.0.x         | Legacy     |
-| 0.6.x           | 22.1.x         | Legacy     |
-| 0.7.x           | 23.0.x         | Maintained |
-| 0.8.x           | 23.1.x         | Maintained |
-| 0.9.x           | 24.0.x         | Maintained |
-| 0.10.x          | 24.0.x         | Maintained |
-
+| 0.2.x           | 20.1.0         | Legacy     |
+| 0.3.x           | 21.1.1         | Legacy     |
+| 0.4.x           | 21.2.0         | Legacy     |
+| 0.5.x           | 22.0.0         | Legacy     |
+| 0.6.x           | 22.1.1         | Legacy     |
+| 0.7.x           | 23.0.1         | Legacy     |
+| 0.8.x           | 23.1.0         | Legacy     |
+| 0.9.x           | 24.0.0         | Maintained |
+| 0.10.x          | 24.1.0         | Maintained |
+| 0.11.x          | 24.2.6         | Maintained |
 
 ### Usage Notes
+
 As opposed to PostGIS, Dremio is only a query engine based on existing/projected data sources/lakes.  
 That means that `Geometry` is not a natively supported data type, and you can only access it if     
 it's being properly projected from the data sources (For example, PostGIS Geometry is read as an `EWKB` HEX encoded string).
 
 In order to successfully use the provided GIS functions, you must first make sure the geometry is in `WKB (BINARY)` format.
-If it's not, you need to decode it:  
+If it's not, you need to decode it:
+
 - if the input is in `WKT` format, use `ST_GeomFromText`
 - if the input is a HEX encoded`WKB`, use Dremio's `FROM_HEX`
 
 This library uses Dremios' Arrow buffers (`ArrowBuf`) to maintain geometry data in binary (`WKB`) format (for performance and efficiency)  
 when interchanging it between GIS functions, which is of course undecipherable for the naked eye. When running queries from the UI,  
-`WKB` output will always be base64 encoded. 
+`WKB` output will always be base64 encoded.
 
 In order to resolve Data back to human-readable format (`WKT`), use `ST_AsText`/`ST_AsGeoJson`
 
@@ -74,11 +77,14 @@ SELECT ST_AsText(
 ```
 
 ### Roadmap
+
 - Frequent version/dependency updates
 - Add more OGC/PostGIS matching functionality
 - Add Geography type support
 
 ### Noteworthy Mentions
-Work in this repository was originally based on the following sources:  
+
+Work in this repository was originally based on the following sources:
+
 - [Apache Drill GIS Functionality](https://github.com/apache/drill/tree/master/contrib/udfs/src/main/java/org/apache/drill/exec/udfs/gis)
 - [Christy Haragan's initial port](https://github.com/christyharagan/dremio-gis)  

.idea/kotlinc.xml

@@ -6,20 +6,21 @@
     <artifactId>dremio-udf-gis</artifactId>
     <properties>
         <checkstyle.version>9.3</checkstyle.version>
-        <dremio.version>24.2.2-202309281602050954-7d5e877f</dremio.version>
-        <dremio-arrow.version>12.0.1-20230728221431-62098927e3-dremio</dremio-arrow.version>
+        <dremio.version>24.2.6-202311250456170399-68acbe47</dremio.version>
+        <dremio-arrow.version>12.0.1-20231103121511-850ae5a2d6-dremio</dremio-arrow.version>
         <proj4j.version>1.3.0</proj4j.version>
         <jts-core.version>1.19.0</jts-core.version>
         <jts-io-common.version>1.19.0</jts-io-common.version>
         <commons-io.version>2.11.0</commons-io.version>
         <junit.version>4.13.2</junit.version>
-        <junit-jupiter.version>5.10.0</junit-jupiter.version>
-        <kotlin.version>1.9.10</kotlin.version>
+        <junit-jupiter.version>5.10.1</junit-jupiter.version>
+        <junit-platform.version>1.10.1</junit-platform.version>
+        <kotlin.version>1.9.21</kotlin.version>
         <kotest.version>5.7.2</kotest.version>
         <joda.version>2.9</joda.version>
-        <detekt.version>1.23.1</detekt.version>
+        <detekt.version>1.23.4</detekt.version>
         <carrotsearch.version>0.7.0</carrotsearch.version>
-        <arrow-memory-netty.version>vector-12.0.1-20230728221431-62098927e3-dremio</arrow-memory-netty.version>
+        <arrow-memory-netty.version>12.0.1</arrow-memory-netty.version>
     </properties>
     <version>0.11.0-SNAPSHOT</version>
     <name>dremio-udf-gis</name>
@@ -192,6 +193,12 @@
             <version>${junit-jupiter.version}</version>
             <scope>test</scope>
         </dependency>
+        <dependency>
+            <groupId>org.junit.platform</groupId>
+            <artifactId>junit-platform-engine</artifactId>
+            <version>${junit-platform.version}</version>
+            <scope>test</scope>
+        </dependency>
         <dependency>
             <groupId>org.jetbrains.kotlin</groupId>
             <artifactId>kotlin-test-junit5</artifactId>
@@ -393,7 +400,7 @@
             <plugin>
                 <groupId>org.apache.maven.plugins</groupId>
                 <artifactId>maven-surefire-plugin</artifactId>
-                <version>3.0.0-M7</version>
+                <version>3.1.2</version>
             </plugin>
             <plugin>
                 <groupId>org.jacoco</groupId>

README.MD

@@ -0,0 +1,55 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.sheinbergon.dremio.udf.gis;
+
+import com.dremio.exec.expr.SimpleFunction;
+import com.dremio.exec.expr.annotations.FunctionTemplate;
+import com.dremio.exec.expr.annotations.Output;
+import com.dremio.exec.expr.annotations.Param;
+
+import javax.inject.Inject;
+
+@FunctionTemplate(
+    name = "ST_GeomFromGeoJSON",
+    scope = FunctionTemplate.FunctionScope.SIMPLE,
+    nulls = FunctionTemplate.NullHandling.INTERNAL)
+public class STGeomFromGeoJson implements SimpleFunction {
+
+  @Param
+  org.apache.arrow.vector.holders.NullableVarCharHolder jsonInput;
+
+  @Output
+  org.apache.arrow.vector.holders.NullableVarBinaryHolder binaryOutput;
+
+  @Inject
+  org.apache.arrow.memory.ArrowBuf buffer;
+
+  public void setup() {
+  }
+
+  public void eval() {
+    if (org.sheinbergon.dremio.udf.gis.util.GeometryHelpers.isHolderSet(jsonInput)) {
+      org.locationtech.jts.geom.Geometry geom = org.sheinbergon.dremio.udf.gis.util.GeometryHelpers.toGeometryFromGeoJson(jsonInput);
+      byte[] bytes = org.sheinbergon.dremio.udf.gis.util.GeometryHelpers.toEWKB(geom);
+      buffer = buffer.reallocIfNeeded(bytes.length);
+      org.sheinbergon.dremio.udf.gis.util.GeometryHelpers.populate(bytes, buffer, binaryOutput);
+    } else {
+      org.sheinbergon.dremio.udf.gis.util.GeometryHelpers.markHolderNotSet(binaryOutput);
+    }
+  }
+}

pom.xml

@@ -27,6 +27,7 @@
 import org.locationtech.jts.algorithm.Angle;
 import org.locationtech.jts.geom.*;
 import org.locationtech.jts.io.*;
+import org.locationtech.jts.io.geojson.GeoJsonReader;
 import org.locationtech.jts.io.geojson.GeoJsonWriter;
 import org.locationtech.jts.operation.buffer.BufferOp;
 import org.locationtech.jts.operation.valid.IsValidOp;
@@ -133,6 +134,17 @@ public static Geometry toGeometry(final @Nonnull NullableVarCharHolder holder) {
     }
   }
 
+  @Nonnull
+  public static Geometry toGeometryFromGeoJson(final @Nonnull NullableVarCharHolder holder) {
+    try {
+      String json = toUTF8String(holder);
+      GeoJsonReader reader = new GeoJsonReader();
+      return reader.read(json);
+    } catch (ParseException x) {
+      throw new RuntimeException(x);
+    }
+  }
+
   @Nonnull
   public static Geometry toGeometryFromEWKT(final @Nonnull NullableVarCharHolder holder) {
     try {

src/main/java/org/sheinbergon/dremio/udf/gis/STGeomFromGeoJson.java

@@ -0,0 +1,37 @@
+package org.sheinbergon.dremio.udf.gis
+
+import org.apache.arrow.vector.holders.NullableVarBinaryHolder
+import org.apache.arrow.vector.holders.NullableVarCharHolder
+import org.sheinbergon.dremio.udf.gis.spec.GeometryInputFunSpec
+import org.sheinbergon.dremio.udf.gis.util.allocateBuffer
+
+internal class STGeomFromGeoJsonTests : GeometryInputFunSpec.NullableVarChar<STGeomFromGeoJson>() {
+
+  init {
+    testGeometryInput(
+      "Calling ST_GeomFromGeoJSON on a POINT",
+      """
+        {"type":"Point","coordinates":[0.5,0.5]}
+      """.trimIndent(),
+      byteArrayOf(1, 1, 0, 0, 32, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, -32, 63, 0, 0, 0, 0, 0, 0, -32, 63)
+    )
+
+    testInvalidGeometryInput(
+      "Calling ST_GeomFromGeoJSON on rubbish text",
+      "42ifon2 fA!@",
+    )
+
+    testNullGeometryInput(
+      "Calling ST_GeomFromGeoJSON on null input"
+    )
+  }
+
+  override val function = STGeomFromGeoJson().apply {
+    jsonInput = NullableVarCharHolder()
+    binaryOutput = NullableVarBinaryHolder()
+    buffer = allocateBuffer()
+  }
+
+  override val STGeomFromGeoJson.input: NullableVarCharHolder get() = function.jsonInput
+  override val STGeomFromGeoJson.output: NullableVarBinaryHolder get() = function.binaryOutput
+}

src/main/java/org/sheinbergon/dremio/udf/gis/util/GeometryHelpers.java

@@ -14,6 +14,7 @@ import org.sheinbergon.dremio.udf.gis.util.reset
 import org.sheinbergon.dremio.udf.gis.util.setBinary
 import org.sheinbergon.dremio.udf.gis.util.setUtf8
 import org.sheinbergon.dremio.udf.gis.util.valueIsNotSet
+import java.util.*
 
 abstract class GeometryInputFunSpec<F : SimpleFunction, I : ValueHolder, V : Any> : FunSpec() {
 
@@ -44,7 +45,9 @@ abstract class GeometryInputFunSpec<F : SimpleFunction, I : ValueHolder, V : Any
       }
     }
 
-    final override fun NullableVarCharHolder.markNotSet() = this.valueIsNotSet()
+    final override fun NullableVarCharHolder.markNotSet() {
+      this.valueIsNotSet()
+    }
 
     final override fun NullableVarCharHolder.set(value: String) = this.setUtf8(value)
   }
@@ -97,7 +100,10 @@ abstract class GeometryInputFunSpec<F : SimpleFunction, I : ValueHolder, V : Any
   }
 
   private fun NullableVarBinaryHolder.valueIs(bytes: ByteArray) =
-    GeometryHelpers.toEWKB(GeometryHelpers.toGeometry(this)) shouldBe bytes
+    print(Arrays.toString(GeometryHelpers.toEWKB(GeometryHelpers.toGeometry(this))))
+      .also {
+        GeometryHelpers.toEWKB(GeometryHelpers.toGeometry(this)) shouldBe bytes
+      }
 
   protected abstract fun I.set(value: V)
   protected abstract val function: F

src/test/kotlin/org/sheinbergon/dremio/udf/gis/STGeomFromGeoJsonTests.kt

@@ -47,8 +47,9 @@ abstract class GeometryRelationFunSpec<F : SimpleFunction, O : ValueHolder> : Fu
       this.isSet shouldBeExactly 1
     }
 
-    override fun NullableVarCharHolder.valueIsNotSet() =
+    override fun NullableVarCharHolder.valueIsNotSet() {
       this.isSet shouldBe 0
+    }
   }
 
   abstract class NullableBitOutput<F : SimpleFunction> : GeometryRelationFunSpec<F, NullableBitHolder>() {
@@ -91,8 +92,9 @@ abstract class GeometryRelationFunSpec<F : SimpleFunction, O : ValueHolder> : Fu
       }
     }
 
-    override fun NullableBitHolder.valueIsNotSet() =
+    override fun NullableBitHolder.valueIsNotSet() {
       this.isSet shouldBe 0
+    }
   }
 
   protected fun testNullGeometryRelation(