You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: docs/modules/nifi/pages/usage_guide/writing-to-iceberg-tables.adoc
-88Lines changed: 0 additions & 88 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -33,91 +33,3 @@ Please read on https://github.com/stackabletech/nifi-iceberg-bundle[its document
33
33
You don't need any special configs on the `NiFiCluster` in case you are using S3 and no Kerberos.
34
34
35
35
HDFS and Kerberos are also supported, please have a look at the https://github.com/stackabletech/nifi-operator/tree/main/tests/templates/kuttl/iceberg[Iceberg integration test] for that.
36
-
37
-
== NiFi 1
38
-
39
-
Starting with `1.19.0`, NiFi supports writing to Iceberg tables.
40
-
41
-
The following example shows an example NiFi setup using the Iceberg integration.
42
-
43
-
[source,yaml]
44
-
----
45
-
apiVersion: nifi.stackable.tech/v1alpha1
46
-
kind: NifiCluster
47
-
metadata:
48
-
name: nifi
49
-
spec:
50
-
clusterConfig:
51
-
# ...
52
-
extraVolumes:
53
-
# Will be mounted at /stackable/userdata/nifi-hive-s3-config/
<description>Enables or disables SSL connections to S3.</description>
94
-
</property>
95
-
</configuration>
96
-
----
97
-
98
-
Please fill in the correct endpoint, access key and secret key for your S3 store, this is a classic Hadoop config file.
99
-
100
-
Use e.g. Trino to create a table for Nifi to write into using something like
101
-
102
-
[source,sql]
103
-
----
104
-
CREATE SCHEMA IF NOT EXISTS lakehouse.demo WITH (location = 's3a://lakehouse/demo/');
105
-
106
-
CREATE TABLE IF NOT EXISTS lakehouse.demo.test (
107
-
test varchar
108
-
);
109
-
----
110
-
111
-
In NiFi you need to create a `HiveCatalogService` first which allows you to access the Hive Metastore storing the Iceberg metadata.
112
-
Set `Hive Metastore URI` to something like `thrift://hive-iceberg.default.svc.cluster.local:9083`,
113
-
`Default Warehouse Location` to `s3a://lakehouse`
114
-
and `Hadoop Configuration Resources` to `/stackable/userdata/nifi-hive-s3-config/core-site.xml`.
115
-
116
-
Afterwards you can create the `PutIceberg` processor and configure the `HiveCatalogService`.
117
-
Also set `Catalog Namespace` to your schema name and the `Table Name`.
118
-
119
-
For the `File Format` it is recommended to use `PARQUET` or `ORC` rather than `AVRO` for performance reasons, but you can leave it empty or choose your desired format.
120
-
121
-
You should end up with the following `PutIceberg` processor:
0 commit comments