Skip to content

Commit 770a118

Browse files
committed
docs: Remove NiFi 1 section on Iceberg table page
1 parent de384ec commit 770a118

2 files changed

Lines changed: 0 additions & 88 deletions

File tree

-85.4 KB
Binary file not shown.

docs/modules/nifi/pages/usage_guide/writing-to-iceberg-tables.adoc

Lines changed: 0 additions & 88 deletions
Original file line numberDiff line numberDiff line change
@@ -33,91 +33,3 @@ Please read on https://github.com/stackabletech/nifi-iceberg-bundle[its document
3333
You don't need any special configs on the `NiFiCluster` in case you are using S3 and no Kerberos.
3434

3535
HDFS and Kerberos are also supported, please have a look at the https://github.com/stackabletech/nifi-operator/tree/main/tests/templates/kuttl/iceberg[Iceberg integration test] for that.
36-
37-
== NiFi 1
38-
39-
Starting with `1.19.0`, NiFi supports writing to Iceberg tables.
40-
41-
The following example shows an example NiFi setup using the Iceberg integration.
42-
43-
[source,yaml]
44-
----
45-
apiVersion: nifi.stackable.tech/v1alpha1
46-
kind: NifiCluster
47-
metadata:
48-
name: nifi
49-
spec:
50-
clusterConfig:
51-
# ...
52-
extraVolumes:
53-
# Will be mounted at /stackable/userdata/nifi-hive-s3-config/
54-
- name: nifi-hive-s3-config
55-
secret:
56-
secretName: nifi-hive-s3-config
57-
---
58-
apiVersion: v1
59-
kind: Secret
60-
metadata:
61-
name: nifi-hive-s3-config
62-
stringData:
63-
core-site.xml: |
64-
<configuration>
65-
<property>
66-
<name>fs.s3a.aws.credentials.provider</name>
67-
<value>org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider</value>
68-
</property>
69-
70-
<property>
71-
<name>fs.s3a.endpoint</name>
72-
<value>http://minio:9000</value>
73-
</property>
74-
75-
<property>
76-
<name>fs.s3a.access.key</name>
77-
<value>xxx</value>
78-
</property>
79-
80-
<property>
81-
<name>fs.s3a.secret.key</name>
82-
<value>xxx</value>
83-
</property>
84-
85-
<property>
86-
<name>fs.s3a.path.style.access</name>
87-
<value>true</value>
88-
</property>
89-
90-
<property>
91-
<name>fs.s3a.connection.ssl.enabled</name>
92-
<value>false</value>
93-
<description>Enables or disables SSL connections to S3.</description>
94-
</property>
95-
</configuration>
96-
----
97-
98-
Please fill in the correct endpoint, access key and secret key for your S3 store, this is a classic Hadoop config file.
99-
100-
Use e.g. Trino to create a table for Nifi to write into using something like
101-
102-
[source,sql]
103-
----
104-
CREATE SCHEMA IF NOT EXISTS lakehouse.demo WITH (location = 's3a://lakehouse/demo/');
105-
106-
CREATE TABLE IF NOT EXISTS lakehouse.demo.test (
107-
test varchar
108-
);
109-
----
110-
111-
In NiFi you need to create a `HiveCatalogService` first which allows you to access the Hive Metastore storing the Iceberg metadata.
112-
Set `Hive Metastore URI` to something like `thrift://hive-iceberg.default.svc.cluster.local:9083`,
113-
`Default Warehouse Location` to `s3a://lakehouse`
114-
and `Hadoop Configuration Resources` to `/stackable/userdata/nifi-hive-s3-config/core-site.xml`.
115-
116-
Afterwards you can create the `PutIceberg` processor and configure the `HiveCatalogService`.
117-
Also set `Catalog Namespace` to your schema name and the `Table Name`.
118-
119-
For the `File Format` it is recommended to use `PARQUET` or `ORC` rather than `AVRO` for performance reasons, but you can leave it empty or choose your desired format.
120-
121-
You should end up with the following `PutIceberg` processor:
122-
123-
image::put-iceberg-processor.png[]

0 commit comments

Comments
 (0)