numaproj-contrib · KeranYang · Apr 9, 2025 · Apr 7, 2025 · Apr 7, 2025 · Apr 8, 2025
diff --git a/docs/get-started/img.png b/docs/get-started/img.png
diff --git a/docs/get-started/img_11.png b/docs/get-started/img_11.png
diff --git a/docs/get-started/img_7.png b/docs/get-started/img_7.png
diff --git a/docs/get-started/img_8.png b/docs/get-started/img_8.png
diff --git a/docs/get-started/img_9.png b/docs/get-started/img_9.png
diff --git a/docs/get-started/pulsar-on-streamnative.md b/docs/get-started/pulsar-on-streamnative.md
@@ -0,0 +1,56 @@
+
+# Deploying a Pulsar Cluster with StreamNative and Using Numaflow to Produce and Consume Messages
+
+## Introduction
+
+This document demonstrates how to:  
+1. Deploy a Pulsar cluster using StreamNative  
+2. Connect the remote Pulsar cluster to a local Numaflow pipeline, and produce and consume messages  
+
+### StreamNative
+
+StreamNative Cloud is a data streaming service, delivered as a fully managed Pulsar and Kafka service. It removes the complexity of managing Pulsar and Kafka. StreamNative provides two options—StreamNative Hosted or BYOC Cloud—to run Pulsar or Kafka in the cloud in a simple, fast, reliable, and cost-effective way.
+
+• Fully Hosted: StreamNative clusters hosted on StreamNative's public cloud account, available on AWS, GCP, and Azure. You can choose between Serverless and Dedicated clusters.  
+• Bring Your Own Cloud (BYOC): Host on your own public cloud account (AWS, GCP, or Azure), managed by StreamNative.
+
+[More info here](https://docs.streamnative.io/docs/cloud-overview)
+
+### Prerequisite
+
+Ensure you have Numaflow installed. If not, see the Quick Start guide at https://numaflow.numaproj.io/quick-start/.
+
+### Getting Started with StreamNative
+
+This guide focuses on using the StreamNative Cloud Console for management and interactions.
+
+1. Go to https://streamnative.io/  
+2. Create an account at the StreamNative Console: http://console.streamnative.cloud/  
+3. Follow the guided prompts to create an organization  
+   ![img.png](img.png)  
+4. Follow steps 2–4 in this document: https://docs.streamnative.io/docs/quickstart-console  
+
+   **In step 3, SAVE THE API KEY.**  
+5. Select an instance, tenant, and namespace in the top-left bar. This will allow you to select "Topics" from the left side of the page:  
+   ![img_7.png](img_7.png)  
+6. To test producing messages, we will test the UDSink. Go to [docs/sink/byte-array/manifests/jwt-token](https://github.com/numaproj-contrib/apache-pulsar-java/blob/master/docs/sink/byte-array/manifests/jwt-token), copy these template files, and provide the service URL and topic name.  
+7. Click "New Topic" and add a name and desired number of partitions. Remember your topic name, and add it in the ConfigMap under the producer. It should follow the format `persistent://tenant/namespace/topic`.  
+   ![img_8.png](img_8.png)  
+8. To get the service URL, select an instance, then click on the "Overview" tab. For this step, you only need the client service URL (in red).  
+   ![img_9.png](img_9.png)  
+   You should see a page like this. Copy the HTTP service URL and paste it as the client service URL.  
+   ![img_11.png](img_11.png)  
+9. Paste the API key as a string for the value corresponding to the key `authParams`. You can then apply the ConfigMap and the pipeline for the UDSink. This pipeline generates one message every 10 seconds, and the UDSink **produces** the messages to the topic.
+
+   **To apply the ConfigMap:**
+   ```
+   kubectl apply -f "byte-arr-producer-config.yaml"
+   ```
+   **To apply the pipeline file:**
+   ```
+   kubectl apply -f "byte-arr-producer-pipeline.yaml"
+   ```
+   You should see changes to throughput and storage size in the StreamNative cluster dashboard.
+
+10. To see the messages produced, you can use the CLI or apply another pipeline for testing. Provide values for the required fields in the UDSource ConfigMap, which can be found at [docs/sink/byte-array/manifests/jwt-token](https://github.com/numaproj-contrib/apache-pulsar-java/tree/master/docs/sink/byte-array/manifests/jwt-token).  
+    Refer to step 8 to get the URL for the admin, the topic name, and the API key. Use the same topic name and API key. Apply the pipeline to **consume** messages from the specified topic and log them. On checking the pod logs, you should see the same messages generated by the first pipeline.
diff --git a/docs/sink/byte-array/byte-arr-sink.md b/docs/sink/byte-array/byte-arr-sink.md
@@ -13,7 +13,8 @@ target topic `persistent://public/default/test-topic`.
 
 #### Pre-requisite
 
-Have a Pulsar cluster running.
+Have a Pulsar cluster running. 
+If you don't have a Pulsar cluster running, you can follow the instructions [here](https://github.com/numaproj-contrib/apache-pulsar-java/blob/master/docs/get-started/pulsar-on-streamnative.md) to deploy a Pulsar cluster using StreamNative.
 
 #### Configure the Pulsar producer
 

diff --git a/docs/sink/byte-array/manifests/api-key/api-key.md b/docs/sink/byte-array/manifests/api-key/api-key.md
@@ -0,0 +1,6 @@
+
+# API Key
+
+This folder contains both a pipeline and a config map providing an example of using a API key for authentication. Note that the authParamMap cannot be used, because the AuthenticationToken class does not have a constructor that accepts a Map or multiple parameters. Therefore, the authParams for this class must be a single string. 
+
+For more details, see: https://github.com/apache/pulsar/blob/master/pulsar-client/src/main/java/org/apache/pulsar/client/impl/auth/AuthenticationToken.java
diff --git a/...s/jwt-token/byte-arr-producer-config.yaml → ...sts/api-key/byte-arr-producer-config.yaml b/...s/jwt-token/byte-arr-producer-config.yaml → ...sts/api-key/byte-arr-producer-config.yaml
@@ -1,4 +1,4 @@
-# JWT Token example
+# API Key example
 apiVersion: v1
 kind: ConfigMap
 metadata:
@@ -11,7 +11,7 @@ data:
           clientConfig:
             serviceUrl: "insert pulsar+ssl:// ...." # brokerServiceUrl here 
             authPluginClassName: org.apache.pulsar.client.impl.auth.AuthenticationToken
-            authParams: "insert token" # add JWT token here
+            authParams: "insert key" # add API key here
         producer:
           enabled: true
           producerConfig: 

diff --git a/...jwt-token/byte-arr-producer-pipeline.yaml → ...s/api-key/byte-arr-producer-pipeline.yaml b/...jwt-token/byte-arr-producer-pipeline.yaml → ...s/api-key/byte-arr-producer-pipeline.yaml
diff --git a/docs/sink/byte-array/manifests/jwt-token/jwt-token.md b/docs/sink/byte-array/manifests/jwt-token/jwt-token.md
diff --git a/docs/source/byte-array/byte-arr-source.md b/docs/source/byte-array/byte-arr-source.md
@@ -12,6 +12,7 @@ In this example, we create a pipeline that reads from Apache Pulsar from the spe
 #### Pre-requisite
 
 Have a Pulsar cluster running and if you want a partitioned topic, you must create it before.
+If you don't have a Pulsar cluster running, you can follow the instructions [here](https://github.com/numaproj-contrib/apache-pulsar-java/blob/master/docs/get-started/pulsar-on-streamnative.md) to deploy a Pulsar cluster using StreamNative.
 
 #### Configure the Pulsar consumer
 

diff --git a/docs/source/byte-array/manifests/api-key/api-key.md b/docs/source/byte-array/manifests/api-key/api-key.md
@@ -0,0 +1,4 @@
+# API Key 
+The pipeline and config map in this folder demonstrate how to use a API key for authentication. Note that the authParamMap cannot be used, because the AuthenticationToken class does not have a constructor that accepts a Map or multiple parameters. Therefore, the authParams for this class must be provided as a single String. Although the admin and client configuration files reference different service URLs, they can still rely on the same key. 
+
+For more details, see: https://github.com/apache/pulsar/blob/master/pulsar-client/src/main/java/org/apache/pulsar/client/impl/auth/AuthenticationToken.java
diff --git a/...s/jwt-token/byte-arr-consumer-config.yaml → ...sts/api-key/byte-arr-consumer-config.yaml b/...s/jwt-token/byte-arr-consumer-config.yaml → ...sts/api-key/byte-arr-consumer-config.yaml
@@ -1,4 +1,4 @@
-# JWT Token example
+# API Key example
 apiVersion: v1
 kind: ConfigMap
 metadata:
@@ -11,7 +11,7 @@ data:
           clientConfig:
             serviceUrl: "insert pulsar+ssl:// ...." # brokerServiceUrl
             authPluginClassName: org.apache.pulsar.client.impl.auth.AuthenticationToken
-            authParams: "insert token" # add JWT token here
+            authParams: "insert key" # add API key here
         consumer:  # see here for all configurations: https://pulsar.apache.org/reference/#/4.0.x/client/client-configuration-consumer
           enabled: true
           consumerConfig: 
@@ -21,4 +21,4 @@ data:
           adminConfig: # Accepts the same key-value pair configurations as pulsar client: https://pulsar.apache.org/reference/#/4.0.x/client/client-configuration-client
             serviceUrl: "insert pulsar+ssl:// ...." # brokerServiceUrl
             authPluginClassName: org.apache.pulsar.client.impl.auth.AuthenticationToken
-            authParams: "insert token" # add JWT token here
+            authParams: "insert key" # add API key here
diff --git a/...jwt-token/byte-arr-consumer-pipeline.yaml → ...s/api-key/byte-arr-consumer-pipeline.yaml b/...jwt-token/byte-arr-consumer-pipeline.yaml → ...s/api-key/byte-arr-consumer-pipeline.yaml
@@ -1,7 +1,7 @@
 apiVersion: numaflow.numaproj.io/v1alpha1
 kind: Pipeline
 metadata:
-  name: raw-producer-pipeline
+  name: raw-consumer-pipeline
 spec:
   limits: 
     readBatchSize: 1 # Change if you want a different batch size

diff --git a/docs/source/byte-array/manifests/jwt-token.md b/docs/source/byte-array/manifests/jwt-token.md