diff --git a/docs/get-started/img.png b/docs/get-started/img.png new file mode 100644 index 0000000..3785fd8 Binary files /dev/null and b/docs/get-started/img.png differ diff --git a/docs/get-started/img_11.png b/docs/get-started/img_11.png new file mode 100644 index 0000000..872c3cd Binary files /dev/null and b/docs/get-started/img_11.png differ diff --git a/docs/get-started/img_7.png b/docs/get-started/img_7.png new file mode 100644 index 0000000..e0acfeb Binary files /dev/null and b/docs/get-started/img_7.png differ diff --git a/docs/get-started/img_8.png b/docs/get-started/img_8.png new file mode 100644 index 0000000..1fe6b84 Binary files /dev/null and b/docs/get-started/img_8.png differ diff --git a/docs/get-started/img_9.png b/docs/get-started/img_9.png new file mode 100644 index 0000000..bb8fbe9 Binary files /dev/null and b/docs/get-started/img_9.png differ diff --git a/docs/get-started/pulsar-on-streamnative.md b/docs/get-started/pulsar-on-streamnative.md new file mode 100644 index 0000000..93f79c4 --- /dev/null +++ b/docs/get-started/pulsar-on-streamnative.md @@ -0,0 +1,56 @@ + +# Deploying a Pulsar Cluster with StreamNative and Using Numaflow to Produce and Consume Messages + +## Introduction + +This document demonstrates how to: +1. Deploy a Pulsar cluster using StreamNative +2. Connect the remote Pulsar cluster to a local Numaflow pipeline, and produce and consume messages + +### StreamNative + +StreamNative Cloud is a data streaming service, delivered as a fully managed Pulsar and Kafka service. It removes the complexity of managing Pulsar and Kafka. StreamNative provides two options—StreamNative Hosted or BYOC Cloud—to run Pulsar or Kafka in the cloud in a simple, fast, reliable, and cost-effective way. + +• Fully Hosted: StreamNative clusters hosted on StreamNative's public cloud account, available on AWS, GCP, and Azure. You can choose between Serverless and Dedicated clusters. +• Bring Your Own Cloud (BYOC): Host on your own public cloud account (AWS, GCP, or Azure), managed by StreamNative. + +[More info here](https://docs.streamnative.io/docs/cloud-overview) + +### Prerequisite + +Ensure you have Numaflow installed. If not, see the Quick Start guide at https://numaflow.numaproj.io/quick-start/. + +### Getting Started with StreamNative + +This guide focuses on using the StreamNative Cloud Console for management and interactions. + +1. Go to https://streamnative.io/ +2. Create an account at the StreamNative Console: http://console.streamnative.cloud/ +3. Follow the guided prompts to create an organization + ![img.png](img.png) +4. Follow steps 2–4 in this document: https://docs.streamnative.io/docs/quickstart-console + + **In step 3, SAVE THE API KEY.** +5. Select an instance, tenant, and namespace in the top-left bar. This will allow you to select "Topics" from the left side of the page: + ![img_7.png](img_7.png) +6. To test producing messages, we will test the UDSink. Go to [docs/sink/byte-array/manifests/jwt-token](https://github.com/numaproj-contrib/apache-pulsar-java/blob/master/docs/sink/byte-array/manifests/jwt-token), copy these template files, and provide the service URL and topic name. +7. Click "New Topic" and add a name and desired number of partitions. Remember your topic name, and add it in the ConfigMap under the producer. It should follow the format `persistent://tenant/namespace/topic`. + ![img_8.png](img_8.png) +8. To get the service URL, select an instance, then click on the "Overview" tab. For this step, you only need the client service URL (in red). + ![img_9.png](img_9.png) + You should see a page like this. Copy the HTTP service URL and paste it as the client service URL. + ![img_11.png](img_11.png) +9. Paste the API key as a string for the value corresponding to the key `authParams`. You can then apply the ConfigMap and the pipeline for the UDSink. This pipeline generates one message every 10 seconds, and the UDSink **produces** the messages to the topic. + + **To apply the ConfigMap:** + ``` + kubectl apply -f "byte-arr-producer-config.yaml" + ``` + **To apply the pipeline file:** + ``` + kubectl apply -f "byte-arr-producer-pipeline.yaml" + ``` + You should see changes to throughput and storage size in the StreamNative cluster dashboard. + +10. To see the messages produced, you can use the CLI or apply another pipeline for testing. Provide values for the required fields in the UDSource ConfigMap, which can be found at [docs/sink/byte-array/manifests/jwt-token](https://github.com/numaproj-contrib/apache-pulsar-java/tree/master/docs/sink/byte-array/manifests/jwt-token). + Refer to step 8 to get the URL for the admin, the topic name, and the API key. Use the same topic name and API key. Apply the pipeline to **consume** messages from the specified topic and log them. On checking the pod logs, you should see the same messages generated by the first pipeline. diff --git a/docs/sink/byte-array/byte-arr-sink.md b/docs/sink/byte-array/byte-arr-sink.md index 364da3d..78ff60b 100644 --- a/docs/sink/byte-array/byte-arr-sink.md +++ b/docs/sink/byte-array/byte-arr-sink.md @@ -13,7 +13,8 @@ target topic `persistent://public/default/test-topic`. #### Pre-requisite -Have a Pulsar cluster running. +Have a Pulsar cluster running. +If you don't have a Pulsar cluster running, you can follow the instructions [here](https://github.com/numaproj-contrib/apache-pulsar-java/blob/master/docs/get-started/pulsar-on-streamnative.md) to deploy a Pulsar cluster using StreamNative. #### Configure the Pulsar producer diff --git a/docs/sink/byte-array/manifests/api-key/api-key.md b/docs/sink/byte-array/manifests/api-key/api-key.md new file mode 100644 index 0000000..e3aed52 --- /dev/null +++ b/docs/sink/byte-array/manifests/api-key/api-key.md @@ -0,0 +1,6 @@ + +# API Key + +This folder contains both a pipeline and a config map providing an example of using a API key for authentication. Note that the authParamMap cannot be used, because the AuthenticationToken class does not have a constructor that accepts a Map or multiple parameters. Therefore, the authParams for this class must be a single string. + +For more details, see: https://github.com/apache/pulsar/blob/master/pulsar-client/src/main/java/org/apache/pulsar/client/impl/auth/AuthenticationToken.java diff --git a/docs/sink/byte-array/manifests/jwt-token/byte-arr-producer-config.yaml b/docs/sink/byte-array/manifests/api-key/byte-arr-producer-config.yaml similarity index 87% rename from docs/sink/byte-array/manifests/jwt-token/byte-arr-producer-config.yaml rename to docs/sink/byte-array/manifests/api-key/byte-arr-producer-config.yaml index 9019363..3a99d5a 100644 --- a/docs/sink/byte-array/manifests/jwt-token/byte-arr-producer-config.yaml +++ b/docs/sink/byte-array/manifests/api-key/byte-arr-producer-config.yaml @@ -1,4 +1,4 @@ -# JWT Token example +# API Key example apiVersion: v1 kind: ConfigMap metadata: @@ -11,7 +11,7 @@ data: clientConfig: serviceUrl: "insert pulsar+ssl:// ...." # brokerServiceUrl here authPluginClassName: org.apache.pulsar.client.impl.auth.AuthenticationToken - authParams: "insert token" # add JWT token here + authParams: "insert key" # add API key here producer: enabled: true producerConfig: diff --git a/docs/sink/byte-array/manifests/jwt-token/byte-arr-producer-pipeline.yaml b/docs/sink/byte-array/manifests/api-key/byte-arr-producer-pipeline.yaml similarity index 100% rename from docs/sink/byte-array/manifests/jwt-token/byte-arr-producer-pipeline.yaml rename to docs/sink/byte-array/manifests/api-key/byte-arr-producer-pipeline.yaml diff --git a/docs/sink/byte-array/manifests/jwt-token/jwt-token.md b/docs/sink/byte-array/manifests/jwt-token/jwt-token.md deleted file mode 100644 index 917b7f2..0000000 --- a/docs/sink/byte-array/manifests/jwt-token/jwt-token.md +++ /dev/null @@ -1,6 +0,0 @@ - -# JWT Token - -This folder contains both a pipeline and a config map providing an example of using a JWT token for authentication. Note that the authParamMap cannot be used, because the AuthenticationToken class does not have a constructor that accepts a Map or multiple parameters. Therefore, the authParams for this class must be a single string. - -For more details, see: https://github.com/apache/pulsar/blob/master/pulsar-client/src/main/java/org/apache/pulsar/client/impl/auth/AuthenticationToken.java diff --git a/docs/source/byte-array/byte-arr-source.md b/docs/source/byte-array/byte-arr-source.md index 67cf9a6..52f32e3 100644 --- a/docs/source/byte-array/byte-arr-source.md +++ b/docs/source/byte-array/byte-arr-source.md @@ -12,6 +12,7 @@ In this example, we create a pipeline that reads from Apache Pulsar from the spe #### Pre-requisite Have a Pulsar cluster running and if you want a partitioned topic, you must create it before. +If you don't have a Pulsar cluster running, you can follow the instructions [here](https://github.com/numaproj-contrib/apache-pulsar-java/blob/master/docs/get-started/pulsar-on-streamnative.md) to deploy a Pulsar cluster using StreamNative. #### Configure the Pulsar consumer diff --git a/docs/source/byte-array/manifests/api-key/api-key.md b/docs/source/byte-array/manifests/api-key/api-key.md new file mode 100644 index 0000000..0a0715b --- /dev/null +++ b/docs/source/byte-array/manifests/api-key/api-key.md @@ -0,0 +1,4 @@ +# API Key +The pipeline and config map in this folder demonstrate how to use a API key for authentication. Note that the authParamMap cannot be used, because the AuthenticationToken class does not have a constructor that accepts a Map or multiple parameters. Therefore, the authParams for this class must be provided as a single String. Although the admin and client configuration files reference different service URLs, they can still rely on the same key. + +For more details, see: https://github.com/apache/pulsar/blob/master/pulsar-client/src/main/java/org/apache/pulsar/client/impl/auth/AuthenticationToken.java diff --git a/docs/source/byte-array/manifests/jwt-token/byte-arr-consumer-config.yaml b/docs/source/byte-array/manifests/api-key/byte-arr-consumer-config.yaml similarity index 88% rename from docs/source/byte-array/manifests/jwt-token/byte-arr-consumer-config.yaml rename to docs/source/byte-array/manifests/api-key/byte-arr-consumer-config.yaml index 78692c3..2e5747b 100644 --- a/docs/source/byte-array/manifests/jwt-token/byte-arr-consumer-config.yaml +++ b/docs/source/byte-array/manifests/api-key/byte-arr-consumer-config.yaml @@ -1,4 +1,4 @@ -# JWT Token example +# API Key example apiVersion: v1 kind: ConfigMap metadata: @@ -11,7 +11,7 @@ data: clientConfig: serviceUrl: "insert pulsar+ssl:// ...." # brokerServiceUrl authPluginClassName: org.apache.pulsar.client.impl.auth.AuthenticationToken - authParams: "insert token" # add JWT token here + authParams: "insert key" # add API key here consumer: # see here for all configurations: https://pulsar.apache.org/reference/#/4.0.x/client/client-configuration-consumer enabled: true consumerConfig: @@ -21,4 +21,4 @@ data: adminConfig: # Accepts the same key-value pair configurations as pulsar client: https://pulsar.apache.org/reference/#/4.0.x/client/client-configuration-client serviceUrl: "insert pulsar+ssl:// ...." # brokerServiceUrl authPluginClassName: org.apache.pulsar.client.impl.auth.AuthenticationToken - authParams: "insert token" # add JWT token here \ No newline at end of file + authParams: "insert key" # add API key here \ No newline at end of file diff --git a/docs/source/byte-array/manifests/jwt-token/byte-arr-consumer-pipeline.yaml b/docs/source/byte-array/manifests/api-key/byte-arr-consumer-pipeline.yaml similarity index 96% rename from docs/source/byte-array/manifests/jwt-token/byte-arr-consumer-pipeline.yaml rename to docs/source/byte-array/manifests/api-key/byte-arr-consumer-pipeline.yaml index 48c6d02..ea7b27c 100644 --- a/docs/source/byte-array/manifests/jwt-token/byte-arr-consumer-pipeline.yaml +++ b/docs/source/byte-array/manifests/api-key/byte-arr-consumer-pipeline.yaml @@ -1,7 +1,7 @@ apiVersion: numaflow.numaproj.io/v1alpha1 kind: Pipeline metadata: - name: raw-producer-pipeline + name: raw-consumer-pipeline spec: limits: readBatchSize: 1 # Change if you want a different batch size diff --git a/docs/source/byte-array/manifests/jwt-token.md b/docs/source/byte-array/manifests/jwt-token.md deleted file mode 100644 index c423ebb..0000000 --- a/docs/source/byte-array/manifests/jwt-token.md +++ /dev/null @@ -1,4 +0,0 @@ -# JWT Token -The pipeline and config map in this folder demonstrate how to use a JWT token for authentication. Note that the authParamMap cannot be used, because the AuthenticationToken class does not have a constructor that accepts a Map or multiple parameters. Therefore, the authParams for this class must be provided as a single String. Although the admin and client configuration files reference different service URLs, they can still rely on the same token. - -For more details, see: https://github.com/apache/pulsar/blob/master/pulsar-client/src/main/java/org/apache/pulsar/client/impl/auth/AuthenticationToken.java