Create UnArchive transformer #11

iosifnicolae2 · 2022-09-01T17:41:52Z

The UnArchiver is useful to backup all kafka messages to S3 and then restore the backup in another topic.

Here is a guide on how to do this:

Use the following S3 Sink & Source Connector - https://github.com/iosifnicolae2/stream-reactor
Include the following S3 Connector plugin - https://github.com/iosifnicolae2/kafka-connect-transform-archive
- you can use this Dockerfile to build the plugin and include it in your Kafka Connector
Configure your Sink connector using:

"connect.s3.kcql" : "insert into <bucket_name>:<path> select * from <topic> STOREAS `JSON` WITH_FLUSH_INTERVAL=60 WITH_FLUSH_COUNT=1000"
"transforms" : "archiveRowForS3",
"transforms.archiveRowForS3.type" : "com.github.jcustenborder.kafka.connect.archive.Archive",

Configure your Source connector using:

"connect.s3.kcql" : "insert into <topic> select * from <bucket_name>:<path>  BATCH = 5 STOREAS `JSON` LIMIT 1000"
"transforms" : "unArchiveRowFromS3",
"transforms.unArchiveRowFromS3.type" : "com.github.jcustenborder.kafka.connect.archive.UnArchive",

Pass keys

Update Archive.java

Create UnArchive.java

sonatype-lift · 2022-09-01T17:47:03Z

Dockerfile

+WORKDIR /tmp
+RUN apt-get update && apt-get install -y git
+COPY . /tmp/kafka-connect-transform-archive
+RUN cd kafka-connect-transform-archive && mvn package


DL3003: Use WORKDIR to switch to a directory

ℹ️ Learn about @sonatype-lift commands

You can reply with the following commands. For example, reply with @sonatype-lift ignoreall to leave out all findings.

Command Usage

@sonatype-lift ignore Leave out the above finding from this PR

@sonatype-lift ignoreall Leave out all the existing findings from this PR

@sonatype-lift exclude <file|issue|path|tool> Exclude specified file|issue|path|tool from Lift findings by updating your config.toml file

Note: When talking to LiftBot, you need to refresh the page to see its response.
_{Click here to add LiftBot to another repo.}

Was this a good recommendation?
[ 🙁 Not relevant ] - [ 😕 Won't fix ] - [ 😑 Not critical, will fix ] - [ 🙂 Critical, will fix ] - [ 😊 Critical, fixing now ]

sonatype-lift · 2022-09-01T17:47:04Z

Dockerfile

@@ -0,0 +1,5 @@
+FROM maven:3-openjdk-8-slim AS BUILD_CONNECT_TRANSFORM_ARCHIVE_PLUGIN
+WORKDIR /tmp
+RUN apt-get update && apt-get install -y git


DL3008: Pin versions in apt get install. Instead of apt-get install <package> use apt-get install <package>=<version>

ℹ️ Learn about @sonatype-lift commands

You can reply with the following commands. For example, reply with @sonatype-lift ignoreall to leave out all findings.

Command Usage

@sonatype-lift ignore Leave out the above finding from this PR

@sonatype-lift ignoreall Leave out all the existing findings from this PR

@sonatype-lift exclude <file|issue|path|tool> Exclude specified file|issue|path|tool from Lift findings by updating your config.toml file

Note: When talking to LiftBot, you need to refresh the page to see its response.
_{Click here to add LiftBot to another repo.}

Was this a good recommendation?
[ 🙁 Not relevant ] - [ 😕 Won't fix ] - [ 😑 Not critical, will fix ] - [ 🙂 Critical, will fix ] - [ 😊 Critical, fixing now ]

sonatype-lift · 2022-09-01T17:47:05Z

Dockerfile

@@ -0,0 +1,5 @@
+FROM maven:3-openjdk-8-slim AS BUILD_CONNECT_TRANSFORM_ARCHIVE_PLUGIN
+WORKDIR /tmp
+RUN apt-get update && apt-get install -y git


DL3009: Delete the apt-get lists after installing something

ℹ️ Learn about @sonatype-lift commands

You can reply with the following commands. For example, reply with @sonatype-lift ignoreall to leave out all findings.

Command Usage

@sonatype-lift ignore Leave out the above finding from this PR

@sonatype-lift ignoreall Leave out all the existing findings from this PR

@sonatype-lift exclude <file|issue|path|tool> Exclude specified file|issue|path|tool from Lift findings by updating your config.toml file

Note: When talking to LiftBot, you need to refresh the page to see its response.
_{Click here to add LiftBot to another repo.}

Was this a good recommendation?
[ 🙁 Not relevant ] - [ 😕 Won't fix ] - [ 😑 Not critical, will fix ] - [ 🙂 Critical, will fix ] - [ 😊 Critical, fixing now ]

sonatype-lift · 2022-09-01T17:47:06Z

Dockerfile

@@ -0,0 +1,5 @@
+FROM maven:3-openjdk-8-slim AS BUILD_CONNECT_TRANSFORM_ARCHIVE_PLUGIN
+WORKDIR /tmp
+RUN apt-get update && apt-get install -y git


DL3015: Avoid additional packages by specifying --no-install-recommends

ℹ️ Learn about @sonatype-lift commands

You can reply with the following commands. For example, reply with @sonatype-lift ignoreall to leave out all findings.

Command Usage

@sonatype-lift ignore Leave out the above finding from this PR

@sonatype-lift ignoreall Leave out all the existing findings from this PR

@sonatype-lift exclude <file|issue|path|tool> Exclude specified file|issue|path|tool from Lift findings by updating your config.toml file

Note: When talking to LiftBot, you need to refresh the page to see its response.
_{Click here to add LiftBot to another repo.}

Was this a good recommendation?
[ 🙁 Not relevant ] - [ 😕 Won't fix ] - [ 😑 Not critical, will fix ] - [ 🙂 Critical, will fix ] - [ 😊 Critical, fixing now ]

sonatype-lift · 2022-09-01T20:33:01Z

src/main/java/com/github/jcustenborder/kafka/connect/archive/UnArchive.java

+        value.get("key"),
+        null,
+        value.get("value"),
+        Long.parseLong(value.get("timestamp").toString())


NULL_DEREFERENCE: object returned by value.get("timestamp") could be null and is dereferenced at line 46.

ℹ️ Learn about @sonatype-lift commands

You can reply with the following commands. For example, reply with @sonatype-lift ignoreall to leave out all findings.

Command Usage

@sonatype-lift ignore Leave out the above finding from this PR

@sonatype-lift ignoreall Leave out all the existing findings from this PR

@sonatype-lift exclude <file|issue|path|tool> Exclude specified file|issue|path|tool from Lift findings by updating your config.toml file

Note: When talking to LiftBot, you need to refresh the page to see its response.
_{Click here to add LiftBot to another repo.}

Was this a good recommendation?
[ 🙁 Not relevant ] - [ 😕 Won't fix ] - [ 😑 Not critical, will fix ] - [ 🙂 Critical, will fix ] - [ 😊 Critical, fixing now ]

This reverts commit dd0c492.

iosifnicolae2 and others added 19 commits August 31, 2022 21:47

Pass keys

9464f9c

Merge pull request #1 from iosifnicolae2/patch-1

6205de2

Pass keys

Update Archive.java

a32f80b

Create UnArchive.java

1302e3d

Merge pull request #2 from iosifnicolae2/iosifnicolae2-add-unarchive

6f8701e

Update Archive.java

Merge pull request #3 from iosifnicolae2/iosifnicolae2-add-unarchive-1

d27c12c

Create UnArchive.java

Update README.md

0815a10

chore: Remove unused imports

7f846fc

Update UnArchive.java

dc522f1

chore: convert the values

b7499eb

chore: fix indentation

c679fa0

chore: fix styling issues

97929ba

feat: add sample Dockerfile

705e7b0

feat: parse json values if it's the case

4b7a5f0

feat: backward compatibility

27514a7

Update UnArchive.java

a091e6d

Update UnArchive.java

51d40e1

Update UnArchive.java

ff36b72

feat: we will do the json decoding using another transformer

e61f5bc

sonatype-lift bot reviewed Sep 1, 2022

View reviewed changes

Iosif Nicolae added 7 commits September 1, 2022 21:06

Add partition

2e0ace0

Update Archive.java

5ad147a

Update Archive.java

864b06e

Up

7922c22

Update UnArchive.java

c4f46e9

Update UnArchive.java

0704396

Update UnArchive.java

9fcc778

Update UnArchive.java

10aa03e

sonatype-lift bot reviewed Sep 1, 2022

View reviewed changes

Iosif Nicolae added 9 commits September 2, 2022 11:30

upgrade to connect-api 3.0.0

239b8aa

archive record headers

dd0c492

Revert "archive record headers"

a9278b1

This reverts commit dd0c492.

Update Archive.java

1914893

Use SerializationUtils

3fe65e3

add key_string value so we can partition by this field

f31c3fe

Update Archive.java

9007776

Extract timestamp year, month and day

b9923bd

Update Archive.java

1b18fe7

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Create UnArchive transformer #11

Create UnArchive transformer #11

Uh oh!

iosifnicolae2 commented Sep 1, 2022 •

edited

Loading

Uh oh!

sonatype-lift bot Sep 1, 2022

Uh oh!

sonatype-lift bot Sep 1, 2022

Uh oh!

sonatype-lift bot Sep 1, 2022

Uh oh!

sonatype-lift bot Sep 1, 2022

Uh oh!

sonatype-lift bot Sep 1, 2022

Uh oh!

Uh oh!

Command	Usage
`@sonatype-lift ignore`	Leave out the above finding from this PR
`@sonatype-lift ignoreall`	Leave out all the existing findings from this PR
`@sonatype-lift exclude <file\|issue\|path\|tool>`	Exclude specified `file\|issue\|path\|tool` from Lift findings by updating your config.toml file

Create UnArchive transformer #11

Are you sure you want to change the base?

Create UnArchive transformer #11

Uh oh!

Conversation

iosifnicolae2 commented Sep 1, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sonatype-lift bot Sep 1, 2022

Choose a reason for hiding this comment

Uh oh!

sonatype-lift bot Sep 1, 2022

Choose a reason for hiding this comment

Uh oh!

sonatype-lift bot Sep 1, 2022

Choose a reason for hiding this comment

Uh oh!

sonatype-lift bot Sep 1, 2022

Choose a reason for hiding this comment

Uh oh!

sonatype-lift bot Sep 1, 2022

Choose a reason for hiding this comment

Uh oh!

Uh oh!

iosifnicolae2 commented Sep 1, 2022 •

edited

Loading