Confluent kafka support#61
Merged
Merged
Conversation
- Replaced the existing Kafka implementation with Confluent Kafka Go client for improved functionality. - Updated task configuration to include new fields for schema registry and idempotent producer support. - Enhanced README documentation to reflect changes in task behavior and configuration options. - Added example pipelines for reading and writing with schema support using Confluent Kafka.
Contributor
|
Since the library is CGO dependent, lets also test it with/wihtout CGO_ENABLED flag. |
… task configuration - Updated the build process in release.yaml and Dockerfile to enable CGO for the caterpillar binary. - Added a new serializer implementation for Avro format with Schema Registry support. - Enhanced README documentation to clarify configuration options and provide examples for using Avro serialization.
| @@ -1,13 +1,13 @@ | |||
| FROM golang:1.24.7-alpine AS builder | |||
| FROM golang:1.24.7 AS builder | |||
Contributor
Author
There was a problem hiding this comment.
Added this becuase
confluent-kafka-go vendors librdkafka_glibc_linux which cannot link against Alpine's musl libc.
|
|
||
| # build executable | ||
| RUN go build -o caterpillar ./cmd/caterpillar/caterpillar.go | ||
| RUN CGO_ENABLED=1 go build -o caterpillar ./cmd/caterpillar/caterpillar.go |
Contributor
Author
There was a problem hiding this comment.
confluent-kafka-go wraps native librdkafka via CGO
Mayuresh Pawar (Mayureshpawar29)
requested review from
Yash Shrivastava (alephys26) and
prasadlohakpure
…rocess - Changed base image from Debian to Alpine for a smaller footprint. - Added necessary packages for building with CGO and librdkafka support. - Updated build command to include dynamic tags for the caterpillar binary.
- Introduced a new codec format handler for Avro serialization in Kafka tasks, improving schema registry integration. - Updated the `newCodecForFormat` function to utilize a map for codec format handling, enhancing maintainability. - Added `schema_format` configuration option to Kafka pipeline YAML files for specifying the serialization format. - Minor comment updates in the Dockerfile to clarify the build process.
| WORKDIR /go/src/github.com/patterninc/caterpillar | ||
|
|
||
| # Alpine 3.20 ships librdkafka 2.4.0; confluent-kafka-go v2 requires 2.14.0+. | ||
| RUN apk add --no-cache gcc musl-dev pkgconf \ |
Contributor
Author
There was a problem hiding this comment.
Build time dependency.
| RUN chmod 755 caterpillar | ||
|
|
||
| FROM alpine:3.20 | ||
| RUN apk add --no-cache \ |
Contributor
Author
There was a problem hiding this comment.
Run time dependency.
Yash Shrivastava (alephys26)
requested changes
May 12, 2026
Contributor
Yash Shrivastava (alephys26)
left a comment
Yash Shrivastava (alephys26)
left a comment
There was a problem hiding this comment.
For the standalone reader, add an Unique suffix on every run, so that it always behaves as a new reader.
Otherwise completely drop the standalone reader functionality.
- Updated the Kafka consumer group ID to include a unique UUID, ensuring better identification and management of consumer instances.
Mayuresh Pawar (Mayureshpawar29)
requested a review
from Yash Shrivastava (alephys26)
Yash Shrivastava (alephys26)
approved these changes
May 12, 2026
Contributor
Yash Shrivastava (alephys26)
left a comment
Yash Shrivastava (alephys26)
left a comment
There was a problem hiding this comment.
LGTM.
prasadlohakpure
approved these changes
May 12, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
segmentio/kafka-gotoconfluent-kafka-go/v2for better SCRAM-SHA-512, TLS, andSchema Registry support
schema_registry_url,schema_registry_username,schema_registry_password)complement to idempotent: true on the producer
kafkaTaskstruct — removed duplicate unexported fields (timeout,batchFlushInterval), deadfields (
BatchSize,UserCert,UserCertPath) and unused constant (defaultBatchSize)and retry logic
Types of changes
Checklist