Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add ability to configure Kafka producers individually #357

Open
jbeemster opened this issue Oct 4, 2023 · 1 comment
Open

Add ability to configure Kafka producers individually #357

jbeemster opened this issue Oct 4, 2023 · 1 comment

Comments

@jbeemster
Copy link
Member

Currently we only allow a single producerConf for both Kafka Producers for the raw and bad output streams. This means we end up dynamically inferring things like client.id which are then set to producer-1 and producer-2 (and which renders the Stream Lineage view in Confluent Cloud effectively useless.

It also means permissions for multiple topics need to be shared rather than being individual.

All other Kafka applications enforce segmented configuration for each producer + consumer and the Collector should follow suit.

@benjben
Copy link
Contributor

benjben commented Oct 5, 2023

We had a discussion about this when starting to work on the refactoring of the collector.

The question was : should we take the opportunity to align the configuration of the collector with the other apps, and have :

input {
}
output {
  good {
  }
  bad {
  }
}

It was decided to keep the existing format for now, so that we don't need to worry about changing Terraform yet (both for OSS and our infra).

Once we have made the switch to http4s collector we can see to make these breaking changes and unify the configuration format of the collector with the other apps, which will make possible to configure good and bad streams differently.

/cc @stanch

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants