Improvements to kafka checks #276
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
smalleni
force-pushed
the
kafka_improvements
branch
2 times, most recently
from
40c5db1
to
212c8e9
Compare
|
/rerun all |
smalleni
force-pushed
the
kafka_improvements
branch
from
212c8e9
to
7a43efb
Compare
- Update backend details to the doc sent to ES - Close kafka consumer - Increase the `end_time` used to get the kafka offset This is because, unlike ES, the timestamp is the one created by Kafka when the log message is recorded in Kafka. For example, in the case of ES, the timestamp added by FluenD becomes the timestamp field in ES, so we can query accunrately for log messages between the `start_time` and `end_time` of the log generation test. However, in the case of Kafka, we cannot accurately check for the log messages generated during the test because the records in Kafka and their corresponding timstamps for the offsets could be well after the `end_time` of the test as the messages slowly show up in the kafka cluster and there is no way to use the timestamp from FluentD as the timstamp for the kafka offset. The FluentD timestamp is actually in the content of the Kafka record. We currently check for all messages from the `start_time` and before the `timeout` configured in the CR. Signed-off-by: Sai Sindhur Malleni <smalleni@redhat.com>
smalleni
force-pushed
the
kafka_improvements
branch
from
7a43efb
to
709e81f
Compare
|
Results for SNAFU CI Test
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
end_timeused to get the kafka offsetThis is because, unlike ES, the timestamp is the one created
by Kafka when the log message is recorded in Kafka. For example,
in the case of ES, the timestamp added by FluenD becomes the timestamp
field in ES, so we can query accunrately for log messages between the
start_timeandend_timeof the log generation test. However, in the caseof Kafka, we cannot accurately check for the log messages generated during
the test because the records in Kafka and their corresponding timstamps for
the offsets could significantly after the
end_timeof the test as the messagesslowly show up in the kafka cluster and there is no way to use the timestamp from
FluentD as the timstamp for the kafka offset and the FluentD timestamp is actually
in the content of the Kafka record. We currently check for all messages from the
start_timeand before thetimeoutconfigured in the CR.Signed-off-by: Sai Sindhur Malleni smalleni@redhat.com
Description
Fixes