[filelogreceiver]: Add ability to sort by mtime #28691
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
djaglowski
reviewed
Oct 30, 2023
| @@ -133,11 +169,12 @@ func (m Matcher) MatchFiles() ([]string, error) { | |||
| if len(files) == 0 { | |||
| return files, errors.Join(fmt.Errorf("no files match the configured criteria"), errs) | |||
| } | |||
| if len(m.filterOpts) == 0 { | |||
|
|
|||
| if m.f == nil || m.f.SkipFiltering() { | |||
There was a problem hiding this comment.
Why do we need SkipFiltering? Isn't it enough to check if m.f == nil?
| @@ -50,7 +50,8 @@ Tails and parses logs from files. | |||
| | `retry_on_failure.initial_interval` | `1s` | [Time](#time-parameters) to wait after the first failure before retrying. | | |||
| | `retry_on_failure.max_interval` | `30s` | Upper bound on retry backoff [interval](#time-parameters). Once this value is reached the delay between consecutive retries will remain constant at the specified value. | | |||
| | `retry_on_failure.max_elapsed_time` | `5m` | Maximum amount of [time](#time-parameters) (including retries) spent trying to send a logs batch to a downstream consumer. Once this value is reached, the data is discarded. Retrying never stops if set to `0`. | |||
| | `ordering_criteria.regex` | | Regular expression used for sorting, should contain a named capture groups that are to be used in `regex_key`. | | |||
| | `ordering_criteria.method` | `regex` | Specifies the mode to use for ordering. Can be `regex` (sort using regex and sort_by rules) or `mtime` (sort by file modified time). | | |||
There was a problem hiding this comment.
Doesn't this serve basically the same purpose as ordering_criteria.sort_by.sort_type? Why not add mtime as an additional option there?
This would also allow users to compose multiple types of ordering and filtering.
There was a problem hiding this comment.
I think I got a little lost looking at the current implementation where every sort type requires the regex to be set, I'll take a stab at reworking to have to be a separate sort type.
There was a problem hiding this comment.
New implementation for sort_type here: #28850
|
Closing in favor of #28850 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description:
ordering_criteria.modeconfig option to theordering_criteriaconfig, which can be used to select the way ordering criteria is applied (regexormtime)mtimemodeIn order to do this, filtering was abstracted out into its own
Filterinterface, where there are now two implementations (a "Regex" implementation and a "mtime" implementation).An optional follow-up performance improvement may be made here, to have the finder return fs.DirEntry directly to query the mtime without making an extra call to os.Stat for each file.
Link to tracking Issue: #27812
Testing:
Documentation:
modeparameter to filelogreceiver docs