Merge Pipeline Processor plugin into Graylog core #4590
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
We should call `InterpreterListener.finishProcessing()` only once, but before that was not necessarily true, as it was being called inside a loop. Instead, now we call it in the same method as `startProcessing()`, after all processing has been finished. Fixes #51
* Providing a message decorator that uses pipelines. * Making decorator configurable. * Allow adding new messages by pipeline decorator. * Adding changes related due to introduced listener. * Adapt to naming changes, using easier forEach idiom. * Changing decorator to work on SearchResponse instead of message list.
Since #32, the behaviour of `to_ip()` changed and now it doesn't raise an exception in those circumstances. Now we use `regex()` to throw an exception instead.
* Providing a message decorator that uses pipelines. * Making decorator configurable. * Allow adding new messages by pipeline decorator. * Adding changes related due to introduced listener. * Adapt to naming changes, using easier forEach idiom. * Changing decorator to work on SearchResponse instead of message list. * Adding decoration stats for pipeline processor decorator. * Add uppercase decorator using pipelines interpreter with preset. * Decorators don't need to generate decoration stats on their own anymore.
Disable building the web-part of this project because it would require a full checkout of the Graylog web interface.
[ci skip]
* Use core Routes instead of literals This allows us to prefix routes if needed. * Make plugin aware of __webpack_public_path__ setting Refs #2564
The server cache is necessary because the named captures support needs a separately compiled regex. So far the cache is only used by the grok function in the pipeline processor Closes #59
* there was a bug with to_string returning null instead of its default value (refs #63) * all core conversion functions now return their "default empty" value if the value is `null` - String: "" - bool: false - double: 0d - long 0L - IP: V4 ANY (0.0.0.0) * adds test cases for all cases, including the edge cases
`evalError()` can no longer trigger the error tested for and was removed. `evalErrorSuppressed()` now tests an illegal default value in `to_ip()`. Fixes #64
This change should not affect `javac` at all, but intellij flags the collect call with having two errors.
Replace `org.jboss.netty.handler.ipfilter.CIDR` with `org.graylog2.utilities.IpSubnet` to get rid of the Netty 3 dependency in the `cidr_match()` function. Refs #4226
The message timestamp used to be in the system default locale instead of UTC. Additionally, this commit adds tests/examples for accessing individual components of the message timestamp in a pipeline rule.
Stabilize date/time based tests
* Add comparison functions for all supported types * Ensure actions were triggered in FunctionsSnippetsTest#comparisons()
Add "parse_unix_milliseconds" functions
* Add support for JsonNode to ´set_fields()`
Sometimes users might want to parse and merge the JSON payload of a message
with the Graylog message without knowing the complete structure of the payload
or without having a fixed structure which could be selectively merged by using
the `json_path()` method.
This commit essentially adds the possiblity to create a pipeline rule emulating
the existing JSON extractor:
rule "json"
when
// some condition
then
let json = parse_json(to_string($message.some_field));
set_fields(json);
end
* Introduce `to_map` function to convert JsonNode to Map
* Fix description of `to_map()` function
...and less Forbidden APIs warnings
Otherwise the named "cached_stageiterators" and "generate_native_code" settings cannot be found. While this *should* work with the `PluginModule` interface and the `PluginModule#getConfigBeans()` method, fixing this would require some refactorings which I'd like to defer to a later point in time
joschi
pushed a commit
that referenced
this pull request
Feb 16, 2018
joschi
pushed a commit
to Graylog2/graylog-plugin-cef
that referenced
this pull request
Feb 16, 2018
In order to be able to build the old Graylog project manifest, this commit temporarily removes the graylog-plugin-pipeline-processor dependency. Refs Graylog2/graylog2-server#4590 Refs Graylog2/graylog2-server#4592 Refs Graylog2/graylog-project-internal#5
joschi
pushed a commit
to Graylog2/graylog-plugin-threatintel
that referenced
this pull request
Feb 16, 2018
In order to be able to build the old Graylog project manifest, this commit temporarily removes the graylog-plugin-pipeline-processor dependency. Refs Graylog2/graylog2-server#4590 Refs Graylog2/graylog2-server#4592 Refs Graylog2/graylog-project-internal#8
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Closes Graylog2/graylog-plugin-pipeline-processor#216