Skip to content

Commit

Permalink
Address comments
Browse files Browse the repository at this point in the history
Signed-off-by: Srikanth Govindarajan <srigovs@amazon.com>
  • Loading branch information
srikanthjg committed Aug 16, 2024
1 parent d8827d3 commit 2eaa778
Show file tree
Hide file tree
Showing 2 changed files with 65 additions and 4 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,7 @@ public class RuleEvaluator {
private static final ObjectMapper yamlMapper = new ObjectMapper(new YAMLFactory());
private final TransformersFactory transformersFactory;


public RuleEvaluator(TransformersFactory transformersFactory) {
this.transformersFactory = transformersFactory;
}
Expand Down Expand Up @@ -63,9 +64,6 @@ public RuleEvaluatorResult isTransformationNeeded(PipelinesDataFlowModel pipelin
.withPipelineName(entry.getKey())
.build();
}
} catch (FileNotFoundException e) {
LOG.error("Template File Not Found");
throw new RuntimeException(e);
} catch (JsonProcessingException e) {
LOG.error("Error processing json");
throw new RuntimeException(e);
Expand Down Expand Up @@ -128,7 +126,7 @@ private RuleFileEvaluation evaluate(String pipelinesJson) {
}

} catch (FileNotFoundException e) {
LOG.debug("Rule File Not Found");
LOG.debug("Rule File Not Found", e);
return RuleFileEvaluation.builder()
.withPluginName(null)
.withRuleFileName(null)
Expand Down
63 changes: 63 additions & 0 deletions data-prepper-plugins/dummy-plugin/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,63 @@
A dummy plugin is used specifically when we want to use pipeline transformation for a
plugin that does not exist.

We can define a rule and template for this plugin by creating a plugin folder and
place rule for which the pipeline configuration would be valid and a corresponding
template to transform to when the rule is valid.

For further details on transformation refer:
/docs/pipeline_configuration_transformation.md


For Example:

User Config:
```yaml
test-pipeline:
source:
file:
path: "/full/path/to/logs_json.log"
record_type: "event"
format: "json"
processor:
- dummy_plugin:
sink:
- stdout:
```
Here dummy_plugin is not really a plugin that is defined in dataprepper, but we can
use the pipeline transformation to convert the user-config to different config based
on the template.
Rule:
```yaml
plugin_name: "dummy-plugin"
apply_when:
- "$.test-pipeline.processor[?(@.dummy_plugin)]"
```
Template:
```yaml
test-pipeline:
source:
file:
path: "/tmp/input-file.log"
processor:
- string_converter:
upper_case: true
sink:
- noop:
```
Output:
```yaml
test-pipeline:
source:
file:
path: "/tmp/input-file.log"
processor:
- string_converter:
upper_case: true
sink:
- noop:
```

0 comments on commit 2eaa778

Please sign in to comment.