Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Data Prepper documentation updates: autogeneration campaign #7707

Merged
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
24 commits
Select commit Hold shift + click to select a range
92253c0
Updates Data Prepper documentation with some missing fields. Adds sup…
dlvenable Jul 15, 2024
207d783
Corrections from PR.
dlvenable Jul 15, 2024
5270f9a
Update _data-prepper/pipelines/configuration/processors/convert_entry…
vagimeli Jul 16, 2024
b7be720
Update _data-prepper/pipelines/configuration/processors/convert_entry…
vagimeli Jul 16, 2024
83b762d
Update _data-prepper/pipelines/configuration/processors/parse_ion.md
vagimeli Jul 16, 2024
a303b27
Update _data-prepper/pipelines/configuration/processors/convert_entry…
vagimeli Jul 16, 2024
13ca7ca
Update _data-prepper/pipelines/configuration/processors/convert_entry…
vagimeli Jul 16, 2024
b42adf7
Update _data-prepper/pipelines/configuration/processors/parse_ion.md
vagimeli Jul 16, 2024
10ce5cd
Update _data-prepper/pipelines/configuration/processors/parse_ion.md
vagimeli Jul 16, 2024
53e6bf3
Update _data-prepper/pipelines/configuration/processors/parse_json.md
vagimeli Jul 16, 2024
dbfb348
Update _data-prepper/pipelines/configuration/processors/parse_json.md
vagimeli Jul 16, 2024
5d57e40
Update _data-prepper/pipelines/configuration/processors/parse_xml.md
vagimeli Jul 16, 2024
8e3a1ae
Update _data-prepper/pipelines/configuration/processors/parse_xml.md
vagimeli Jul 16, 2024
eb2b16b
Update _data-prepper/pipelines/configuration/processors/parse_json.md
vagimeli Jul 16, 2024
73cb9ce
Update _data-prepper/pipelines/configuration/processors/parse_ion.md
vagimeli Jul 16, 2024
dcb06b4
Update _data-prepper/pipelines/configuration/processors/parse_json.md
vagimeli Jul 16, 2024
c6a791e
Update _data-prepper/pipelines/configuration/processors/parse_ion.md
vagimeli Jul 16, 2024
fc90c15
Update _data-prepper/pipelines/configuration/processors/convert_entry…
vagimeli Jul 16, 2024
4d55b4c
Corrections to wording from PR review.
dlvenable Jul 16, 2024
9c56721
Update _data-prepper/pipelines/configuration/processors/parse_ion.md
vagimeli Jul 16, 2024
fc9cd7e
Merge branch 'main' into data-prepper-autogenerate-campaign-1
vagimeli Jul 16, 2024
1914ed6
Merge branch 'main' into data-prepper-autogenerate-campaign-1
vagimeli Jul 18, 2024
f56783c
fixed link - hyphen to underscore
hdhalter Jul 19, 2024
81e354f
Merge branch 'main' into data-prepper-autogenerate-campaign-1
hdhalter Jul 19, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -14,10 +14,22 @@ The `convert_entry_type` processor converts a value type associated with the spe

You can configure the `convert_entry_type` processor with the following options.

<!--
This table is autogenerated. Do not edit it.
- name: convert_entry_type
- pluginType: processor
- source: https://github.com/opensearch-project/data-prepper/blob/7d15115c281687aab50e5c471fd210cb1ef90fc5/data-prepper-plugins/mutate-event-processors/src/main/java/org/opensearch/dataprepper/plugins/processor/mutateevent/ConvertEntryTypeProcessorConfig.java
-->

| Option | Required | Description |
| :--- | :--- | :--- |
| `key`| Yes | Keys whose value needs to be converted to a different type. |
| `type` | No | Target type for the key-value pair. Possible values are `integer`, `double`, `string`, and `Boolean`. Default value is `integer`. |
| `key`| Yes | Key whose value needs to be converted to a different type. |
| `keys`| Yes | Keys whose value needs to be converted to a different type. |
| `type` | No | Target type for the key-value pair. Possible values are `integer`, `long`, `double`, `big_decimal`, `string`, and `boolean`. Default value is `integer`. |
| `null_values` | No | String representation of what constitutes a `null` value. If the field value equals one of these, then value is considered `null` and is converted to `null`. |
vagimeli marked this conversation as resolved.
Show resolved Hide resolved
| `scale` | No | Modifies the scale when used to convert to a `big_decimal`. The default value is `0`. |
Copy link
Collaborator

@vagimeli vagimeli Jul 16, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@dlvenable @chenqi0805 Is this saying that scale is only modified when the option is converted to big_decimal?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I just pushed a new commit that attempts to correct the wording.

We are modifying the scale of the big_decimal value itself. This field is only applicable for big_decimal.

| `tags_on_failure` | No | A list of tags to add to the event metadata when the event fails to convert. |
vagimeli marked this conversation as resolved.
Show resolved Hide resolved
| `convert_when` | No | Specifies a condition for performing the `convert_entry_type` operation using a [Data Prepper expression]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/expression-syntax/). If specified, the `convert_entry_type` operation will only run when the expression evaluates to true. |
vagimeli marked this conversation as resolved.
Show resolved Hide resolved

## Usage

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -14,12 +14,22 @@ The `parse_ion` processor parses [Amazon Ion](https://amazon-ion.github.io/ion-d

You can configure the `parse_ion` processor with the following options.

<!--
This table is autogenerated. Do not edit it.
- name: parse_ion
- pluginType: processor
- source: https://github.com/opensearch-project/data-prepper/blob/253e59245fd9c39c959c1c8caaeff1b226a5a0ab/data-prepper-plugins/parse-json-processor/src/main/java/org/opensearch/dataprepper/plugins/processor/parse/ion/ParseIonProcessorConfig.java
-->

| Option | Required | Type | Description |
| :--- | :--- | :--- | :--- |
| `source` | No | String | The field in the `event` that is parsed. Default value is `message`. |
| `destination` | No | String | The destination field of the parsed JSON. Defaults to the root of the `event`. Cannot be `""`, `/`, or any white-space-only `string` because these are not valid `event` fields. |
| `pointer` | No | String | A JSON pointer to the field to be parsed. There is no `pointer` by default, meaning that the entire `source` is parsed. The `pointer` can access JSON array indexes as well. If the JSON pointer is invalid, then the entire `source` data is parsed into the outgoing `event`. If the key that is pointed to already exists in the `event` and the `destination` is the root, then the pointer uses the entire path of the key. |
| `tags_on_failure` | No | String | A list of strings that specify the tags to be set in the event that the processors fails or an unknown exception occurs while parsing.
| `parse_when` | No | String | Specifies under what conditions the processor should perform parsing. Default is no condition. Accepts a Data Prepper expression string following the [Data Prepper Expression Syntax]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/expression-syntax/). |
vagimeli marked this conversation as resolved.
Show resolved Hide resolved
| `overwrite_if_destination_exists` | No | Boolean | If set to `true` then overwrite the destination. Set to `false` to prevent changing a destination value that exists. Defaults to `true`. |
vagimeli marked this conversation as resolved.
Show resolved Hide resolved
| `delete_source` | No | Boolean | If set to `true` then overwrite the destination. Set to `false` to prevent changing a destination value that exists. Defaults to `true` |
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@dlvenable @chenqi0805 The description for delete_source is the same as line 30. Is this intentional?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That was a mistake on my part. I just pushed a new change that should correct this.

| `tags_on_failure` | No | String | A list of strings that specify the tags to be set in the event that the processors fails or an unknown exception occurs while parsing.
vagimeli marked this conversation as resolved.
Show resolved Hide resolved

## Usage

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -15,11 +15,21 @@ The `parse_json` processor parses JSON data for an event, including any nested f

You can configure the `parse_json` processor with the following options.

<!--
This table is autogenerated. Do not edit it.
- name: obfuscate
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Miscopy

- pluginType: parse_json
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

processor

- source: https://github.com/opensearch-project/data-prepper/blob/253e59245fd9c39c959c1c8caaeff1b226a5a0ab/data-prepper-plugins/parse-json-processor/src/main/java/org/opensearch/dataprepper/plugins/processor/parse/json/ParseJsonProcessorConfig.java
-->

| Option | Required | Type | Description |
| :--- | :--- | :--- | :--- |
| `source` | No | String | The field in the `event` that will be parsed. Default value is `message`. |
| `destination` | No | String | The destination field of the parsed JSON. Defaults to the root of the `event`. Cannot be `""`, `/`, or any white-space-only `string` because these are not valid `event` fields. |
| `pointer` | No | String | A JSON pointer to the field to be parsed. There is no `pointer` by default, meaning the entire `source` is parsed. The `pointer` can access JSON array indexes as well. If the JSON pointer is invalid then the entire `source` data is parsed into the outgoing `event`. If the key that is pointed to already exists in the `event` and the `destination` is the root, then the pointer uses the entire path of the key. |
| `parse_when` | No | String | Specifies under what conditions the processor should perform parsing. Default is no condition. Accepts a Data Prepper expression string following the [Data Prepper Expression Syntax]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/expression-syntax/). |
vagimeli marked this conversation as resolved.
Show resolved Hide resolved
| `overwrite_if_destination_exists` | No | Boolean | If set to `true` then overwrite the destination. Set to `false` to prevent changing a destination value that exists. Defaults to `true`. |
vagimeli marked this conversation as resolved.
Show resolved Hide resolved
| `tags_on_failure` | No | String | A list of strings that specify the tags to be set in the event that the processors fails or an unknown exception occurs while parsing.
vagimeli marked this conversation as resolved.
Show resolved Hide resolved

## Usage

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -14,13 +14,21 @@ The `parse_xml` processor parses XML data for an event.

You can configure the `parse_xml` processor with the following options.

<!--
This table is autogenerated. Do not edit it.
- name: parse_xml
- pluginType: processor
- source: https://github.com/opensearch-project/data-prepper/blob/253e59245fd9c39c959c1c8caaeff1b226a5a0ab/data-prepper-plugins/parse-json-processor/src/main/java/org/opensearch/dataprepper/plugins/processor/parse/xml/ParseXmlProcessorConfig.java
-->

| Option | Required | Type | Description |
| :--- | :--- | :--- | :--- |
| `source` | No | String | Specifies which `event` field to parse. |
| `destination` | No | String | The destination field of the parsed XML. Defaults to the root of the `event`. Cannot be `""`, `/`, or any white-space-only string because these are not valid `event` fields. |
| `pointer` | No | String | A JSON pointer to the field to be parsed. The value is null by default, meaning that the entire `source` is parsed. The `pointer` can access JSON array indexes as well. If the JSON pointer is invalid, then the entire `source` data is parsed into the outgoing `event` object. If the key that is pointed to already exists in the `event` object and the `destination` is the root, then the pointer uses the entire path of the key. |
| `parse_when` | No | String | Specifies under what conditions the processor should perform parsing. Default is no condition. Accepts a Data Prepper expression string following the [Data Prepper Expression Syntax]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/expression-syntax/). |
| `tags_on_failure` | No | String | A list of strings that specify the tags to be set if the processor fails or an unknown exception occurs while parsing.
| `overwrite_if_destination_exists` | No | Boolean | If set to `true` then overwrite the destination. Set to `false` to prevent changing a destination value that exists. Defaults to `true`. |
vagimeli marked this conversation as resolved.
Show resolved Hide resolved
| `tags_on_failure` | No | String | A list of strings that specify the tags to be set if the processor fails or an unknown exception occurs while parsing.
vagimeli marked this conversation as resolved.
Show resolved Hide resolved

## Usage

Expand Down
11 changes: 9 additions & 2 deletions _data-prepper/pipelines/configuration/processors/write_json.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,8 +11,15 @@ nav_order: 56

The `write_json` processor converts an object in an event into a JSON string. You can customize the processor to choose the source and target field names.

Option | Description | Example
:--- | :--- | :---
<!--
This table is autogenerated. Do not edit it.
- name: write_json
- pluginType: processor
- source: https://github.com/opensearch-project/data-prepper/blob/f0bd8d8e4773dc3d7318c19b8623c2213a099a86/data-prepper-plugins/write-json-processor/src/main/java/org/opensearch/dataprepper/plugins/processor/write_json/WriteJsonProcessorConfig.java
-->

Option | Description | Example
:--- | :--- | :---
source | Mandatory field that specifies the name of the field in the event containing the message or object to be parsed. | If `source` is set to `"message"` and the input is `{"message": {"key1":"value1", "key2":{"key3":"value3"}}}`, then the `write_json` processor outputs the event as `"{\"key1\":\"value1\",\"key2\":{\"key3\":\"value3\"}}"`.
target | An optional field that specifies the name of the field in which the resulting JSON string should be stored. If `target` is not specified, then the `source` field is used. | `key1`

Loading