Skip to content

Commit

Permalink
MAINT: documentation update for certain processors (#7713) (#7882)
Browse files Browse the repository at this point in the history
  • Loading branch information
opensearch-trigger-bot[bot] committed Jul 31, 2024
1 parent 63bb06d commit 37b8c72
Show file tree
Hide file tree
Showing 14 changed files with 86 additions and 9 deletions.
2 changes: 1 addition & 1 deletion _data-prepper/common-use-cases/log-enrichment.md
Original file line number Diff line number Diff line change
Expand Up @@ -370,7 +370,7 @@ The `date` processor can generate timestamps for incoming events if you specify

### Deriving punctuation patterns

The [`substitute_string`]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/configuration/processors/substitute-string/) processor (which is one of the mutate string processors) lets you derive a punctuation pattern from incoming events. In the following example pipeline, the processor will scan incoming Apache log events and derive punctuation patterns from them:
The [`substitute_string`]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/configuration/processors/substitute_string/) processor (which is one of the mutate string processors) lets you derive a punctuation pattern from incoming events. In the following example pipeline, the processor will scan incoming Apache log events and derive punctuation patterns from them:

```yaml
processor:
Expand Down
6 changes: 3 additions & 3 deletions _data-prepper/common-use-cases/trace-analytics.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ To monitor trace analytics in Data Prepper, we provide three pipelines: `entry-p

### OpenTelemetry trace source

The [OpenTelemetry source]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/configuration/processors/otel-trace-raw/) accepts trace data from the OpenTelemetry Collector. The source follows the [OpenTelemetry Protocol](https://github.com/open-telemetry/opentelemetry-specification/tree/master/specification/protocol) and officially supports transport over gRPC and the use of industry-standard encryption (TLS/HTTPS).
The [OpenTelemetry source]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/configuration/processors/otel_traces/) accepts trace data from the OpenTelemetry Collector. The source follows the [OpenTelemetry Protocol](https://github.com/open-telemetry/opentelemetry-specification/tree/master/specification/protocol) and officially supports transport over gRPC and the use of industry-standard encryption (TLS/HTTPS).

### Processor

Expand All @@ -49,8 +49,8 @@ OpenSearch provides a generic sink that writes data to OpenSearch as the destina

The sink provides specific configurations for the trace analytics feature. These configurations allow the sink to use indexes and index templates specific to trace analytics. The following OpenSearch indexes are specific to trace analytics:

* otel-v1-apm-span –- The *otel-v1-apm-span* index stores the output from the [otel_traces_raw]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/configuration/processors/otel-trace-raw/) processor.
* otel-v1-apm-service-map –- The *otel-v1-apm-service-map* index stores the output from the [service_map_stateful]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/configuration/processors/service-map-stateful/) processor.
* otel-v1-apm-span –- The *otel-v1-apm-span* index stores the output from the [otel_traces_raw]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/configuration/processors/otel_traces/) processor.
* otel-v1-apm-service-map –- The *otel-v1-apm-service-map* index stores the output from the [service_map_stateful]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/configuration/processors/service_map/) processor.

## Trace tuning

Expand Down
7 changes: 7 additions & 0 deletions _data-prepper/pipelines/configuration/processors/date.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,13 @@ The `date` processor adds a default timestamp to an event, parses timestamp fiel

The following table describes the options you can use to configure the `date` processor.

<!--
This table is autogenerated. Do not edit it.
- name: date
- pluginType: processor
- source: https://github.com/opensearch-project/data-prepper/blob/c4455a7785bc2da4358067c217be7085e0bc8d0f/data-prepper-plugins/date-processor/src/main/java/org/opensearch/dataprepper/plugins/processor/date/DateProcessorConfig.java
-->


Option | Required | Type | Description
:--- | :--- | :--- | :---
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,13 @@ The `delete_entries` processor deletes entries, such as key-value pairs, from an

You can configure the `delete_entries` processor with the following options.

<!--
This table is autogenerated. Do not edit it.
- name: delete_entries
- pluginType: processor
- source: https://github.com/opensearch-project/data-prepper/blob/c4455a7785bc2da4358067c217be7085e0bc8d0f/data-prepper-plugins/mutate-event-processors/src/main/java/org/opensearch/dataprepper/plugins/processor/mutateevent/DeleteEntryProcessorConfig.java
-->

| Option | Required | Description |
:--- | :--- | :---
| `with_keys` | Yes | An array of keys for the entries to be deleted. |
Expand Down
9 changes: 8 additions & 1 deletion _data-prepper/pipelines/configuration/processors/grok.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
layout: default
title: Grok
title: grok
parent: Processors
grand_parent: Pipelines
nav_order: 50
Expand All @@ -14,6 +14,13 @@ The Grok processor uses pattern matching to structure and extract important keys

The following table describes options you can use with the Grok processor to structure your data and make your data easier to query.

<!--
This table is autogenerated. Do not edit it.
- name: grok
- pluginType: processor
- source: https://github.com/opensearch-project/data-prepper/blob/c4455a7785bc2da4358067c217be7085e0bc8d0f/data-prepper-plugins/grok-processor/src/main/java/org/opensearch/dataprepper/plugins/processor/grok/GrokProcessorConfig.java
-->

Option | Required | Type | Description
:--- | :--- |:--- | :---
`break_on_match` | No | Boolean | Specifies whether to match all patterns (`true`) or stop once the first successful match is found (`false`). Default is `true`.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -16,14 +16,21 @@ To get started, add the following processor to your `pipeline.yaml` configuratio

``` yaml
processor:
- otel_metrics_raw_processor:
- otel_metrics:
```
{% include copy.html %}
## Configuration
You can use the following optional parameters to configure histogram buckets and their default values. A histogram displays numerical data by grouping data into buckets. You can use histogram buckets to view sets of events that are organized by the total event count and aggregate sum for all events. For more detailed information, see [OpenTelemetry Histograms](https://opentelemetry.io/docs/reference/specification/metrics/data-model/#histogram).
<!--
This table is autogenerated. Do not edit it.
- name: otel_metrics
- pluginType: processor
- source: https://github.com/opensearch-project/data-prepper/blob/main/data-prepper-plugins/otel-metrics-raw-processor/src/main/java/org/opensearch/dataprepper/plugins/processor/otelmetrics/OtelMetricsRawProcessorConfig.java
-->
| Parameter | Default value | Description |
| :--- | :--- | :--- |
| `calculate_histogram_buckets` | `True` | Whether or not to calculate histogram buckets. |
Expand Down
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
layout: default
title: otel_trace
title: otel_traces
parent: Processors
grand_parent: Pipelines
nav_order: 75
Expand All @@ -23,6 +23,13 @@ This processor includes the following parameters.

The following table describes the options you can use to configure the `otel_trace` processor.

<!--
This table is autogenerated. Do not edit it.
- name: otel_traces
- pluginType: processor
- source: https://github.com/opensearch-project/data-prepper/blob/c4455a7785bc2da4358067c217be7085e0bc8d0f/data-prepper-plugins/otel-trace-raw-processor/src/main/java/org/opensearch/dataprepper/plugins/processor/oteltrace/OtelTraceRawProcessorConfig.java
-->

Option | Required | Type | Description
:--- | :--- | :--- | :---
trace_flush_interval | No | Integer | Represents the time interval in seconds to flush all the descendant spans without any root span. Default is 180.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,13 @@ The `service_map` processor uses OpenTelemetry data to create a distributed serv

The following table describes the option you can use to configure the `service_map` processor.

<!--
This table is autogenerated. Do not edit it.
- name: service_map
- pluginType: processor
- source: https://github.com/opensearch-project/data-prepper/blob/c4455a7785bc2da4358067c217be7085e0bc8d0f/data-prepper-plugins/service-map-stateful/src/main/java/org/opensearch/dataprepper/plugins/processor/ServiceMapProcessorConfig.java
-->

Option | Required | Type | Description
:--- | :--- | :--- | :---
window_duration | No | Integer | Represents the fixed time window, in seconds, during which service map relationships are evaluated. Default value is 180.
Expand All @@ -32,7 +39,7 @@ The following table describes common [Abstract processor](https://github.com/ope
| `recordsOut` | Counter | Metric representing the egress of records from a pipeline component. |
| `timeElapsed` | Timer | Metric representing the time elapsed during execution of a pipeline component. |

The `service-map-stateful` processor includes following custom metrics:
The `service_map` processor includes following custom metrics:

* `traceGroupCacheCount`: The number of trace groups in the trace group cache.
* `spanSetCount`: The number of span sets in the span set collection.
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,13 @@ nav_order: 100

The `split_string` processor splits a field into an array using a delimiting character and is a [mutate string](https://github.com/opensearch-project/data-prepper/tree/main/data-prepper-plugins/mutate-string-processors#mutate-string-processors) processor. The following table describes the options you can use to configure the `split_string` processor.

<!--
This table is autogenerated. Do not edit it.
- name: split_string
- pluginType: processor
- source: https://github.com/opensearch-project/data-prepper/blob/c4455a7785bc2da4358067c217be7085e0bc8d0f/data-prepper-plugins/mutate-string-processors/src/main/java/org/opensearch/dataprepper/plugins/processor/mutatestring/SplitStringProcessorConfig.java
-->

Option | Required | Type | Description
:--- | :--- | :--- | :---
entries | Yes | List | List of entries. Valid values are `source`, `delimiter`, and `delimiter_regex`.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,13 @@ nav_order: 105

The `string_converter` processor converts a string to uppercase or lowercase. You can use it as an example for developing your own processor. The following table describes the option you can use to configure the `string_converter` processor.

<!--
This table is autogenerated. Do not edit it.
- name: string_converter
- pluginType: processor
- source: https://github.com/opensearch-project/data-prepper/blob/c4455a7785bc2da4358067c217be7085e0bc8d0f/data-prepper-plugins/common/src/main/java/org/opensearch/dataprepper/plugins/processor/StringProcessor.java
-->

Option | Required | Type | Description
:--- | :--- | :--- | :---
upper_case | No | Boolean | Whether to convert to uppercase (`true`) or lowercase (`false`).
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,13 @@ The `substitute_string` processor matches a key's value against a regular expres

The following table describes the options you can use to configure the `substitute_string` processor.

<!--
This table is autogenerated. Do not edit it.
- name: substitute_string
- pluginType: processor
- source: https://github.com/opensearch-project/data-prepper/blob/c4455a7785bc2da4358067c217be7085e0bc8d0f/data-prepper-plugins/mutate-string-processors/src/main/java/org/opensearch/dataprepper/plugins/processor/mutatestring/SubstituteStringProcessorConfig.java
-->

Option | Required | Type | Description
:--- | :--- | :--- | :---
entries | Yes | List | List of entries. Valid values are `source`, `from`, and `to`.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,13 @@ nav_order: 120

The `trim_string` processor removes white space from the beginning and end of a key and is a [mutate string](https://github.com/opensearch-project/data-prepper/tree/main/data-prepper-plugins/mutate-string-processors#mutate-string-processors) processor. The following table describes the option you can use to configure the `trim_string` processor.

<!--
This table is autogenerated. Do not edit it.
- name: trim_string
- pluginType: processor
- source: https://github.com/opensearch-project/data-prepper/blob/c4455a7785bc2da4358067c217be7085e0bc8d0f/data-prepper-plugins/mutate-string-processors/src/main/java/org/opensearch/dataprepper/plugins/processor/mutatestring/WithKeysConfig.java
-->

Option | Required | Type | Description
:--- | :--- | :--- | :---
with_keys | Yes | List | A list of keys to trim the white space from.
Expand Down
7 changes: 7 additions & 0 deletions _data-prepper/pipelines/configuration/processors/truncate.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,13 @@ The `truncate` processor truncates a key's value at the beginning, the end, or o

You can configure the `truncate` processor using the following options.

<!--
This table is autogenerated. Do not edit it.
- name: truncate
- pluginType: processor
- source: https://github.com/opensearch-project/data-prepper/blob/c4455a7785bc2da4358067c217be7085e0bc8d0f/data-prepper-plugins/truncate-processor/src/main/java/org/opensearch/dataprepper/plugins/processor/truncate/TruncateProcessorConfig.java
-->

Option | Required | Type | Description
:--- | :--- | :--- | :---
`entries` | Yes | String list | A list of entries to add to an event.
Expand Down
2 changes: 1 addition & 1 deletion _observing-your-data/trace/ta-dashboards.md
Original file line number Diff line number Diff line change
Expand Up @@ -48,7 +48,7 @@ The **Trace Analytics** application includes two options: **Services** and **Tra
The plugin requires you to use [Data Prepper]({{site.url}}{{site.baseurl}}/data-prepper/) to process and visualize OTel data and relies on the following Data Prepper pipelines for OTel correlations and service map calculations:

- [Trace analytics pipeline]({{site.url}}{{site.baseurl}}/data-prepper/common-use-cases/trace-analytics/)
- [Service map pipeline]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/configuration/processors/service-map-stateful/)
- [Service map pipeline]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/configuration/processors/service_map/)

### Standardized telemetry data

Expand Down

0 comments on commit 37b8c72

Please sign in to comment.