Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add documentation for hardware-accelerated compression codecs. #6841

Merged
23 changes: 23 additions & 0 deletions _im-plugin/index-codecs.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,24 @@ For the `zstd` and `zstd_no_dict` codecs, you can optionally specify a compressi
When an index segment is created, it uses the current index codec for compression. If you update the index codec, any segment created after the update will use the new compression algorithm. For specific operation considerations, see [Index codec considerations for index operations](#index-codec-considerations-for-index-operations).
{: .note}

As of OpenSearch 2.14, hardware-accelerated compression codecs for DEFLATE and LZ4 compression algorithms are available. These hardware-accelerated codecs are available on the latest 4th and 5th Gen Intel®️ Xeon®️ processors running Linux kernel 3.10 and later. For all other systems and platforms, the codecs use their corresponding software implementations.
Naarcha-AWS marked this conversation as resolved.
Show resolved Hide resolved

The new hard-accelerated codecs can be used by setting the value of `index.codec`:
Naarcha-AWS marked this conversation as resolved.
Show resolved Hide resolved
* `qat_lz4` (OpenSearch 2.14 and later) -- uses hardware-accelerated LZ4. 
Naarcha-AWS marked this conversation as resolved.
Show resolved Hide resolved
* `qat_deflate` (OpenSearch 2.14 and later) -- uses hardware-accelerated DEFLATE.
Naarcha-AWS marked this conversation as resolved.
Show resolved Hide resolved

`qat_deflate` offers a much better compression ratio than `qat_lz4` with a modest drop in compression and decompression speed.
Naarcha-AWS marked this conversation as resolved.
Show resolved Hide resolved
{: .note}

The `index.codec.compression_level` can be used to specify the compression level for both `qat_lz4` and `qat_deflate`. The new setting, `index.codec.qatmode`, controls the behavior of the hardware accelerator. It takes two valid values:
Naarcha-AWS marked this conversation as resolved.
Show resolved Hide resolved

* `auto` -- uses hardware-accelerated compression. Fails over to use software if the hardware accelerator is not available.
Naarcha-AWS marked this conversation as resolved.
Show resolved Hide resolved
* `hardware` -- use hardware-accelerated compression. An exception is thrown if hardware is not available (guarantees a hardware-only compression).
Naarcha-AWS marked this conversation as resolved.
Show resolved Hide resolved

Refer to the snapshot section of this page for the implications `index.codec.qatmode` on snapshot and restore.
Naarcha-AWS marked this conversation as resolved.
Show resolved Hide resolved

The hardware accelerator that is used is [Intel (R) QAT accelerator](https://www.intel.com/content/www/us/en/architecture-and-technology/intel-quick-assist-technology-overview.html).
Naarcha-AWS marked this conversation as resolved.
Show resolved Hide resolved

## Choosing a codec

The choice of index codec impacts the amount of disk space required to store the index data. Codecs like `best_compression`, `zstd`, and `zstd_no_dict` can achieve higher compression ratios, resulting in smaller index sizes. Conversely, the `default` codec doesn’t prioritize compression ratio, resulting in larger index sizes but faster search operations than `best_compression`.
Expand All @@ -58,6 +76,11 @@ When creating a [snapshot]({{site.url}}{{site.baseurl}}/tuning-your-cluster/avai

When you restore the indexes from a snapshot of a cluster to another cluster, it is important to verify that the target cluster supports the codecs of the segments in the source snapshot. For example, if the source snapshot contains segments of the `zstd` or `zstd_no_dict` codecs (introduced in OpenSearch 2.9), you won't be able to restore the snapshot to a cluster that runs on an older OpenSearch version because it doesn't support these codecs.

For hardware-accelerated compression codecs, available in OpenSearch 2.14 and later, the value of `index.codec.qatmode` affects how snapshots and restores are performed. If the value is `auto` (the default), then snapshots and restores can be performed with no additional instructions. However, if the value is `hardware`, it must be reset to `auto` for the restore process to succeed on systems lacking the hardware accelerator
Naarcha-AWS marked this conversation as resolved.
Show resolved Hide resolved
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is "instructions" the right word here?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How about "without issue"?


You can modify the value of `index.codec.qatmode` during a restore process by setting its value as follows: `"index_settings": {"index.codec.qatmode": "auto"}`.
Naarcha-AWS marked this conversation as resolved.
Show resolved Hide resolved
{: .note}

### Reindexing

When you are performing a [reindex]({{site.url}}{{site.baseurl}}/im-plugin/reindex-data/) operation from a source index, the new segments created in the target index will have the properties of the codec settings of the target index.
Expand Down
15 changes: 13 additions & 2 deletions _install-and-configure/configuring-opensearch/index-settings.md
Original file line number Diff line number Diff line change
Expand Up @@ -79,10 +79,21 @@ OpenSearch supports the following static index-level index settings:
- `best_compression`
- `zstd` (OpenSearch 2.9 and later)
- `zstd_no_dict`(OpenSearch 2.9 and later)
- `qat_lz4`(OpenSearch 2.14 and later, on supported systems)
Naarcha-AWS marked this conversation as resolved.
Show resolved Hide resolved
- `qat_deflate`(OpenSearch 2.14 and later, on supported systems)
Naarcha-AWS marked this conversation as resolved.
Show resolved Hide resolved

For `zstd` and `zstd_no_dict`, you can specify the compression level in the `index.codec.compression_level` setting. For more information, see [Index codec settings]({{site.url}}{{site.baseurl}}/im-plugin/index-codecs/). Optional. Default is `default`.
For `zstd`, `zstd_no_dict`, `qat_lz4`, and `qat_deflate` you can specify the compression level in the `index.codec.compression_level` setting. For more information, see [Index codec settings]({{site.url}}{{site.baseurl}}/im-plugin/index-codecs/). Optional. Default is `default`.
Naarcha-AWS marked this conversation as resolved.
Show resolved Hide resolved

- `index.codec.compression_level` (Integer): The compression level setting provides a tradeoff between compression ratio and speed. A higher compression level results in a higher compression ratio (smaller storage size) with a tradeoff in speed (slower compression and decompression speeds lead to greater indexing and search latencies). Can only be specified if `index.codec` is set to `zstd` and `zstd_no_dict` compression levels in OpenSearch 2.9 and later. Valid values are integers in the [1, 6] range. For more information, see [Index codec settings]({{site.url}}{{site.baseurl}}/im-plugin/index-codecs/). Optional. Default is 3.
- `index.codec.compression_level` (Integer): The compression level setting provides a trade-off between compression ratio and speed. A higher compression level results in a higher compression ratio (smaller storage size) with a trad-eoff in speed (slower compression and decompression speeds lead to greater indexing and search latencies). Can only be specified if index.codec is set to `zstd` and `zstd_no_dict` in OpenSearch 2.9 and later, and `qat_lz4` and `qat_deflate` in OpenSearch 2.14 and later. Valid values are integers in the [1, 6] range. For more information, see [Index codec settings]({{site.url}}{{site.baseurl}}/im-plugin/index-codecs/). Optional. Default is 3.
Naarcha-AWS marked this conversation as resolved.
Show resolved Hide resolved
Naarcha-AWS marked this conversation as resolved.
Show resolved Hide resolved

- `index.codec.qatmode` (String): The hardware acceleration mode used for `qat_lz4` and `qat_deflate` compression codecs:
Naarcha-AWS marked this conversation as resolved.
Show resolved Hide resolved

Valid values are:
Naarcha-AWS marked this conversation as resolved.
Show resolved Hide resolved

- `auto`
Naarcha-AWS marked this conversation as resolved.
Show resolved Hide resolved
- `hardware`
Naarcha-AWS marked this conversation as resolved.
Show resolved Hide resolved

Optional. Default is `auto`. For more information, see [Index codec settings]({{site.url}}{{site.baseurl}}/im-plugin/index-codecs/).
Naarcha-AWS marked this conversation as resolved.
Show resolved Hide resolved

- `index.routing_partition_size` (Integer): The number of shards a custom routing value can go to. Routing helps an imbalanced cluster by relocating values to a subset of shards rather than a single shard. To enable routing, set this value to greater than 1 but less than `index.number_of_shards`. Default is 1.

Expand Down
Loading