Register system index descriptors through SystemIndexPlugin.getSystemIndexDescriptors #2772

cwperks · 2024-06-24T18:35:47Z

Description

This PR registers the system indices in this plugin through the SystemIndexPlugin extension point in core. These indices will not be functionally different than they are today, its just a formal registration as a system index.

Issues Resolved

Related to opensearch-project/security#4439

Check List

New functionality includes testing.
- All tests pass, including unit test, integration test and doctest
New functionality has been documented.
- New functionality has javadoc added
- New functionality has user manual doc added
Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

…IndexDescriptors Signed-off-by: Craig Perkins <cwperx@amazon.com>

noCharger · 2024-06-24T18:44:02Z

@cwperks The Flint Request Index is another system index. Will it be covered in this PR?

cwperks · 2024-06-24T18:49:50Z

@noCharger Can you link me to where the index is defined?

I created this PR based off of the indices currently tracked by the security repo: https://github.com/opensearch-project/security/blob/main/src/main/java/org/opensearch/security/tools/democonfig/SecuritySettingsConfigurer.java#L72

cwperks · 2024-06-25T18:30:10Z

@noCharger This index? https://github.com/opensearch-project/sql/blob/main/async-query-core/src/main/java/org/opensearch/sql/spark/execution/statestore/OpenSearchStateStoreUtil.java#L18

Should I add .query_execution_request* to the list of system index descriptors?

derek-ho · 2024-06-27T18:03:57Z

@noCharger This index? https://github.com/opensearch-project/sql/blob/main/async-query-core/src/main/java/org/opensearch/sql/spark/execution/statestore/OpenSearchStateStoreUtil.java#L18

Should I add .query_execution_request* to the list of system index descriptors?

Yes looks like that should be added

noCharger · 2024-06-27T18:42:40Z

@noCharger This index? https://github.com/opensearch-project/sql/blob/main/async-query-core/src/main/java/org/opensearch/sql/spark/execution/statestore/OpenSearchStateStoreUtil.java#L18

Should I add .query_execution_request* to the list of system index descriptors?

Yes please, also could you point me if any additional change needed for CI?

Signed-off-by: Craig Perkins <cwperx@amazon.com>

cwperks · 2024-07-01T14:25:36Z

Yes looks like that should be added

Added

cwperks · 2024-07-01T16:24:34Z

Yes please, also could you point me if any additional change needed for CI?

@noCharger On the main branch, security is targeting Java 21. The CI failure is related to the Java version used for CI checks.

Exception in thread "main" java.lang.IllegalStateException: opensearch-security requires Java 21:, your system: 11.0.23+9

noCharger · 2024-07-05T21:36:16Z

Yes please, also could you point me if any additional change needed for CI?

@noCharger On the main branch, security is targeting Java 21. The CI failure is related to the Java version used for CI checks.

Exception in thread "main" java.lang.IllegalStateException: opensearch-security requires Java 21:, your system: 11.0.23+9

@cwperks #2787 is merged now

cwperks · 2024-07-08T18:05:34Z

@noCharger I rebased, but there are still CI Check failures.

Have you seen these failures before?

Run actions/checkout@v3
/usr/bin/docker exec  7e91a6adc94f2190b678754e7db3328083f510b4023d9a3240f1f3a65a1b6e00 sh -c "cat /etc/*release | grep ^ID"
/__e/node20/bin/node: /lib64/libm.so.6: version `GLIBC_2.27' not found (required by /__e/node20/bin/node)
/__e/node20/bin/node: /lib64/libc.so.6: version `GLIBC_2.28' not found (required by /__e/node20/bin/node)

ref: https://github.com/opensearch-project/sql/actions/runs/9810713201/job/27092980700?pr=2772

Signed-off-by: Craig Perkins <cwperx@amazon.com>

This reverts commit b35573b.

Signed-off-by: Derek Ho <dxho@amazon.com>

cwperks · 2024-07-09T13:50:40Z

@derek-ho @noCharger All CI checks have passed. Can you re-review this PR?

derek-ho · 2024-07-09T14:40:28Z

plugin/src/main/java/org/opensearch/sql/plugin/SQLPlugin.java

+            OpenSearchDataSourceMetadataStorage.DATASOURCE_INDEX_NAME, "SQL DataSources index"));
+    systemIndexDescriptors.add(
+        new SystemIndexDescriptor(
+            SPARK_REQUEST_BUFFER_INDEX_NAME + "*", "SQL Spark Request Buffer index pattern"));


@noCharger does each datasource create its own request index?

Seems like it is the case based on:

sql/async-query-core/src/main/java/org/opensearch/sql/spark/execution/statestore/OpenSearchStateStoreUtil.java

Line 18 in f241f34

"%s_%s", SPARK_REQUEST_BUFFER_INDEX_NAME, datasourceName.toLowerCase(Locale.ROOT));

Yes each datasource creates an index based on its name. Is it enough to define the pattern?

What is the purpose of this registration?

opensearch-project/security#4535

Security currently maintains a list of system indices through an OpenSearch setting. This change will allow security to rely on a registry of these indices from the core instead of relying on a setting.

Part of this greater effort: opensearch-project/security#4439

derek-ho

LGTM but let's confirm we need the * pattern @noCharger can you take a look here?

…IndexDescriptors (#2772) * Register system index descriptors through SystemIndexPlugin.getSystemIndexDescriptors Signed-off-by: Craig Perkins <cwperx@amazon.com> * Add Spark Buffer index pattern Signed-off-by: Craig Perkins <cwperx@amazon.com> * Upgrade to checkout v4 Signed-off-by: Craig Perkins <cwperx@amazon.com> * Fix CI issues Signed-off-by: Craig Perkins <cwperx@amazon.com> * Move env Signed-off-by: Craig Perkins <cwperx@amazon.com> * Revert "Upgrade to checkout v4" This reverts commit b35573b. --------- Signed-off-by: Craig Perkins <cwperx@amazon.com> Signed-off-by: Derek Ho <dxho@amazon.com> Co-authored-by: Derek Ho <dxho@amazon.com> (cherry picked from commit d3ecb2d) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…IndexDescriptors (#2772) (#2817) * Register system index descriptors through SystemIndexPlugin.getSystemIndexDescriptors * Add Spark Buffer index pattern * Upgrade to checkout v4 * Fix CI issues * Move env * Revert "Upgrade to checkout v4" This reverts commit b35573b. --------- (cherry picked from commit d3ecb2d) Signed-off-by: Craig Perkins <cwperx@amazon.com> Signed-off-by: Derek Ho <dxho@amazon.com> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: Derek Ho <dxho@amazon.com>

…IndexDescriptors (opensearch-project#2772) * Register system index descriptors through SystemIndexPlugin.getSystemIndexDescriptors Signed-off-by: Craig Perkins <cwperx@amazon.com> * Add Spark Buffer index pattern Signed-off-by: Craig Perkins <cwperx@amazon.com> * Upgrade to checkout v4 Signed-off-by: Craig Perkins <cwperx@amazon.com> * Fix CI issues Signed-off-by: Craig Perkins <cwperx@amazon.com> * Move env Signed-off-by: Craig Perkins <cwperx@amazon.com> * Revert "Upgrade to checkout v4" This reverts commit b35573b. --------- Signed-off-by: Craig Perkins <cwperx@amazon.com> Signed-off-by: Derek Ho <dxho@amazon.com> Co-authored-by: Derek Ho <dxho@amazon.com>

Register system index descriptors through SystemIndexPlugin.getSystem…

b78436d

…IndexDescriptors Signed-off-by: Craig Perkins <cwperx@amazon.com>

cwperks requested review from ps48, kavithacm, derek-ho, joshuali925, dai-chen, YANG-DB, rupal-bq, mengweieric, vamsi-amazon, Swiddis, penghuo, seankao-az, MaxKsyunz, Yury-Fridlyand, anirudha, forestmvey, acarbonetto, GumpacG and ykmr1224 as code owners June 24, 2024 18:35

Add Spark Buffer index pattern

aea687b

Signed-off-by: Craig Perkins <cwperx@amazon.com>

cwperks added 2 commits July 2, 2024 16:01

Merge branch 'main' into system-index-registration

fc9e90e

Merge branch 'main' into system-index-registration

8fa8234

cwperks and others added 5 commits July 8, 2024 14:13

Upgrade to checkout v4

b35573b

Signed-off-by: Craig Perkins <cwperx@amazon.com>

Fix CI issues

e3c4dbe

Signed-off-by: Craig Perkins <cwperx@amazon.com>

Move env

9fd5ed0

Signed-off-by: Craig Perkins <cwperx@amazon.com>

Revert "Upgrade to checkout v4"

146173f

This reverts commit b35573b.

Merge branch 'main' into system-index-registration

cb5fbae

Signed-off-by: Derek Ho <dxho@amazon.com>

derek-ho reviewed Jul 9, 2024

View reviewed changes

derek-ho approved these changes Jul 9, 2024

View reviewed changes

noCharger approved these changes Jul 9, 2024

View reviewed changes

vamsi-amazon approved these changes Jul 10, 2024

View reviewed changes

derek-ho merged commit d3ecb2d into opensearch-project:main Jul 10, 2024
12 checks passed

derek-ho added the backport 2.x label Jul 10, 2024

opensearch-trigger-bot bot mentioned this pull request Jul 10, 2024

[Backport 2.x] Register system index descriptors through SystemIndexPlugin.getSystemIndexDescriptors #2817

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Register system index descriptors through SystemIndexPlugin.getSystemIndexDescriptors #2772

Register system index descriptors through SystemIndexPlugin.getSystemIndexDescriptors #2772

cwperks commented Jun 24, 2024

noCharger commented Jun 24, 2024

cwperks commented Jun 24, 2024

cwperks commented Jun 25, 2024

derek-ho commented Jun 27, 2024

noCharger commented Jun 27, 2024

cwperks commented Jul 1, 2024

cwperks commented Jul 1, 2024

noCharger commented Jul 5, 2024

cwperks commented Jul 8, 2024 •

edited

Loading

cwperks commented Jul 9, 2024

derek-ho Jul 9, 2024

derek-ho Jul 9, 2024

vamsi-amazon Jul 10, 2024

vamsi-amazon Jul 10, 2024

cwperks Jul 10, 2024

cwperks Jul 10, 2024

derek-ho left a comment

Register system index descriptors through SystemIndexPlugin.getSystemIndexDescriptors #2772

Register system index descriptors through SystemIndexPlugin.getSystemIndexDescriptors #2772

Conversation

cwperks commented Jun 24, 2024

Description

Issues Resolved

Check List

noCharger commented Jun 24, 2024

cwperks commented Jun 24, 2024

cwperks commented Jun 25, 2024

derek-ho commented Jun 27, 2024

noCharger commented Jun 27, 2024

cwperks commented Jul 1, 2024

cwperks commented Jul 1, 2024

noCharger commented Jul 5, 2024

cwperks commented Jul 8, 2024 • edited Loading

cwperks commented Jul 9, 2024

derek-ho Jul 9, 2024

Choose a reason for hiding this comment

derek-ho Jul 9, 2024

Choose a reason for hiding this comment

vamsi-amazon Jul 10, 2024

Choose a reason for hiding this comment

vamsi-amazon Jul 10, 2024

Choose a reason for hiding this comment

cwperks Jul 10, 2024

Choose a reason for hiding this comment

cwperks Jul 10, 2024

Choose a reason for hiding this comment

derek-ho left a comment

Choose a reason for hiding this comment

cwperks commented Jul 8, 2024 •

edited

Loading