Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Extensions] Introduce Identity Plugin to Core #7246

Merged
merged 39 commits into from
May 2, 2023

Conversation

stephen-crawford
Copy link
Contributor

@stephen-crawford stephen-crawford commented Apr 19, 2023

Description

Takes over for #5925 since Peter is out.
Per the origing PR:

Adding a new module and plugin interface that provides identity and access control inside of OpenSearch. This is the founding building block, see more high level thoughts here on our recent blog.

The new extension point, IdentityPlugin, is added with IdentityService handling that plugin interface. IdentitySevice authenticates users and enables access control systems. Adding HTTP basic authentication in the RestController the default NoopIdentityPlugin changes no behavior.

When the identity feature flag is enabled the identity-shiro module is loaded into the IdentityService. This implementations includes an admin user backed through the Apache Shiro system. By partitioning these features through a plugin interface it isolate the dependencies if we ever switch out the underlying library.

Project tracking addition features and functionality: https://github.com/orgs/opensearch-project/projects/89?pane=info

Issues Resolved:

#5880

Check List

  • New functionality includes testing.
    • All tests pass
  • New functionality has been documented.
    • New functionality has javadoc added
  • Commits are signed per the DCO using --signoff
  • Commit changes are listed out in CHANGELOG.md file (See: Changelog)

peternied and others added 4 commits February 3, 2023 20:12
Adding a new module and plugin interface that provides identity and access control inside of OpenSearch. This is the founding building block, see more high level thoughts here on our recent blog. https://opensearch.org/blog/Introducing-Identity/

The new extension point, IdentityPlugin, is added with IdentityService handling that plugin interface. IdentitySevice authenticates users and enables access control systems. Adding HTTP basic authentication in the RestController the default NoopIdentityPlugin changes no behavior.

When the identity feature flag is enabled the identity-shiro module is loaded into the IdentityService. This implementations includes an admin user backed through the Apache Shiro system. By partitioning these features through a plugin interface it isolate the dependencies if we ever switch out the underlying library.

Project tracking addition features and functionality
- https://github.com/orgs/opensearch-project/projects/89?pane=info

Signed-off-by: Peter Nied <petern@amazon.com>
Signed-off-by: Peter Nied <petern@amazon.com>
Signed-off-by: Peter Nied <petern@amazon.com>
Signed-off-by: Stephen Crawford <steecraw@amazon.com>
@stephen-crawford stephen-crawford changed the title Identity core Introduce Identity Plugin to Core Apr 19, 2023
@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

Signed-off-by: Stephen Crawford <steecraw@amazon.com>
Signed-off-by: Stephen Crawford <steecraw@amazon.com>
Signed-off-by: Stephen Crawford <steecraw@amazon.com>
@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

Signed-off-by: Stephen Crawford <steecraw@amazon.com>
Signed-off-by: Stephen Crawford <steecraw@amazon.com>
@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

Signed-off-by: Stephen Crawford <steecraw@amazon.com>
@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

  • RESULT: UNSTABLE ❕
  • TEST FAILURES:
      1 org.opensearch.cluster.routing.allocation.decider.DiskThresholdDeciderIT.testIndexCreateBlockIsRemovedWhenAnyNodesNotExceedHighWatermarkWithAutoReleaseEnabled

@codecov-commenter
Copy link

codecov-commenter commented Apr 20, 2023

Codecov Report

Merging #7246 (3515609) into main (8386d10) will increase coverage by 0.03%.
The diff coverage is 52.29%.

❗ Current head 3515609 differs from pull request most recent head 0f1bcf8. Consider uploading reports for the commit 0f1bcf8 to get more accurate results

📣 This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more

@@             Coverage Diff              @@
##               main    #7246      +/-   ##
============================================
+ Coverage     70.54%   70.57%   +0.03%     
- Complexity    59387    59450      +63     
============================================
  Files          4859     4873      +14     
  Lines        285370   285542     +172     
  Branches      41134    41151      +17     
============================================
+ Hits         201317   201535     +218     
+ Misses        67428    67413      -15     
+ Partials      16625    16594      -31     
Impacted Files Coverage Δ
...opensearch/identity/shiro/ShiroIdentityPlugin.java 0.00% <0.00%> (ø)
...pensearch/identity/shiro/ShiroSecurityManager.java 0.00% <0.00%> (ø)
...identity/shiro/UnsupportedAuthenticationToken.java 0.00% <0.00%> (ø)
...pensearch/common/settings/FeatureFlagSettings.java 50.00% <ø> (ø)
...opensearch/identity/tokens/RestTokenExtractor.java 0.00% <0.00%> (ø)
.../main/java/org/opensearch/rest/RestController.java 76.06% <4.76%> (-7.04%) ⬇️
...java/org/opensearch/identity/noop/NoopSubject.java 22.22% <22.22%> (ø)
...va/org/opensearch/identity/shiro/ShiroSubject.java 44.44% <44.44%> (ø)
...n/java/org/opensearch/identity/NamedPrincipal.java 54.54% <54.54%> (ø)
server/src/main/java/org/opensearch/node/Node.java 84.44% <57.14%> (-0.27%) ⬇️
... and 10 more

... and 526 files with indirect coverage changes

Signed-off-by: Stephen Crawford <steecraw@amazon.com>
@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

Signed-off-by: Stephen Crawford <steecraw@amazon.com>
Signed-off-by: Stephen Crawford <steecraw@amazon.com>
@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

@dblock
Copy link
Member

dblock commented Apr 20, 2023

Go through the code and get it to a minimum PR. Let me know when this is ready to review, I was tracking the previous one.

@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

identityPlugin = identityPlugins.get(0);
} else {
throw new OpenSearchException(
"Multiple identity plugins are not supported, found: "
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since this PR introduces a default module and this file enforces at most 1 IdentityPlugin is installed at a time there is no way to install an IdentityPlugin with this. The opensearch node fails with this error:

OpenSearchException[Multiple identity plugins are not supported, found: org.opensearch.identity.shiro.ShiroIdentityPlugin,org.opensearch.security.OpenSearchSecurityPlugin]
	at org.opensearch.identity.IdentityService.<init>(IdentityService.java:38)
	at org.opensearch.node.Node.<init>(Node.java:464)
	at org.opensearch.node.Node.<init>(Node.java:377)
	at org.opensearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:242)
	at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242)
	at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404)
	at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:180)
	at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:171)
	at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104)
	at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138)
	at org.opensearch.cli.Command.main(Command.java:101)
	at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:137)
	at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:103)
For complete error details, refer to the log at /Users/cwperx/Desktop/distributions/opensearch-stephen-identity/opensearch-3.0.0-SNAPSHOT/logs/opensearch.log

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @cwperks, great catch. This can be resolved in a couple minor changes--making the identity plugin used configurable or allowing shiro to be turned off--but it may be best to wait until @peternied is available and ask him what he thinks considering much of this was his original work.

Copy link
Member

@peternied peternied Apr 28, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since moving forward the we are not intending on using [1] security plugin build out of core, I think we shouldn't make it harder for identity plugin developers to get started by requiring them to remove/uninstall the identity-shiro plugin. I think we can revisit how this works as we get feedback from other security implementations, until then we should default to use the 'no op' provider.

To support this seems like we might need to move the location from module to plugin. @scrawfor99 I think your question [2] framed this well and we can figure out how to best move forward with a recommendation from the maintainers.

@stephen-crawford
Copy link
Contributor Author

Hi @reta, @dblock, and @saratvemulapalli. I am tagging you because we are trying to move this PR forward but wanted your input. As mentioned by @cwperks, in this comment, the way that this PR is configured will need to be modified so that the IdentityPlugin being used is configurable. Currently, we cannot install any other IdentityPlugins since Shiro will always be installed by virtue of being a module.

Before I change this, I wanted to get the opinion of some core maintainers about whether we could move forward with this PR afterwards. An alternative approach to getting our changes into Core would be to start with a barebones Identity Interface and then merging smaller changes. This may be easier to review, but will not show the same functional demonstration that the Shiro implementation @peternied setup would.

Which direction do you prefer I take?

@reta
Copy link
Collaborator

reta commented May 1, 2023

As mentioned by @cwperks, in this comment, the way that this PR is configured will need to be modified so that the IdentityPlugin being used is configurable.

I think this is a good point but do we want it to be configurable? I mean if in 99.9% use cases the default is enough, the other 0.1% of the users could always uninstall it and replace with the custom plugin. Also, do we always want to bundle it? May be we could just move it to plugins folder to become opt-in plugin?

@stephen-crawford
Copy link
Contributor Author

Hi @reta, I think moving it from modules to a bundled plugin would be an appropriate solution. The reason we are planning on bundling it would be so that users could test the functionality of the Identity Interface and potentially create their own implementations. Basically, we are going a similar route as the SDK's HelloWorld extension where we doubt anyone will choose to use Shiro but this lets us demonstrate how the interface works.

Moving the code from a module to a plugin should allow us to ignore the installation issue and we could look into further configuration changes as needed.

@peternied
Copy link
Member

May be we could just move it to plugins folder to become opt-in plugin?

+1

@github-actions
Copy link
Contributor

github-actions bot commented May 1, 2023

Gradle Check (Jenkins) Run Completed with:

Signed-off-by: Stephen Crawford <steecraw@amazon.com>
logger.info("Identity on so found plugins implementing: " + pluginsService.filterPlugins(IdentityPlugin.class).toString());
identityPlugins.addAll(pluginsService.filterPlugins(IdentityPlugin.class));
}
logger.info("Creating identity service in Node.java with identity plugins: " + identityPlugins.toString());
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oops, @scrawfor99 my apologies, I missed that, probably Node.java is too precise:

Suggested change
logger.info("Creating identity service in Node.java with identity plugins: " + identityPlugins.toString());
logger.info("Creating identity service for node with identity plugins: " + identityPlugins.toString());

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oops, I did not mean to include that print. I will remove it.

Signed-off-by: Stephen Crawford <steecraw@amazon.com>
@github-actions
Copy link
Contributor

github-actions bot commented May 2, 2023

Gradle Check (Jenkins) Run Completed with:

  • RESULT: UNSTABLE ❕
  • TEST FAILURES:
      1 org.opensearch.cluster.routing.allocation.decider.DiskThresholdDeciderIT.testIndexCreateBlockIsRemovedWhenAnyNodesNotExceedHighWatermarkWithAutoReleaseEnabled

@github-actions
Copy link
Contributor

github-actions bot commented May 2, 2023

Gradle Check (Jenkins) Run Completed with:

@peternied peternied merged commit 1151308 into opensearch-project:main May 2, 2023
@stephen-crawford stephen-crawford added the backport 2.x Backport to 2.x branch label May 3, 2023
@opensearch-trigger-bot
Copy link
Contributor

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/backport-2.x
# Create a new branch
git switch --create backport/backport-7246-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 1151308ed4ac22f17eb0167746b062d04f755cf4
# Push it to GitHub
git push --set-upstream origin backport/backport-7246-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-7246-to-2.x.

stephen-crawford added a commit to stephen-crawford/OpenSearch that referenced this pull request May 3, 2023
* Identity and access control for OpenSearch

Adding a new service and plugin interface that provides identity and access control inside of OpenSearch. This is the founding building block, see more high level thoughts here on our recent blog. https://opensearch.org/blog/Introducing-Identity/

The new extension point, IdentityPlugin, is added with IdentityService handling that plugin interface. IdentityService authenticates users and enables access control systems. Adding HTTP basic authentication in the RestController the default NoopIdentityPlugin changes no behavior.

Signed-off-by: Stephen Crawford <steecraw@amazon.com>
Signed-off-by: Peter Nied <petern@amazon.com>
Co-authored-by: Peter Nied <petern@amazon.com>
Co-authored-by: Andriy Redko <drreta@gmail.com>
@@ -395,6 +403,11 @@ private void tryAllHandlers(final RestRequest request, final RestChannel channel
return;
}
} else {
if (FeatureFlags.isEnabled(FeatureFlags.IDENTITY)) {
if (!handleAuthenticateUser(request, channel)) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@peternied @scrawfor99 Is the idea of this change in the RestController to use this to authenticate the request instead of ActionPlugin.getRestHandlerWrapper? If identity is enabled to you envision that the security plugin would make the rest wrapper a noop and the logic would be executed in handleAuthenticateUser instead?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I imagine at some point the Rest Wrapper would be completely removed from the security plugin as there will be tighter interfaces that better support audit logging or www-authentication features.

mch2 pushed a commit that referenced this pull request May 3, 2023
…7246) (#7393)

* [Extensions] Introduce Identity Plugin to Core (#7246)

* Identity and access control for OpenSearch

Adding a new service and plugin interface that provides identity and access control inside of OpenSearch. This is the founding building block, see more high level thoughts here on our recent blog. https://opensearch.org/blog/Introducing-Identity/

The new extension point, IdentityPlugin, is added with IdentityService handling that plugin interface. IdentityService authenticates users and enables access control systems. Adding HTTP basic authentication in the RestController the default NoopIdentityPlugin changes no behavior.

Signed-off-by: Stephen Crawford <steecraw@amazon.com>
Signed-off-by: Peter Nied <petern@amazon.com>
Co-authored-by: Peter Nied <petern@amazon.com>
Co-authored-by: Andriy Redko <drreta@gmail.com>

* swap changelog

Signed-off-by: Stephen Crawford <steecraw@amazon.com>

* Swap plugin name

Signed-off-by: Stephen Crawford <steecraw@amazon.com>

---------

Signed-off-by: Stephen Crawford <steecraw@amazon.com>
Signed-off-by: Peter Nied <petern@amazon.com>
Co-authored-by: Peter Nied <petern@amazon.com>
Co-authored-by: Andriy Redko <drreta@gmail.com>
shiv0408 pushed a commit to Gaurav614/OpenSearch that referenced this pull request Apr 25, 2024
* Identity and access control for OpenSearch

Adding a new service and plugin interface that provides identity and access control inside of OpenSearch. This is the founding building block, see more high level thoughts here on our recent blog. https://opensearch.org/blog/Introducing-Identity/

The new extension point, IdentityPlugin, is added with IdentityService handling that plugin interface. IdentityService authenticates users and enables access control systems. Adding HTTP basic authentication in the RestController the default NoopIdentityPlugin changes no behavior.

Signed-off-by: Stephen Crawford <steecraw@amazon.com>
Signed-off-by: Peter Nied <petern@amazon.com>
Co-authored-by: Peter Nied <petern@amazon.com>
Co-authored-by: Andriy Redko <drreta@gmail.com>
Signed-off-by: Shivansh Arora <hishiv@amazon.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport 2.x Backport to 2.x branch
Projects
Status: Done
Development

Successfully merging this pull request may close these issues.

6 participants