Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Task] Automated Rule deletion fails if some matched targets cannot be connected #1093

Closed
andrewazores opened this issue Oct 4, 2022 · 2 comments · Fixed by #1095
Closed

[Task] Automated Rule deletion fails if some matched targets cannot be connected #1093

andrewazores opened this issue Oct 4, 2022 · 2 comments · Fixed by #1095
Labels
bug Something isn't working high-priority

Comments

@andrewazores
Copy link
Member

To reproduce, using smoketest.sh for consistent setup:

  1. Ensure there are no backend stored JMX credentials for the es.andrewazor.Main sample applications
  2. Create an automated rule definition as follows:
{
  "name": "broken_deletion",
  "description": "",
  "matchExpression": "target.alias == 'es.andrewazor.demo.Main'",
  "eventSpecifier": "template=Continuous,type=TARGET",
  "archivalPeriodSeconds": 0,
  "initialDelaySeconds": 0,
  "preservedArchives": 0,
  "maxAgeSeconds": 0,
  "maxSizeBytes": 0,
  "enabled": true
}
  1. Notice that this rule should apply to all three 9093/9094/9095 es.andrewazor.Main sample applications, The 9093 one does not require JMX credentials, the other two do (and 9095 also requires an SSL cert). Because there are no stored credentials for these targets the connection will fail in the background, so notifications for the rule creation will only appear in the UI for the 9093 target.
  2. Delete the new rule definition. Leave the clean checkbox checked.
    image
INFO: (10.0.2.100:59538): OPTIONS /api/v2/rules/broken_deletion 204 0ms
Oct 04, 2022 3:09:39 PM io.cryostat.core.log.Logger info
INFO: Outgoing WS message: {"meta":{"category":"RuleDeleted","type":{"type":"application","subType":"json"},"serverTime":1664896179},"message":{"name":"broken_deletion","description":"","matchExpression":"target.alias == 'es.andrewazor.demo.Main'","eventSpecifier":"template=Continuous,type=TARGET","archivalPeriodSeconds":0,"initialDelaySeconds":0,"preservedArchives":0,"maxAgeSeconds":0,"maxSizeBytes":0,"enabled":true}}
Hibernate: 
    /* select
        generatedAlias0 
    from
        PluginInfo as generatedAlias0 */ select
            plugininfo0_.id as id1_0_,
            plugininfo0_.callback as callback2_0_,
            plugininfo0_.realm as realm3_0_,
            plugininfo0_.subtree as subtree4_0_ 
        from
            PluginInfo plugininfo0_
Hibernate: 
    /* select
        generatedAlias0 
    from
        StoredCredentials as generatedAlias0 */ select
            storedcred0_.id as id1_1_,
            storedcred0_.matchExpression as matchexp2_1_,
            storedcred0_.password as password3_1_,
            storedcred0_.username as username4_1_ 
        from
            StoredCredentials storedcred0_
Oct 04, 2022 3:09:39 PM io.cryostat.core.log.Logger info
INFO: Creating connection for service:jmx:rmi:///jndi/rmi://cryostat:9095/jmxrmi
Oct 04, 2022 3:09:39 PM io.cryostat.core.log.Logger info
INFO: connection attempt failed.

Oct 04, 2022 3:09:39 PM io.cryostat.core.log.Logger info
INFO: Connection for service:jmx:rmi:///jndi/rmi://cryostat:9095/jmxrmi closed
Oct 04, 2022 3:09:39 PM io.cryostat.core.log.Logger error
SEVERE: Exception thrown
io.cryostat.net.web.http.api.v2.ApiException: Authentication failed! Invalid username or password
	at io.cryostat.net.web.http.api.v2.RuleDeleteHandler.handle(RuleDeleteHandler.java:180)
	at io.cryostat.net.web.http.api.v2.AbstractV2RequestHandler.handle(AbstractV2RequestHandler.java:121)
	at io.cryostat.net.web.http.api.v2.AbstractV2RequestHandler.handle(AbstractV2RequestHandler.java:75)
	at io.vertx.ext.web.impl.RouteState.handleContext(RouteState.java:1212)
	at io.vertx.ext.web.impl.RoutingContextImplBase.iterateNext(RoutingContextImplBase.java:163)
	at io.vertx.ext.web.impl.RoutingContextImpl.next(RoutingContextImpl.java:141)
	at io.vertx.ext.web.handler.impl.CorsHandlerImpl.handle(CorsHandlerImpl.java:227)
	at io.vertx.ext.web.handler.impl.CorsHandlerImpl.handle(CorsHandlerImpl.java:41)
	at io.cryostat.net.web.http.generic.CorsEnablingHandler.handle(CorsEnablingHandler.java:113)
	at io.cryostat.net.web.http.generic.CorsEnablingHandler.handle(CorsEnablingHandler.java:57)
	at io.vertx.ext.web.impl.RouteState.handleContext(RouteState.java:1212)
	at io.vertx.ext.web.impl.RoutingContextImplBase.iterateNext(RoutingContextImplBase.java:163)
	at io.vertx.ext.web.impl.RoutingContextImpl.next(RoutingContextImpl.java:141)
	at io.vertx.ext.web.impl.RouterImpl.handle(RouterImpl.java:67)
	at io.vertx.ext.web.impl.RouterImpl.handle(RouterImpl.java:37)
	at io.cryostat.net.web.WebServer.lambda$start$4(WebServer.java:272)
	at io.cryostat.net.HttpServer$HandlerDelegate.handle(HttpServer.java:168)
	at io.vertx.core.http.impl.Http1xServerRequestHandler.handle(Http1xServerRequestHandler.java:57)
	at io.vertx.core.http.impl.Http1xServerRequestHandler.handle(Http1xServerRequestHandler.java:30)
	at io.vertx.core.impl.EventLoopContext.emit(EventLoopContext.java:50)
	at io.vertx.core.impl.DuplicatedContext.emit(DuplicatedContext.java:168)
	at io.vertx.core.http.impl.Http1xServerConnection.handleMessage(Http1xServerConnection.java:145)
	at io.vertx.core.net.impl.ConnectionBase.read(ConnectionBase.java:156)
	at io.vertx.core.net.impl.VertxHandler.channelRead(VertxHandler.java:153)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
	at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
	at io.netty.handler.codec.http.websocketx.extensions.WebSocketServerExtensionHandler.channelRead(WebSocketServerExtensionHandler.java:99)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
	at io.netty.handler.codec.MessageToMessageCodec.channelRead(MessageToMessageCodec.java:111)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
	at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:327)
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:299)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:280)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:722)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:658)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:584)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496)
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: org.openjdk.jmc.rjmx.ConnectionException caused by java.lang.SecurityException: Authentication failed! Invalid username or password
	at org.openjdk.jmc.rjmx.internal.RJMXConnection.connect(RJMXConnection.java:441)
	at io.cryostat.core.net.JFRConnection.attemptConnect(JFRConnection.java:300)
	at io.cryostat.core.net.JFRConnection.connect(JFRConnection.java:257)
	at io.cryostat.core.net.JFRConnection.getService(JFRConnection.java:120)
	at io.cryostat.net.web.http.api.v2.RuleDeleteHandler.lambda$handle$2(RuleDeleteHandler.java:158)
	at io.cryostat.net.TargetConnectionManager.executeConnectedTask(TargetConnectionManager.java:163)
	at io.cryostat.net.web.http.api.v2.RuleDeleteHandler.handle(RuleDeleteHandler.java:155)
	... 58 more
Caused by: java.lang.SecurityException: Authentication failed! Invalid username or password
	at java.management/com.sun.jmx.remote.security.JMXPluggableAuthenticator.authenticationFailure(JMXPluggableAuthenticator.java:231)
	at java.management/com.sun.jmx.remote.security.JMXPluggableAuthenticator.authenticate(JMXPluggableAuthenticator.java:208)
	at jdk.management.agent/sun.management.jmxremote.ConnectorBootstrap$AccessFileCheckerAuthenticator.authenticate(ConnectorBootstrap.java:233)
	at java.management.rmi/javax.management.remote.rmi.RMIServerImpl.doNewClient(RMIServerImpl.java:231)
	at java.management.rmi/javax.management.remote.rmi.RMIServerImpl.newClient(RMIServerImpl.java:198)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
	at java.rmi/sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:360)
	at java.rmi/sun.rmi.transport.Transport$1.run(Transport.java:200)
	at java.rmi/sun.rmi.transport.Transport$1.run(Transport.java:197)
	at java.base/java.security.AccessController.doPrivileged(AccessController.java:712)
	at java.rmi/sun.rmi.transport.Transport.serviceCall(Transport.java:196)
	at java.rmi/sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:587)
	at java.rmi/sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:828)
	at java.rmi/sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$0(TCPTransport.java:705)
	at java.base/java.security.AccessController.doPrivileged(AccessController.java:399)
	at java.rmi/sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:704)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:833)
	at java.rmi/sun.rmi.transport.StreamRemoteCall.exceptionReceivedFromServer(StreamRemoteCall.java:304)
	at java.rmi/sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:280)
	at java.rmi/sun.rmi.server.UnicastRef.invoke(UnicastRef.java:165)
	at java.management.rmi/javax.management.remote.rmi.RMIServerImpl_Stub.newClient(RMIServerImpl_Stub.java:83)
	at java.management.rmi/javax.management.remote.rmi.RMIConnector.getConnection(RMIConnector.java:2107)
	at java.management.rmi/javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:321)
	at org.openjdk.jmc.rjmx.internal.RJMXConnection.connectJmxConnector(RJMXConnection.java:487)
	at org.openjdk.jmc.rjmx.internal.RJMXConnection.establishConnection(RJMXConnection.java:472)
	at org.openjdk.jmc.rjmx.internal.RJMXConnection.connect(RJMXConnection.java:434)
	... 64 more
Caused by: javax.security.auth.login.FailedLoginException: Invalid username or password
	at java.management/com.sun.jmx.remote.security.FileLoginModule.attemptAuthentication(FileLoginModule.java:458)
	at java.management/com.sun.jmx.remote.security.FileLoginModule.login(FileLoginModule.java:309)
	at java.base/javax.security.auth.login.LoginContext.invoke(LoginContext.java:755)
	at java.base/javax.security.auth.login.LoginContext$4.run(LoginContext.java:679)
	at java.base/javax.security.auth.login.LoginContext$4.run(LoginContext.java:677)
	at java.base/java.security.AccessController.doPrivileged(AccessController.java:712)
	at java.base/javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:677)
	at java.base/javax.security.auth.login.LoginContext.login(LoginContext.java:587)
	at java.management/com.sun.jmx.remote.security.JMXPluggableAuthenticator.authenticate(JMXPluggableAuthenticator.java:195)
	at jdk.management.agent/sun.management.jmxremote.ConnectorBootstrap$AccessFileCheckerAuthenticator.authenticate(ConnectorBootstrap.java:233)
	at java.management.rmi/javax.management.remote.rmi.RMIServerImpl.doNewClient(RMIServerImpl.java:231)
	at java.management.rmi/javax.management.remote.rmi.RMIServerImpl.newClient(RMIServerImpl.java:198)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
	at java.rmi/sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:360)
	at java.rmi/sun.rmi.transport.Transport$1.run(Transport.java:200)
	at java.rmi/sun.rmi.transport.Transport$1.run(Transport.java:197)
	at java.base/java.security.AccessController.doPrivileged(AccessController.java:712)
	at java.rmi/sun.rmi.transport.Transport.serviceCall(Transport.java:196)
	at java.rmi/sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:587)
	at java.rmi/sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:828)
	at java.rmi/sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$0(TCPTransport.java:705)
	at java.base/java.security.AccessController.doPrivileged(AccessController.java:399)
	at java.rmi/sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:704)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	... 1 more

Oct 04, 2022 3:09:39 PM io.cryostat.core.log.Logger info
INFO: Removing cached connection for service:jmx:rmi:///jndi/rmi://cryostat:9095/jmxrmi: EXPLICIT
Hibernate: 
    /* select
        generatedAlias0 
    from
        StoredCredentials as generatedAlias0 */ select
            storedcred0_.id as id1_1_,
            storedcred0_.matchExpression as matchexp2_1_,
            storedcred0_.password as password3_1_,
            storedcred0_.username as username4_1_ 
        from
            StoredCredentials storedcred0_
Oct 04, 2022 3:09:39 PM io.cryostat.core.log.Logger info
INFO: Creating connection for service:jmx:rmi:///jndi/rmi://cryostat:9093/jmxrmi
Hibernate: 
    /* select
        generatedAlias0 
    from
        StoredCredentials as generatedAlias0 */ select
            storedcred0_.id as id1_1_,
            storedcred0_.matchExpression as matchexp2_1_,
            storedcred0_.password as password3_1_,
            storedcred0_.username as username4_1_ 
        from
            StoredCredentials storedcred0_
Oct 04, 2022 3:09:39 PM io.cryostat.core.log.Logger info
INFO: Creating connection for service:jmx:rmi:///jndi/rmi://cryostat:9094/jmxrmi
Oct 04, 2022 3:09:39 PM io.cryostat.core.log.Logger info
INFO: connection attempt failed.

Oct 04, 2022 3:09:39 PM io.cryostat.core.log.Logger info
INFO: Connection for service:jmx:rmi:///jndi/rmi://cryostat:9094/jmxrmi closed
Oct 04, 2022 3:09:39 PM io.cryostat.core.log.Logger error
SEVERE: Exception thrown
io.cryostat.net.web.http.api.v2.ApiException: Authentication failed! Invalid username or password
	at io.cryostat.net.web.http.api.v2.RuleDeleteHandler.handle(RuleDeleteHandler.java:180)
	at io.cryostat.net.web.http.api.v2.AbstractV2RequestHandler.handle(AbstractV2RequestHandler.java:121)
	at io.cryostat.net.web.http.api.v2.AbstractV2RequestHandler.handle(AbstractV2RequestHandler.java:75)
	at io.vertx.ext.web.impl.RouteState.handleContext(RouteState.java:1212)
	at io.vertx.ext.web.impl.RoutingContextImplBase.iterateNext(RoutingContextImplBase.java:163)
	at io.vertx.ext.web.impl.RoutingContextImpl.next(RoutingContextImpl.java:141)
	at io.vertx.ext.web.handler.impl.CorsHandlerImpl.handle(CorsHandlerImpl.java:227)
	at io.vertx.ext.web.handler.impl.CorsHandlerImpl.handle(CorsHandlerImpl.java:41)
	at io.cryostat.net.web.http.generic.CorsEnablingHandler.handle(CorsEnablingHandler.java:113)
	at io.cryostat.net.web.http.generic.CorsEnablingHandler.handle(CorsEnablingHandler.java:57)
	at io.vertx.ext.web.impl.RouteState.handleContext(RouteState.java:1212)
	at io.vertx.ext.web.impl.RoutingContextImplBase.iterateNext(RoutingContextImplBase.java:163)
	at io.vertx.ext.web.impl.RoutingContextImpl.next(RoutingContextImpl.java:141)
	at io.vertx.ext.web.impl.RouterImpl.handle(RouterImpl.java:67)
	at io.vertx.ext.web.impl.RouterImpl.handle(RouterImpl.java:37)
	at io.cryostat.net.web.WebServer.lambda$start$4(WebServer.java:272)
	at io.cryostat.net.HttpServer$HandlerDelegate.handle(HttpServer.java:168)
	at io.vertx.core.http.impl.Http1xServerRequestHandler.handle(Http1xServerRequestHandler.java:57)
	at io.vertx.core.http.impl.Http1xServerRequestHandler.handle(Http1xServerRequestHandler.java:30)
	at io.vertx.core.impl.EventLoopContext.emit(EventLoopContext.java:50)
	at io.vertx.core.impl.DuplicatedContext.emit(DuplicatedContext.java:168)
	at io.vertx.core.http.impl.Http1xServerConnection.handleMessage(Http1xServerConnection.java:145)
	at io.vertx.core.net.impl.ConnectionBase.read(ConnectionBase.java:156)
	at io.vertx.core.net.impl.VertxHandler.channelRead(VertxHandler.java:153)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
	at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
	at io.netty.handler.codec.http.websocketx.extensions.WebSocketServerExtensionHandler.channelRead(WebSocketServerExtensionHandler.java:99)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
	at io.netty.handler.codec.MessageToMessageCodec.channelRead(MessageToMessageCodec.java:111)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
	at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:327)
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:299)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:280)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:722)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:658)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:584)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496)
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: org.openjdk.jmc.rjmx.ConnectionException caused by java.lang.SecurityException: Authentication failed! Invalid username or password
	at org.openjdk.jmc.rjmx.internal.RJMXConnection.connect(RJMXConnection.java:441)
	at io.cryostat.core.net.JFRConnection.attemptConnect(JFRConnection.java:300)
	at io.cryostat.core.net.JFRConnection.connect(JFRConnection.java:257)
	at io.cryostat.core.net.JFRConnection.getService(JFRConnection.java:120)
	at io.cryostat.net.web.http.api.v2.RuleDeleteHandler.lambda$handle$2(RuleDeleteHandler.java:158)
	at io.cryostat.net.TargetConnectionManager.executeConnectedTask(TargetConnectionManager.java:163)
	at io.cryostat.net.web.http.api.v2.RuleDeleteHandler.handle(RuleDeleteHandler.java:155)
	... 58 more
Caused by: java.lang.SecurityException: Authentication failed! Invalid username or password
	at java.management/com.sun.jmx.remote.security.JMXPluggableAuthenticator.authenticationFailure(JMXPluggableAuthenticator.java:231)
	at java.management/com.sun.jmx.remote.security.JMXPluggableAuthenticator.authenticate(JMXPluggableAuthenticator.java:208)
	at jdk.management.agent/sun.management.jmxremote.ConnectorBootstrap$AccessFileCheckerAuthenticator.authenticate(ConnectorBootstrap.java:233)
	at java.management.rmi/javax.management.remote.rmi.RMIServerImpl.doNewClient(RMIServerImpl.java:231)
	at java.management.rmi/javax.management.remote.rmi.RMIServerImpl.newClient(RMIServerImpl.java:198)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
	at java.rmi/sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:360)
	at java.rmi/sun.rmi.transport.Transport$1.run(Transport.java:200)
	at java.rmi/sun.rmi.transport.Transport$1.run(Transport.java:197)
	at java.base/java.security.AccessController.doPrivileged(AccessController.java:712)
	at java.rmi/sun.rmi.transport.Transport.serviceCall(Transport.java:196)
	at java.rmi/sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:587)
	at java.rmi/sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:828)
	at java.rmi/sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$0(TCPTransport.java:705)
	at java.base/java.security.AccessController.doPrivileged(AccessController.java:399)
	at java.rmi/sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:704)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:833)
	at java.rmi/sun.rmi.transport.StreamRemoteCall.exceptionReceivedFromServer(StreamRemoteCall.java:304)
	at java.rmi/sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:280)
	at java.rmi/sun.rmi.server.UnicastRef.invoke(UnicastRef.java:165)
	at java.management.rmi/javax.management.remote.rmi.RMIServerImpl_Stub.newClient(RMIServerImpl_Stub.java:83)
	at java.management.rmi/javax.management.remote.rmi.RMIConnector.getConnection(RMIConnector.java:2107)
	at java.management.rmi/javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:321)
	at org.openjdk.jmc.rjmx.internal.RJMXConnection.connectJmxConnector(RJMXConnection.java:487)
	at org.openjdk.jmc.rjmx.internal.RJMXConnection.establishConnection(RJMXConnection.java:464)
	at org.openjdk.jmc.rjmx.internal.RJMXConnection.connect(RJMXConnection.java:434)
	... 64 more
Caused by: javax.security.auth.login.FailedLoginException: Invalid username or password
	at java.management/com.sun.jmx.remote.security.FileLoginModule.attemptAuthentication(FileLoginModule.java:458)
	at java.management/com.sun.jmx.remote.security.FileLoginModule.login(FileLoginModule.java:309)
	at java.base/javax.security.auth.login.LoginContext.invoke(LoginContext.java:755)
	at java.base/javax.security.auth.login.LoginContext$4.run(LoginContext.java:679)
	at java.base/javax.security.auth.login.LoginContext$4.run(LoginContext.java:677)
	at java.base/java.security.AccessController.doPrivileged(AccessController.java:712)
	at java.base/javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:677)
	at java.base/javax.security.auth.login.LoginContext.login(LoginContext.java:587)
	at java.management/com.sun.jmx.remote.security.JMXPluggableAuthenticator.authenticate(JMXPluggableAuthenticator.java:195)
	at jdk.management.agent/sun.management.jmxremote.ConnectorBootstrap$AccessFileCheckerAuthenticator.authenticate(ConnectorBootstrap.java:233)
	at java.management.rmi/javax.management.remote.rmi.RMIServerImpl.doNewClient(RMIServerImpl.java:231)
	at java.management.rmi/javax.management.remote.rmi.RMIServerImpl.newClient(RMIServerImpl.java:198)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
	at java.rmi/sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:360)
	at java.rmi/sun.rmi.transport.Transport$1.run(Transport.java:200)
	at java.rmi/sun.rmi.transport.Transport$1.run(Transport.java:197)
	at java.base/java.security.AccessController.doPrivileged(AccessController.java:712)
	at java.rmi/sun.rmi.transport.Transport.serviceCall(Transport.java:196)
	at java.rmi/sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:587)
	at java.rmi/sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:828)
	at java.rmi/sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$0(TCPTransport.java:705)
	at java.base/java.security.AccessController.doPrivileged(AccessController.java:399)
	at java.rmi/sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:704)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	... 1 more

Oct 04, 2022 3:09:39 PM io.cryostat.core.log.Logger info
INFO: Removing cached connection for service:jmx:rmi:///jndi/rmi://cryostat:9094/jmxrmi: EXPLICIT
Oct 04, 2022 3:09:39 PM io.cryostat.core.log.Logger info
INFO: (10.0.2.100:59542): DELETE /api/v2/rules/broken_deletion 500 69ms
Oct 04, 2022 3:09:49 PM io.cryostat.core.log.Logger info
INFO: Removing cached connection for service:jmx:rmi:///jndi/rmi://cryostat:9093/jmxrmi: EXPIRED
Oct 04, 2022 3:09:49 PM io.cryostat.core.log.Logger info
INFO: Connection for service:jmx:rmi:///jndi/rmi://cryostat:9093/jmxrmi closed

The request fails because the 9094/9095 targets cannot be connected to attempt the rule cleanup. This should be silently ignored (just logged) and the deletion request should have a 2xx response code.
@andrewazores andrewazores added the bug Something isn't working label Oct 4, 2022
@andrewazores
Copy link
Member Author

Hold on, this might be invalid. It may be specific to https://github.com/cryostatio/cryostat/pull/1083 .

@andrewazores
Copy link
Member Author

This is not specific to #1083, it can be reproduced on current main.

@andrewazores andrewazores mentioned this issue Oct 5, 2022
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working high-priority
Projects
No open projects
2.2.0 Release
Status: Done
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

fix(rules): handle deletion async andrewazores/cryostat
1 participant
@andrewazores