Increase `stop-domain` command timeout in tests #24812

avpinchuk · 2024-02-14T20:55:01Z

This should decrease frequency of failures of the application-tests.

Signed-off-by: Alexander Pinčuk <alexander.v.pinchuk@gmail.com>

dmatej · 2024-02-14T22:08:30Z

I am quite sure that domain should stop much faster than in 10 seconds if everything is alright, especially with --kill and --force.

dmatej · 2024-02-14T22:16:47Z

To me it seems there is some different issue - the shutdown hook from admin-tests is not executed or is executed too late sometimes, for some reason, and just on Jenkins.

14:14:34  13:14:34.427359    INFO                 main               org.glassfish.main.itest.tools.asadmin.Asadmin.exec exec(timeout=30,000, detached=false, args=[start-domain, --debug])
14:14:35  There is a process already using the admin port 4,848 -- it probably is another instance of a GlassFish server.
14:14:35  Command start-domain failed.

Or maybe these is yet different issue ... I know I wrote this code, but why the hell the change-admin-password command failed?

14:14:33  [INFO] Running org.glassfish.main.test.app.security.multirolemapping.MultiRoleMappingTest
14:14:34  13:14:33.958583    INFO                 main      org.glassfish.main.itest.tools.GlassFishTestEnvironment.<clinit> Using basedir: /home/jenkins/agent/workspace/_test-using-jenkinsfile_PR-24811/appserver/tests/application
14:14:34  13:14:33.962103    INFO                 main      org.glassfish.main.itest.tools.GlassFishTestEnvironment.<clinit> Expected GlassFish directory: /home/jenkins/agent/workspace/_test-using-jenkinsfile_PR-24811/appserver/tests/application/target/glassfish7/glassfish
14:14:34  13:14:33.963955    INFO                 main               org.glassfish.main.itest.tools.asadmin.Asadmin.exec exec(timeout=5,000, detached=false, args=[change-admin-password])
14:14:34  Command change-admin-password failed.
14:14:34  
14:14:34  Authentication failed for user: admin (Usually, this means invalid user name and/or password)
14:14:34  
14:14:34  
14:14:34  Admin password NOT changed.

https://ci.eclipse.org/glassfish/job/glassfish_build-and-test-using-jenkinsfile/job/PR-24811/7/execution/node/37/log/

dmatej · 2024-02-14T22:19:52Z

And the next build was successful:

21:42:19  [INFO] Running org.glassfish.main.test.app.security.multirolemapping.MultiRoleMappingTest
21:42:19  20:42:18.970950    INFO                 main      org.glassfish.main.itest.tools.GlassFishTestEnvironment.<clinit> Using basedir: /home/jenkins/agent/workspace/_test-using-jenkinsfile_PR-24811/appserver/tests/application
21:42:19  20:42:18.974180    INFO                 main      org.glassfish.main.itest.tools.GlassFishTestEnvironment.<clinit> Expected GlassFish directory: /home/jenkins/agent/workspace/_test-using-jenkinsfile_PR-24811/appserver/tests/application/target/glassfish7/glassfish
21:42:19  20:42:18.975770    INFO                 main               org.glassfish.main.itest.tools.asadmin.Asadmin.exec exec(timeout=5,000, detached=false, args=[change-admin-password])
21:42:19  Command change-admin-password executed successfully.
21:42:19  
21:42:19  
21:42:19  
21:42:19  Admin password changed.

I know that asadmin sometimes creates files in user's home directory ... could it be related?
EDIT: Nope, those are mostly related to gfclient and clusters.

avpinchuk · 2024-02-14T22:25:23Z

You're right. This is a temporary solution. But now we have a problem with admin console. I'm working on it.

You can reproduce the problem with these steps:

start domain
touch the admin console
stop domain
start domain and immediately stop domain
check the tail of the server.log

Shortly, admin console try to start when GlassFish already stopped.

avpinchuk · 2024-02-14T22:30:35Z

start-domain, then pause 10 sec, then stop-domain - Waiting finished after 149 ms.
start-domain, then immediate stop-domain - Waiting finished after 5,260 ms.

on my local machine

avpinchuk · 2024-02-14T22:35:45Z

This is a part of the server.log:

[2024-02-15T01:29:36.622664+03:00] [GF 7.0.13-SNAPSHOT] [INFO] [] [jakarta.enterprise.logging.stdout] [tid: _ThreadID=22 _ThreadName=GlassFish Shutdown Hook] [levelValue: 800] [[
  Completed shutdown of GlassFish runtime]]

[2024-02-15T01:29:37.335099+03:00] [GF 7.0.13-SNAPSHOT] [SEVERE] [] [] [tid: _ThreadID=104 _ThreadName=Thread-11] [levelValue: 1000] [[
  NCLS-CORE-00029
MultiException stack 1 of 3
java.lang.IllegalStateException: Service com.sun.enterprise.v3.services.impl.GrizzlyService was started at level 0 but it has a run level of 10.  The full descriptor is SystemDescriptor(
	implementation=com.sun.enterprise.v3.services.impl.GrizzlyService
	contracts={com.sun.enterprise.v3.services.impl.GrizzlyService,org.glassfish.api.container.RequestDispatcher}
	scope=org.glassfish.hk2.runlevel.RunLevel
	qualifiers={}
	descriptorType=CLASS
	descriptorVisibility=NORMAL
	metadata=runLevelValue={10},Bundle-SymbolicName={org.glassfish.main.core.kernel},Bundle-Version={7.0.13.SNAPSHOT}
	rank=50
	loader=OsgiPopulatorPostProcessor.HK2Loader(OSGiModuleImpl:: Bundle = [org.glassfish.main.core.kernel [67]], State = [READY],1338010098)
	proxiable=null
	proxyForSameScope=null
	analysisName=null
	id=888
	locatorId=0
	identityHashCode=1276276154
	reified=true)
	at org.glassfish.hk2.runlevel.internal.AsyncRunLevelContext.validate(AsyncRunLevelContext.java:422)
	at org.glassfish.hk2.runlevel.internal.AsyncRunLevelContext.findOrCreate(AsyncRunLevelContext.java:275)
	at org.glassfish.hk2.runlevel.RunLevelContext.findOrCreate(RunLevelContext.java:65)
	at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2109)
	at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:93)
	at org.jvnet.hk2.internal.ServiceLocatorImpl.getService(ServiceLocatorImpl.java:690)
	at org.jvnet.hk2.internal.ThreeThirtyResolver.resolve(ThreeThirtyResolver.java:55)
	at org.jvnet.hk2.internal.ClazzCreator.resolve(ClazzCreator.java:197)
	at org.jvnet.hk2.internal.ClazzCreator.resolveAllDependencies(ClazzCreator.java:234)
	at org.jvnet.hk2.internal.ClazzCreator.create(ClazzCreator.java:387)
	at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:466)
	at org.jvnet.hk2.internal.SingletonContext$1.compute(SingletonContext.java:59)
	at org.jvnet.hk2.internal.SingletonContext$1.compute(SingletonContext.java:47)
	at org.glassfish.hk2.utilities.cache.Cache$OriginThreadAwareFuture$1.call(Cache.java:74)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
	at org.glassfish.hk2.utilities.cache.Cache$OriginThreadAwareFuture.run(Cache.java:131)
	at org.glassfish.hk2.utilities.cache.Cache.compute(Cache.java:176)
	at org.jvnet.hk2.internal.SingletonContext.findOrCreate(SingletonContext.java:98)
	at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2109)
	at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:93)
	at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:67)
	at org.glassfish.internal.data.EngineInfo.getContainer(EngineInfo.java:62)
	at com.sun.enterprise.v3.server.ApplicationLifecycle.startContainers(ApplicationLifecycle.java:1064)
	at com.sun.enterprise.v3.server.ApplicationLifecycle.setupContainerInfos(ApplicationLifecycle.java:782)
	at com.sun.enterprise.v3.server.ApplicationLifecycle.deploy(ApplicationLifecycle.java:413)
	at com.sun.enterprise.v3.server.ApplicationLoaderService.processApplication(ApplicationLoaderService.java:404)
	at com.sun.enterprise.v3.admin.adapter.InstallerThread.load(InstallerThread.java:211)
	at com.sun.enterprise.v3.admin.adapter.InstallerThread.run(InstallerThread.java:82)
MultiException stack 2 of 3
java.lang.IllegalArgumentException: While attempting to resolve the dependencies of com.sun.enterprise.web.WebContainer errors were found
	at org.jvnet.hk2.internal.ClazzCreator.resolveAllDependencies(ClazzCreator.java:247)
	at org.jvnet.hk2.internal.ClazzCreator.create(ClazzCreator.java:387)
	at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:466)
	at org.jvnet.hk2.internal.SingletonContext$1.compute(SingletonContext.java:59)
	at org.jvnet.hk2.internal.SingletonContext$1.compute(SingletonContext.java:47)
	at org.glassfish.hk2.utilities.cache.Cache$OriginThreadAwareFuture$1.call(Cache.java:74)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
	at org.glassfish.hk2.utilities.cache.Cache$OriginThreadAwareFuture.run(Cache.java:131)
	at org.glassfish.hk2.utilities.cache.Cache.compute(Cache.java:176)
	at org.jvnet.hk2.internal.SingletonContext.findOrCreate(SingletonContext.java:98)
	at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2109)
	at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:93)
	at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:67)
	at org.glassfish.internal.data.EngineInfo.getContainer(EngineInfo.java:62)
	at com.sun.enterprise.v3.server.ApplicationLifecycle.startContainers(ApplicationLifecycle.java:1064)
	at com.sun.enterprise.v3.server.ApplicationLifecycle.setupContainerInfos(ApplicationLifecycle.java:782)
	at com.sun.enterprise.v3.server.ApplicationLifecycle.deploy(ApplicationLifecycle.java:413)
	at com.sun.enterprise.v3.server.ApplicationLoaderService.processApplication(ApplicationLoaderService.java:404)
	at com.sun.enterprise.v3.admin.adapter.InstallerThread.load(InstallerThread.java:211)
	at com.sun.enterprise.v3.admin.adapter.InstallerThread.run(InstallerThread.java:82)
MultiException stack 3 of 3
java.lang.IllegalStateException: Unable to perform operation: resolve on com.sun.enterprise.web.WebContainer
	at org.jvnet.hk2.internal.ClazzCreator.create(ClazzCreator.java:422)
	at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:466)
	at org.jvnet.hk2.internal.SingletonContext$1.compute(SingletonContext.java:59)
	at org.jvnet.hk2.internal.SingletonContext$1.compute(SingletonContext.java:47)
	at org.glassfish.hk2.utilities.cache.Cache$OriginThreadAwareFuture$1.call(Cache.java:74)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
	at org.glassfish.hk2.utilities.cache.Cache$OriginThreadAwareFuture.run(Cache.java:131)
	at org.glassfish.hk2.utilities.cache.Cache.compute(Cache.java:176)
	at org.jvnet.hk2.internal.SingletonContext.findOrCreate(SingletonContext.java:98)
	at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2109)
	at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:93)
	at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:67)
	at org.glassfish.internal.data.EngineInfo.getContainer(EngineInfo.java:62)
	at com.sun.enterprise.v3.server.ApplicationLifecycle.startContainers(ApplicationLifecycle.java:1064)
	at com.sun.enterprise.v3.server.ApplicationLifecycle.setupContainerInfos(ApplicationLifecycle.java:782)
	at com.sun.enterprise.v3.server.ApplicationLifecycle.deploy(ApplicationLifecycle.java:413)
	at com.sun.enterprise.v3.server.ApplicationLoaderService.processApplication(ApplicationLoaderService.java:404)
	at com.sun.enterprise.v3.admin.adapter.InstallerThread.load(InstallerThread.java:211)
	at com.sun.enterprise.v3.admin.adapter.InstallerThread.run(InstallerThread.java:82)
]]

[2024-02-15T01:29:37.336963+03:00] [GF 7.0.13-SNAPSHOT] [SEVERE] [] [jakarta.enterprise.system.core] [tid: _ThreadID=104 _ThreadName=Thread-11] [levelValue: 1000] [[
  Aborting, Failed to start container com.sun.enterprise.web.WebContainer]]

[2024-02-15T01:29:37.337472+03:00] [GF 7.0.13-SNAPSHOT] [SEVERE] [] [jakarta.enterprise.system.core] [tid: _ThreadID=104 _ThreadName=Thread-11] [levelValue: 1000] [[
  Exception while deploying the app [__admingui]]]

[2024-02-15T01:29:37.337774+03:00] [GF 7.0.13-SNAPSHOT] [SEVERE] [NCLS-CORE-00026] [jakarta.enterprise.system.core] [tid: _ThreadID=104 _ThreadName=Thread-11] [levelValue: 1000] [[
  Exception during lifecycle processing
java.lang.Exception: Aborting, Failed to start container com.sun.enterprise.web.WebContainer
	at com.sun.enterprise.v3.server.ApplicationLifecycle.setupContainerInfos(ApplicationLifecycle.java:785)
	at com.sun.enterprise.v3.server.ApplicationLifecycle.deploy(ApplicationLifecycle.java:413)
	at com.sun.enterprise.v3.server.ApplicationLoaderService.processApplication(ApplicationLoaderService.java:404)
	at com.sun.enterprise.v3.admin.adapter.InstallerThread.load(InstallerThread.java:211)
	at com.sun.enterprise.v3.admin.adapter.InstallerThread.run(InstallerThread.java:82)
]]

[2024-02-15T01:29:37.338515+03:00] [GF 7.0.13-SNAPSHOT] [INFO] [] [org.glassfish.web.loader.WebappClassLoader] [tid: _ThreadID=104 _ThreadName=Thread-11] [levelValue: 800] [[
  close(), this:
org.glassfish.web.loader.WebappClassLoader@2050cabf[name=unknown], urls=[
file:/home/avpinchuk/build/glassfish7/glassfish/lib/install/applications/__admingui/WEB-INF/classes/
file:/home/avpinchuk/build/glassfish7/glassfish/lib/install/applications/__admingui/WEB-INF/lib/console-core-7.0.13-SNAPSHOT.jar
file:/home/avpinchuk/build/glassfish7/glassfish/domains/domain1/generated/ejb/__admingui//
file:/home/avpinchuk/build/glassfish7/glassfish/lib/install/applications/__admingui/WEB-INF/extra/woodstock-webui-jsf-suntheme-6.0.1.jar
file:/home/avpinchuk/build/glassfish7/glassfish/lib/install/applications/__admingui/WEB-INF/extra/dojo-ajax-nodemo-1.12.4.jar
file:/home/avpinchuk/build/glassfish7/glassfish/lib/install/applications/__admingui/WEB-INF/extra/woodstock-webui-jsf-6.0.1.jar
file:/home/avpinchuk/build/glassfish7/glassfish/lib/install/applications/__admingui/WEB-INF/extra/json-2.0.jar
file:/home/avpinchuk/build/glassfish7/glassfish/lib/install/applications/__admingui/WEB-INF/extra/prototype-1.7.3.jar
file:/home/avpinchuk/build/glassfish7/glassfish/lib/install/applications/__admingui/WEB-INF/extra/commons-io-2.15.1.jar
][delegate=true, context=unknown, status=RUNNING, antiJARLocking=false, securityManager=false, packageDefinitionSecurityEnabled=false, repositories=RepositoryManager[WEB-INF/classes/], notFound.size=10, pathTimestamps.size=1, resourceEntryCache.size=0]]]

[2024-02-15T01:29:37.360875+03:00] [GF 7.0.13-SNAPSHOT] [SEVERE] [NCLS-CORE-00041] [jakarta.enterprise.system.core] [tid: _ThreadID=104 _ThreadName=Thread-11] [levelValue: 1000] [[
  Application deployment failed: Exception while deploying the app [__admingui]]]

dmatej · 2024-02-14T22:35:48Z

Wow, that is really interesting and might be the cause ... hmmm

dmatej@dmatej-P7740:/app/appservers$ asadmin start-domain && asadmin stop-domain
Waiting for domain1 to start ...
Waiting finished after 2 597 ms.
Successfully started the domain : domain1
domain  Location: /media/data-samsung/data/dmatej/work/app/appservers/glassfish7/glassfish/domains/domain1
Log File: /media/data-samsung/data/dmatej/work/app/appservers/glassfish7/glassfish/domains/domain1/logs/server.log
Admin Port: 4 848
Command start-domain executed successfully.
Waiting for the domain to stop .....
Waiting finished after 4 090 ms.
Command stop-domain executed successfully.

dmatej@dmatej-P7740:/app/appservers$ asadmin start-domain
Waiting for domain1 to start ...
Waiting finished after 2 453 ms.
Successfully started the domain : domain1
domain  Location: /media/data-samsung/data/dmatej/work/app/appservers/glassfish7/glassfish/domains/domain1
Log File: /media/data-samsung/data/dmatej/work/app/appservers/glassfish7/glassfish/domains/domain1/logs/server.log
Admin Port: 4 848
Command start-domain executed successfully.

dmatej@dmatej-P7740:/app/appservers$ asadmin stop-domain
Waiting for the domain to stop .
Waiting finished after 147 ms.
Command stop-domain executed successfully.
dmatej@dmatej-P7740:/app/appservers$

dmatej

Ok, lets do that. I see this as good news, at least now it is not a mystery any more :-)

avpinchuk added the build and test improvement label Feb 14, 2024

avpinchuk added this to the 7.0.13 milestone Feb 14, 2024

avpinchuk self-assigned this Feb 14, 2024

Increase stop-domain command timeout in tests

9dde62a

Signed-off-by: Alexander Pinčuk <alexander.v.pinchuk@gmail.com>

avpinchuk force-pushed the increase-stop-timeout branch from 65f71ef to 9dde62a Compare February 14, 2024 20:57

avpinchuk requested review from dmatej, arjantijms, pzygielo and hs536 February 14, 2024 21:21

pzygielo approved these changes Feb 14, 2024

View reviewed changes

dmatej approved these changes Feb 14, 2024

View reviewed changes

arjantijms approved these changes Feb 15, 2024

View reviewed changes

arjantijms merged commit ff7388f into eclipse-ee4j:master Feb 15, 2024
2 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Increase `stop-domain` command timeout in tests #24812

Increase `stop-domain` command timeout in tests #24812

avpinchuk commented Feb 14, 2024

dmatej commented Feb 14, 2024

dmatej commented Feb 14, 2024

dmatej commented Feb 14, 2024 •

edited

Loading

avpinchuk commented Feb 14, 2024

avpinchuk commented Feb 14, 2024

avpinchuk commented Feb 14, 2024

dmatej commented Feb 14, 2024

dmatej left a comment •

edited

Loading

Increase stop-domain command timeout in tests #24812

Increase stop-domain command timeout in tests #24812

Conversation

avpinchuk commented Feb 14, 2024

dmatej commented Feb 14, 2024

dmatej commented Feb 14, 2024

dmatej commented Feb 14, 2024 • edited Loading

avpinchuk commented Feb 14, 2024

avpinchuk commented Feb 14, 2024

avpinchuk commented Feb 14, 2024

dmatej commented Feb 14, 2024

dmatej left a comment • edited Loading

Choose a reason for hiding this comment

Increase `stop-domain` command timeout in tests #24812

Increase `stop-domain` command timeout in tests #24812

dmatej commented Feb 14, 2024 •

edited

Loading

dmatej left a comment •

edited

Loading