Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bug: Bitbucket 5.15.0 error 500x on a few pages #349

Closed
Zauxst opened this issue Nov 26, 2018 · 16 comments · Fixed by #468
Closed

Bug: Bitbucket 5.15.0 error 500x on a few pages #349

Zauxst opened this issue Nov 26, 2018 · 16 comments · Fixed by #468
Assignees
Labels
bug Bugs

Comments

@Zauxst
Copy link

Zauxst commented Nov 26, 2018

Describe the bug
Http Error 500 when accessing Bitbucket User Directories.

To Reproduce
Steps to reproduce the behavior:

  1. Bitbucket 5.15.0 (Didn't test on production which is a version lower)
  2. APM latest version
  3. Go to Bitbucket URI : /plugins/servlet/embedded-crowd/directories/list
  4. See error

Expected behavior
Get the User Directories page without any error

Debug logs
Attach your debug logs.
See the documentation
about how to enable debug logging.

Click to expand
2018-11-26 08:58:18,287 ERROR [http-nio-7990-exec-7]  o.a.c.c.C.[.[.[/].[plugins] Servlet.service() for servlet [plugins] in context with path [] threw exception [Servlet execution threw an exception] with root cause
java.lang.ClassNotFoundException: co.elastic.apm.servlet.ServletApiAdvice not found by org.apache.servicemix.bundles.spring-webmvc [8]
        at org.apache.felix.framework.BundleWiringImpl.findClassOrResourceByDelegation(BundleWiringImpl.java:1532)
        at com.atlassian.applinks.core.rest.context.ContextFilter.doFilter(ContextFilter.java:24)
        at com.atlassian.applinks.core.rest.context.ContextFilter.doFilter(ContextFilter.java:24)
        at com.atlassian.applinks.core.rest.context.ContextFilter.doFilter(ContextFilter.java:24)
        at com.atlassian.applinks.core.rest.context.ContextFilter.doFilter(ContextFilter.java:24)
        at com.atlassian.applinks.core.rest.context.ContextFilter.doFilter(ContextFilter.java:24)
        at com.atlassian.analytics.client.filter.UniversalAnalyticsFilter.doFilter(UniversalAnalyticsFilter.java:92)
        at com.atlassian.analytics.client.filter.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:39)
        at com.atlassian.stash.internal.spring.lifecycle.LifecycleJohnsonServletFilterModuleContainerFilter.doFilter(LifecycleJohnsonServletFilterModuleContainerFilter.java:42)
        at com.opensymphony.sitemesh.webapp.SiteMeshFilter.obtainContent(SiteMeshFilter.java:181)
        at com.opensymphony.sitemesh.webapp.SiteMeshFilter.doFilter(SiteMeshFilter.java:85)
        at com.atlassian.plugin.connect.plugin.auth.scope.ApiScopingFilter.doFilter(ApiScopingFilter.java:81)
        at com.atlassian.stash.internal.spring.lifecycle.LifecycleJohnsonServletFilterModuleContainerFilter.doFilter(LifecycleJohnsonServletFilterModuleContainerFilter.java:42)
        at com.atlassian.stash.internal.spring.security.StashAuthenticationFilter.doFilter(StashAuthenticationFilter.java:85)
        at com.atlassian.stash.internal.web.auth.BeforeLoginPluginAuthenticationFilter.doInsideSpringSecurityChain(BeforeLoginPluginAuthenticationFilter.java:112)
        at com.atlassian.stash.internal.web.auth.BeforeLoginPluginAuthenticationFilter.doFilter(BeforeLoginPluginAuthenticationFilter.java:75)
        at com.atlassian.security.auth.trustedapps.filter.TrustedApplicationsFilter.doFilter(TrustedApplicationsFilter.java:94)
        at com.atlassian.oauth.serviceprovider.internal.servlet.OAuthFilter.doFilter(OAuthFilter.java:67)
        at com.atlassian.stash.internal.spring.lifecycle.LifecycleJohnsonServletFilterModuleContainerFilter.doFilter(LifecycleJohnsonServletFilterModuleContainerFilter.java:42)
        at com.atlassian.plugin.connect.plugin.auth.oauth2.DefaultSalAuthenticationFilter.doFilter(DefaultSalAuthenticationFilter.java:69)
        at com.atlassian.plugin.connect.plugin.auth.user.ThreeLeggedAuthFilter.doFilter(ThreeLeggedAuthFilter.java:109)
        at com.atlassian.jwt.internal.servlet.JwtAuthFilter.doFilter(JwtAuthFilter.java:32)
        at com.atlassian.analytics.client.filter.DefaultAnalyticsFilter.doFilter(DefaultAnalyticsFilter.java:38)
        at com.atlassian.analytics.client.filter.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:39)
        at com.atlassian.bitbucket.internal.scm.git.lfs.servlet.filter.GitLfsLockingFilter.doFilter(GitLfsLockingFilter.java:50)
        at com.atlassian.stash.internal.spring.lifecycle.LifecycleJohnsonServletFilterModuleContainerFilter.doFilter(LifecycleJohnsonServletFilterModuleContainerFilter.java:42)
        at com.atlassian.stash.internal.web.auth.BeforeLoginPluginAuthenticationFilter.doBeforeBeforeLoginFilters(BeforeLoginPluginAuthenticationFilter.java:90)
        at com.atlassian.stash.internal.web.auth.BeforeLoginPluginAuthenticationFilter.doFilter(BeforeLoginPluginAuthenticationFilter.java:73)
        at com.atlassian.stash.internal.request.DefaultRequestManager.doAsRequest(DefaultRequestManager.java:87)
        at com.atlassian.stash.internal.hazelcast.ConfigurableWebFilter.doFilter(ConfigurableWebFilter.java:38)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.lang.Thread.run(Thread.java:748)
        ... 232 frames trimmed

2018-11-26 08:58:18,291 ERROR [http-nio-7990-exec-7] @1BKCLG4x538x38428x0 1lonpf8 10.185.116.60,127.0.0.1 "GET /mvc/error500 HTTP/1.1" c.a.s.i.web.ErrorPageController There was an unhandled exception loading [/plugins/servlet/embedded-crowd/directories/list]
javax.servlet.ServletException: Servlet execution threw an exception
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:238)
        at com.atlassian.applinks.core.rest.context.ContextFilter.doFilter(ContextFilter.java:24)
        at com.atlassian.applinks.core.rest.context.ContextFilter.doFilter(ContextFilter.java:24)
        at com.atlassian.applinks.core.rest.context.ContextFilter.doFilter(ContextFilter.java:24)
        at com.atlassian.applinks.core.rest.context.ContextFilter.doFilter(ContextFilter.java:24)
        at com.atlassian.applinks.core.rest.context.ContextFilter.doFilter(ContextFilter.java:24)
        at com.atlassian.analytics.client.filter.UniversalAnalyticsFilter.doFilter(UniversalAnalyticsFilter.java:92)
        at com.atlassian.analytics.client.filter.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:39)
        at com.atlassian.stash.internal.spring.lifecycle.LifecycleJohnsonServletFilterModuleContainerFilter.doFilter(LifecycleJohnsonServletFilterModuleContainerFilter.java:42)
        at com.opensymphony.sitemesh.webapp.SiteMeshFilter.obtainContent(SiteMeshFilter.java:181)
        at com.opensymphony.sitemesh.webapp.SiteMeshFilter.doFilter(SiteMeshFilter.java:85)
        at com.atlassian.plugin.connect.plugin.auth.scope.ApiScopingFilter.doFilter(ApiScopingFilter.java:81)
        at com.atlassian.stash.internal.spring.lifecycle.LifecycleJohnsonServletFilterModuleContainerFilter.doFilter(LifecycleJohnsonServletFilterModuleContainerFilter.java:42)
        at com.atlassian.stash.internal.spring.security.StashAuthenticationFilter.doFilter(StashAuthenticationFilter.java:85)
        at com.atlassian.stash.internal.web.auth.BeforeLoginPluginAuthenticationFilter.doInsideSpringSecurityChain(BeforeLoginPluginAuthenticationFilter.java:112)
        at com.atlassian.stash.internal.web.auth.BeforeLoginPluginAuthenticationFilter.doFilter(BeforeLoginPluginAuthenticationFilter.java:75)
        at com.atlassian.security.auth.trustedapps.filter.TrustedApplicationsFilter.doFilter(TrustedApplicationsFilter.java:94)
        at com.atlassian.oauth.serviceprovider.internal.servlet.OAuthFilter.doFilter(OAuthFilter.java:67)
        at com.atlassian.stash.internal.spring.lifecycle.LifecycleJohnsonServletFilterModuleContainerFilter.doFilter(LifecycleJohnsonServletFilterModuleContainerFilter.java:42)
        at com.atlassian.plugin.connect.plugin.auth.oauth2.DefaultSalAuthenticationFilter.doFilter(DefaultSalAuthenticationFilter.java:69)
        at com.atlassian.plugin.connect.plugin.auth.user.ThreeLeggedAuthFilter.doFilter(ThreeLeggedAuthFilter.java:109)
        at com.atlassian.jwt.internal.servlet.JwtAuthFilter.doFilter(JwtAuthFilter.java:32)
        at com.atlassian.analytics.client.filter.DefaultAnalyticsFilter.doFilter(DefaultAnalyticsFilter.java:38)
        at com.atlassian.analytics.client.filter.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:39)
        at com.atlassian.bitbucket.internal.scm.git.lfs.servlet.filter.GitLfsLockingFilter.doFilter(GitLfsLockingFilter.java:50)
        at com.atlassian.stash.internal.spring.lifecycle.LifecycleJohnsonServletFilterModuleContainerFilter.doFilter(LifecycleJohnsonServletFilterModuleContainerFilter.java:42)
        at com.atlassian.stash.internal.web.auth.BeforeLoginPluginAuthenticationFilter.doBeforeBeforeLoginFilters(BeforeLoginPluginAuthenticationFilter.java:90)
        at com.atlassian.stash.internal.web.auth.BeforeLoginPluginAuthenticationFilter.doFilter(BeforeLoginPluginAuthenticationFilter.java:73)
        at com.atlassian.stash.internal.request.DefaultRequestManager.doAsRequest(DefaultRequestManager.java:87)
        at com.atlassian.stash.internal.hazelcast.ConfigurableWebFilter.doFilter(ConfigurableWebFilter.java:38)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.lang.Thread.run(Thread.java:748)
        ... 222 frames trimmed
Caused by: java.lang.NoClassDefFoundError: co/elastic/apm/servlet/ServletApiAdvice
        at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:841)
        ... 32 common frames omitted
Caused by: java.lang.ClassNotFoundException: co.elastic.apm.servlet.ServletApiAdvice not found by org.apache.servicemix.bundles.spring-webmvc [8]
        at org.apache.felix.framework.BundleWiringImpl.findClassOrResourceByDelegation(BundleWiringImpl.java:1532)
        ... 32 common frames omitted

The first error repeats itself around 15times.

@felixbarny
Copy link
Member

That one should already be fixed by #345

Could you try the latest snapshot to see if it works for you?

@Zauxst
Copy link
Author

Zauxst commented Nov 27, 2018

I did. First I used the install guide from Kibana. Then I got to github and tried the latest snapshot on the main page.

I got the same error in both cases.

@Zauxst
Copy link
Author

Zauxst commented Dec 20, 2018

Update: With the latest version available 1.2.0 I am not able to start bitbucket
Following this documentation:
https://www.elastic.co/guide/en/apm/agent/java/current/application-server-setup.html#setup-tomcat

JVM_SUPPORT_RECOMMENDED_ARGS=-javaagent:/edited/out/prometheus_jmx_exporter -javaagent:/tmp/elastic-apm-agent-1.2.0.jar -Delastic.apm.service_name=bitbucket-test -Delastic.apm.server_urls=edited-out-elk-apm-server-url -Delastic.apm.application_packages=bitbucket-test

@felixbarny
Copy link
Member

Are you getting the same error like in the beginning? Did it work in the mean time?

@Zauxst
Copy link
Author

Zauxst commented Dec 21, 2018

No it didn't work.

The error is:

A fatal error has occurred
The following problems occurred, which prevents Atlassian Bitbucket from starting:
Required plugin com.atlassian.activeobjects.activeobjects-plugin has failed to start
Required plugin com.atlassian.applinks.applinks-plugin has failed to start
Required plugin com.atlassian.atlassian-failure-cache-plugin has failed to start
Required plugin com.atlassian.auiplugin has failed to start
Required plugin com.atlassian.bitbucket.server.bitbucket-nav-links has failed to start
Required plugin com.atlassian.bitbucket.server.bitbucket-rest has failed to start
Required plugin com.atlassian.bitbucket.server.bitbucket-sal has failed to start
Required plugin com.atlassian.bitbucket.server.bitbucket-velocity-helper has failed to start
Required plugin com.atlassian.crowd.embedded.admin has failed to start
Required plugin com.atlassian.markup.atlassian-markup-renderer-plugin has failed to start
Required plugin com.atlassian.oauth.admin has failed to start
Required plugin com.atlassian.oauth.atlassian-oauth-service-provider-spi has failed to start
Required plugin com.atlassian.oauth.consumer has failed to start
Required plugin com.atlassian.oauth.consumer.sal has failed to start
Required plugin com.atlassian.oauth.serviceprovider has failed to start
Required plugin com.atlassian.oauth.serviceprovider.sal has failed to start
Required plugin com.atlassian.plugins.atlassian-client-resource has failed to start
Required plugin com.atlassian.plugins.atlassian-nav-links-plugin has failed to start
Required plugin com.atlassian.plugins.atlassian-plugins-webresource-plugin has failed to start
Required plugin com.atlassian.plugins.atlassian-plugins-webresource-rest has failed to start
Required plugin com.atlassian.plugins.less-transformer-plugin has failed to start
Required plugin com.atlassian.plugins.rest.atlassian-rest-module has failed to start
Required plugin com.atlassian.soy.soy-template-plugin has failed to start
Required plugin com.atlassian.stash.ssh-plugin has failed to start
Required plugin com.atlassian.templaterenderer.api has failed to start
Required plugin com.atlassian.templaterenderer.atlassian-template-renderer-velocity1.6-plugin has failed to start
Required plugin com.atlassian.troubleshooting.plugin-bitbucket has failed to start
Required plugin com.atlassian.upm.atlassian-universal-plugin-manager-plugin has failed to start

And in the logs:

2018-12-21 08:23:17,567 ERROR [ThreadPoolAsyncTaskExecutor::Thread 55]  o.e.g.b.e.i.d.s.DependencyWaiterApplicationContextExecutor Unable to create application context for [com.atlassian.bitbucket.server.bitbucket-jira-development-integration], unsatisfied dependencies: none
java.lang.NoClassDefFoundError: sun/reflect/ConstructorAccessorImpl
        at sun.misc.Unsafe.defineClass(Native Method)
        at java.security.AccessController.doPrivileged(Native Method)
        at org.eclipse.gemini.blueprint.context.support.TrackingUtil.getService(TrackingUtil.java:137)
        at org.eclipse.gemini.blueprint.context.support.OsgiBundleXmlApplicationContext.lookupNamespaceHandlerResolver(OsgiBundleXmlApplicationContext.java:269)
        at org.eclipse.gemini.blueprint.context.support.OsgiBundleXmlApplicationContext.createNamespaceHandlerResolver(OsgiBundleXmlApplicationContext.java:229)
        at org.eclipse.gemini.blueprint.context.support.OsgiBundleXmlApplicationContext.loadBeanDefinitions(OsgiBundleXmlApplicationContext.java:130)
        at org.springframework.context.support.AbstractRefreshableApplicationContext.refreshBeanFactory(AbstractRefreshableApplicationContext.java:129)
        at org.springframework.context.support.AbstractApplicationContext.obtainFreshBeanFactory(AbstractApplicationContext.java:614)
        at org.eclipse.gemini.blueprint.context.support.AbstractDelegatedExecutionApplicationContext.access$800(AbstractDelegatedExecutionApplicationContext.java:60)
        at org.eclipse.gemini.blueprint.context.support.AbstractDelegatedExecutionApplicationContext$3.run(AbstractDelegatedExecutionApplicationContext.java:242)
        at org.eclipse.gemini.blueprint.util.internal.PrivilegedUtils.executeWithCustomTCCL(PrivilegedUtils.java:85)
        at org.eclipse.gemini.blueprint.context.support.AbstractDelegatedExecutionApplicationContext.startRefresh(AbstractDelegatedExecutionApplicationContext.java:220)
        at org.eclipse.gemini.blueprint.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor.stageOne(DependencyWaiterApplicationContextExecutor.java:224)
        at org.eclipse.gemini.blueprint.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor.refresh(DependencyWaiterApplicationContextExecutor.java:177)
        at org.eclipse.gemini.blueprint.context.support.AbstractDelegatedExecutionApplicationContext.refresh(AbstractDelegatedExecutionApplicationContext.java:157)
        at org.eclipse.gemini.blueprint.extender.internal.activator.LifecycleManager$1.run(LifecycleManager.java:207)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.lang.Thread.run(Thread.java:748)
        ... 10 frames trimmed
Caused by: java.lang.ClassNotFoundException: sun.reflect.ConstructorAccessorImpl not found by org.apache.servicemix.bundles.spring-beans [10]
        at org.apache.felix.framework.BundleWiringImpl.findClassOrResourceByDelegation(BundleWiringImpl.java:1532)
        ... 19 common frames omitted
2018-12-21 08:23:17,583 INFO  [ThreadPoolAsyncTaskExecutor::Thread 58]  o.e.g.b.e.i.s.ExtenderConfiguration Application context successfully refreshed (NonValidatingOsgiBundleXmlApplicationContext(bundle=com.atlassian.bitbucket.server.bitbucket-crowd-spi, config=osgibundle:/META-INF/spring/*.xml))
2018-12-21 08:23:17,584 ERROR [ThreadPoolAsyncTaskExecutor::Thread 53]  c.a.plugin.osgi.factory.OsgiPlugin Unable to start the plugin container for plugin 'com.atlassian.plugins.atlassian-remote-event-common-plugin'

I cannot find anything relevant that might help.
Bitbucket version 5.15.0
APM version: 1.2.0

I have also disabled the prometheus Exporter for this last test to ensure it is not interfering.

@eyalkoren
Copy link
Contributor

eyalkoren commented Dec 23, 2018

Can you please check if any of the configuration options from the org.osgi.framework.* group may block class loading delegation of the sun packages? Did you change any of these manually? If not, try setting org.osgi.framework.bootdelegation=sun.*,com.sun.* and see what happens.

@Zauxst
Copy link
Author

Zauxst commented Dec 31, 2018

@eyalkoren sorry for the late response.
I have the default settings as I haven't made modifications to the packages that come after a standard installation with the Bitbucket Installer.

I am also using the JRE and Springboot packaged by Atlassian Bitbucket.
Spring boot version is 1.2, I believe.

With that in mind I am a bit lost by your request.
Should I pass the argument as described here, to the java VM?
java -Dorg.osgi.framework.bootdelegation=sun.*,com.sun.* ?

@eyalkoren
Copy link
Contributor

Yes, you can pass it this way and see what happens. If another problem arises, please try org.osgi.framework.system.packages.extra instead.
If both don't work, it would be best if you figure out whether there is any other value being set to any of these system properties somewhere in the startup scripts or configuration files. A text search for these properties's names on the installation may find something. This may be important if one setting overrides the other.
Please let me know what the results were.

@Zauxst
Copy link
Author

Zauxst commented Dec 31, 2018

I have passed the argument as suggested. The error is exactly the same.

Furthermore I have searched for the org.osgi.framework keyword, I have found a jar file that might describe the libraries used: org/osgi/framework/ but I do not have any system/ path in this file.

What has changed between version updates is that the Proxy in front of the APM server has been configured to work on HTTPS only, with HTTP returning a xkcd comic for the users that try to get there. But this setting has been tested and it's not the cause of the possible error as we have also reverted back to HTTP only and it still crashed.


I do not believe the error is caused by Bitbucket and I would not comfortably change the system settings of the application when it is not recommended by Atlassian.

The initial version which was bellow 1.0 has worked as intended, with the exception that the specific administration page mentioned didn't function (I do not have any evidence of the version number used at that date).

I'll try again with the next version hopefully the agent will be more stable by then.

@eyalkoren
Copy link
Contributor

OK, thanks for assisting with analysis. I added that to our backlog so hopefully we can schedule some more time to look into this, maybe reproduce.

@ameshach
Copy link

ameshach commented Jan 28, 2019

Received the same error and I have resolved it by adding the below line in "_start-webapp.sh" .

-Datlassian.org.osgi.framework.bootdelegation=META-INF.services,com.yourkit,com.singularity.*,com.jprofiler,com.jprofiler.*,org.apache.xerces,org.apache.xerces.*,org.apache.xalan,org.apache.xalan.*,sun.*,com.sun.jndi.*,com.icl.saxon,com.icl.saxon.*,javax.servlet,javax.servlet.*,com.sun.xml.bind.*

After adding the above line..

JVM_SUPPORT_RECOMMENDED_ARGS="-javaagent:/opt/elastic-apm-agent-1.3.0.jar -Delastic.apm.service_name=test-bitbucket-service -Delastic.apm.application_packages=org.example,org.another.example -Delastic.apm.server_urls=http://192.168.56.105:8200 -Datlassian.org.osgi.framework.bootdelegation=META-INF.services,com.yourkit,com.singularity.*,com.jprofiler,com.jprofiler.*,org.apache.xerces,org.apache.xerces.*,org.apache.xalan,org.apache.xalan.*,sun.*,com.sun.jndi.*,com.icl.saxon,com.icl.saxon.*,javax.servlet,javax.servlet.*,com.sun.xml.bind.*"

Please refer this link for more information.

https://confluence.atlassian.com/jirakb/using-javaagent-with-jira-790793295.html#UsingjavaagentwithJIRA-Resolution

@felixbarny
Copy link
Member

We do add co.elastic.apm.agent.* automatically to atlassian.org.osgi.framework.bootdelegation. But it seems we actually need to add all of these (META-INF.services,com.yourkit,com.singularity.*,com.jprofiler,com.jprofiler.*,org.apache.xerces,org.apache.xerces.*,org.apache.xalan,org.apache.xalan.*,sun.*,com.sun.jndi.*,com.icl.saxon,com.icl.saxon.*,javax.servlet,javax.servlet.*,com.sun.xml.bind.*) to that as well. However, I don't quite understand why this is necessary as we don't override the setting but we just add co.elastic.apm.agent.* to the existing set.

@felixbarny
Copy link
Member

Somewhat related: #436

@Zauxst
Copy link
Author

Zauxst commented Feb 5, 2019

Hey.

When is this version going to release? Or how can I test it right now?

We have upgraded the test instances of Bitbucket to 5.16.0 on which I noticed they have changed something internally on how they work with the Java Agents because now the Prometheus Exporter stopped working as well.

I'll have to dig more into it but for now I want to focus on the Elastic APM agent to get it working.

The issue described now might affect the APM agent as well.

@felixbarny
Copy link
Member

You can try a snapshot from master: https://github.com/elastic/apm-agent-java/blob/master/README.md#snapshots

Let me know how it goes :)

@Zauxst
Copy link
Author

Zauxst commented Feb 5, 2019

I have great news!
Not only now Bitbucket v5.16.0 starts. But it also displays the page that initially gave error 500.
Thanks.

When can we expect a stable release of this version?

@felixbarny felixbarny self-assigned this Feb 7, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Bugs
Projects
None yet
Development

Successfully merging a pull request may close this issue.

6 participants