Operational concern: Offline plugin installation

[jordansissel]

User-story-ish-thing: As an person deploying Logstash, I may operate within an air-gapped datacenter or otherwise not-internet-connected facility. I would like to install and upgrade Logstash plugins without requiring internet connectivity at installation-time.

---

Let's gather some options for providing plugin installation in the above situation.

Ideally, the solution chosen should be one which is least painful for users and operators of Logstash.

---

Proposals

(This will be updated as discussion progresses on this topic)

• The user could locally mirror rubygems.org and sonatype/maven and use this as the installation source.
• We could periodically publish a full-resolved build artifact that includes all known plugins and dependencies. This artifact could be unpacked on a local machine and used as a source for plugin installation.

[jordansissel]

A meeting lead by @tbragin and some lovely support and solutions-architect folk discussed some of these concerns.

[pickypg]

I think that option 2 is the least painful for users/operators because there's no setup or knowledge of Ruby/dependencies required. Find the plugin that you want, download its bundle, and install it just like with Elasticsearch.

For offline use cases, this avoids a lot of infrastructure to download a single plugin in many cases.

[pickypg]

Ideally building the bundle should be done with every plugin's release rather than periodically. Whatever is done to make this work should be exposed (and strongly encouraged) for any community-based plugin.

[jsvd]

To implement option 2, a rake plugin:package_all task could ask bundler to download all plugins and their dependencies to a folder, using bundler package.

[jsvd]

Also we might want to check if the plugin installation process is http/https only (no git://) and if we properly respect http/https proxies. In my old company we were walled off but had access to a proxy, so we pulled everything through http.

[jordansissel]

@jsvd I believe we support proxies already because our installation method is through the normal Gem APIs which support it. I think @electrical verified this during early testing of the plugin installer, but I may be hallucinating :)

[jsvd]

Yes in theory we should. If Richard hasn't tried yet I'll test it tomorrow.

[electrical]

Proxy support has been tested but not with the file-dependencies parts.
That's something that might need extra work.

[purbon]

If we aim to support kinda offline installation I would most probably take two approaches:

• Create a bundle with everything simplify the installation for offline users.
• I think it would also nice to facilitate a process to create rubygems source where people can install their plugins locally and avoid hitting the external network. This should not be hard to archive as far as I know.

I think that both things can make the live of operators and users easiers.

Regarding Proxy, I think is a different discussion. We aim to solve here an offline installation.

[electrical]

Installations take more then just the ruby gems. We have to think about Jars and remote file dependencies as well.
The current workflow is fully design to work in an online situation.
For limited internet access systems we got proxy support.

Offline installation has never been a consideration at any stage of the creation of the plugin system and all underlying functions.

If we want to support offline installations it will mean that we will have to redesign the whole system to support that.
Not just the plugin manager, but also from a CI and packaging perspective.
And we have to make it decently work in a multi server environment.

Any solution we can think of may work on a single server but may not work in an automated environment...

[colinsurprenant]

At this point, our choice of using gem install-time jar & files dependencies makes it very difficult to enable offline installation. This is probably why most if not all JRuby gems embed jars in the .gem file, so that the gem is self-contained and can be packaged for offline installation.

I think we should revisit our strategy for jar and file dependencies. The only benefit for install time dependencies installation is the published gem size and possibly licensing restrictions for 3rd party files. Having embedded jars and files (when possible) would simplify the plugins model.

I am not very worried about a .gem file size. In the end, the gem and its dependencies have to be installed. It would impact a logstash package that include some plugins. We could offer different packages, like logstash+all-plugins, logstash+base-plugins, logstash+no-plugins.

For licensing restrictions for 3rd party files, I think that if we allow plugins to extend the command line interface, each plugin could support per-plugin command line options, like installing a required 3rd party file using a url or a local file path. Example: bin/plugin geoip install /tmp/GeoLiteCity-2013-01-18.dat

[mkristian]

@colinsurprenant the reason why gem embed their jars is that until recently there is no other way to do it. years ago we tried to wrap jars as gem which just failed badly since rubygems can not resolve maven versions conflicts. now with jar-dependencies things look more promising - though I have to admit that I was not aware about all those problems within logstash. just look at a gem like this:

+--- rubygems:leafy-complete:0.3.1
|    +--- rubygems:leafy-metrics:[0.2.0,0.2.99999] -> 0.2.1
|    |    +--- rubygems:jar-dependencies:[0.1.8,0.1.99999] -> 0.1.11
|    |    +--- io.dropwizard.metrics:metrics-core:3.1.0
|    |    |    \--- org.slf4j:slf4j-api:1.7.7 -> 1.7.10
|    |    \--- io.dropwizard.metrics:metrics-graphite:3.1.0
|    |         +--- io.dropwizard.metrics:metrics-core:3.1.0 (*)
|    |         \--- org.slf4j:slf4j-api:1.7.7 -> 1.7.10
|    +--- rubygems:leafy-health:[0.2.0,0.2.99999] -> 0.2.1
|    |    +--- rubygems:jar-dependencies:[0.1.8,0.1.99999] -> 0.1.11
|    |    +--- io.dropwizard.metrics:metrics-healthchecks:3.1.0
|    |    |    \--- org.slf4j:slf4j-api:1.7.7 -> 1.7.10
|    |    \--- io.dropwizard.metrics:metrics-jvm:3.1.0
|    |         +--- io.dropwizard.metrics:metrics-core:3.1.0 (*)
|    |         \--- org.slf4j:slf4j-api:1.7.7 -> 1.7.10
|    +--- rubygems:leafy-rack:[0.2.0,0.2.99999] -> 0.2.1
|    |    +--- rubygems:jar-dependencies:[0.1.8,0.1.99999] -> 0.1.11
|    |    +--- rubygems:leafy-metrics:[0.2.0,0.2.99999] -> 0.2.1 (*)
|    |    +--- rubygems:leafy-health:[0.2.0,0.2.99999] -> 0.2.1 (*)
|    |    +--- io.dropwizard.metrics:metrics-json:3.1.0
|    |    |    +--- io.dropwizard.metrics:metrics-core:3.1.0 (*)
|    |    |    +--- com.fasterxml.jackson.core:jackson-databind:2.4.2 -> 2.5.1
|    |    |    |    +--- com.fasterxml.jackson.core:jackson-annotations:2.5.0
|    |    |    |    \--- com.fasterxml.jackson.core:jackson-core:2.5.1
|    |    |    \--- org.slf4j:slf4j-api:1.7.7 -> 1.7.10
|    |    \--- io.dropwizard.metrics:metrics-jvm:3.1.0 (*)
|    \--- rubygems:leafy-logger:[0.2.0,0.2.99999] -> 0.2.1
|         +--- rubygems:jar-dependencies:[0.1.8,0.1.99999] -> 0.1.11
|         +--- rubygems:leafy-metrics:[0.2.0,0.2.99999] -> 0.2.1 (*)
|         +--- io.dropwizard:dropwizard-logging:0.8.0-rc5
|         |    +--- io.dropwizard:dropwizard-jackson:0.8.0-rc5
|         |    |    +--- com.google.guava:guava:18.0
|         |    |    +--- io.dropwizard:dropwizard-util:0.8.0-rc5
|         |    |    |    +--- com.fasterxml.jackson.core:jackson-annotations:2.5.0
|         |    |    |    +--- com.google.guava:guava:18.0
|         |    |    |    +--- com.google.code.findbugs:jsr305:3.0.0
|         |    |    |    \--- joda-time:joda-time:2.7
|         |    |    +--- com.fasterxml.jackson.core:jackson-core:2.5.1
|         |    |    +--- com.fasterxml.jackson.core:jackson-databind:2.5.1 (*)
|         |    |    +--- com.fasterxml.jackson.datatype:jackson-datatype-jdk7:2.5.1
|         |    |    +--- com.fasterxml.jackson.datatype:jackson-datatype-guava:2.5.1
|         |    |    +--- com.fasterxml.jackson.module:jackson-module-afterburner:2.5.1
|         |    |    +--- com.fasterxml.jackson.datatype:jackson-datatype-joda:2.5.1
|         |    |    |    \--- com.fasterxml.jackson.core:jackson-annotations:2.5.0
|         |    |    +--- org.slf4j:slf4j-api:1.7.10
|         |    |    \--- ch.qos.logback:logback-classic:1.1.2
|         |    |         +--- ch.qos.logback:logback-core:1.1.2
|         |    |         \--- org.slf4j:slf4j-api:1.7.6 -> 1.7.10
|         |    +--- io.dropwizard:dropwizard-validation:0.8.0-rc5
|         |    |    +--- io.dropwizard:dropwizard-util:0.8.0-rc5 (*)
|         |    |    +--- org.hibernate:hibernate-validator:5.1.3.Final
|         |    |    |    +--- javax.validation:validation-api:1.1.0.Final
|         |    |    |    +--- org.jboss.logging:jboss-logging:3.1.3.GA
|         |    |    |    \--- com.fasterxml:classmate:1.0.0
|         |    |    \--- org.glassfish:javax.el:3.0.0
|         |    +--- io.dropwizard.metrics:metrics-logback:3.1.0
|         |    |    \--- io.dropwizard.metrics:metrics-core:3.1.0 (*)
|         |    +--- org.slf4j:slf4j-api:1.7.10
|         |    +--- org.slf4j:jul-to-slf4j:1.7.10
|         |    |    \--- org.slf4j:slf4j-api:1.7.10
|         |    +--- ch.qos.logback:logback-core:1.1.2
|         |    +--- ch.qos.logback:logback-classic:1.1.2 (*)
|         |    +--- org.slf4j:log4j-over-slf4j:1.7.10
|         |    |    \--- org.slf4j:slf4j-api:1.7.10
|         |    +--- org.slf4j:jcl-over-slf4j:1.7.10
|         |    |    \--- org.slf4j:slf4j-api:1.7.10
|         |    \--- org.eclipse.jetty:jetty-util:9.2.9.v20150224
|         \--- io.dropwizard:dropwizard-configuration:0.8.0-rc5
|              +--- io.dropwizard:dropwizard-jackson:0.8.0-rc5 (*)
|              +--- io.dropwizard:dropwizard-validation:0.8.0-rc5 (*)
|              +--- com.fasterxml.jackson.dataformat:jackson-dataformat-yaml:2.5.1
|              |    +--- com.fasterxml.jackson.core:jackson-core:2.5.1
|              |    +--- com.fasterxml.jackson.core:jackson-databind:2.5.1 (*)
|              |    \--- org.yaml:snakeyaml:1.12
|              \--- org.apache.commons:commons-lang3:3.3.2
\--- rubygems:rake:10.3.+ -> 10.3.2

for example you do not want to "embed" the jars for leafy-logger gem and you want to be able to "pick" a different version of of com.fasterxml.jackson.core:jackson-core or org.yaml:snakeyaml or org.slf4j:slf4j-api, etc in the project using this gem.

but thanx for trying a reporting back to jar-dependencies.

[purbon]

Your comments are totally reasonable to me @mkristian, however in our experience, also mine very recently, we encounter a bunch of errors that make the the user experience of this gem not good. Nowadays I see jar-dependencies as the way to go, but there is still some work to be done to make this idea to work in most of the situations.

For example one of the typical issues I find a lot is:

[ERROR] The build could not read 1 project -> [Help 1]
[ERROR]   
[ERROR]   The project rubygems:jruby-kafka:1.4.0 (/Users/purbon/work/logstash/vendor/bundle/jruby/1.9/specifications/jruby-kafka-1.4.0-java.gemspec) has 3 errors
[ERROR]     Unresolveable build extension: Plugin de.saumya.mojo:gem-extension: or one of its dependencies could not be resolved: Failure to find de.saumya.mojo:gem-extension:jar: in http://repo.maven.apache.org/maven2 was cached in the local repository, resolution will not be reattempted until the update interval of central has elapsed or updates are forced -> [Help 2]
[ERROR]     Unknown packaging: gem
[ERROR]     'build.plugins.plugin.version' for de.saumya.mojo:gem-maven-plugin must be a valid version but is ''.
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/ProjectBuildingException
[ERROR] [Help 2] http://cwiki.apache.org/confluence/display/MAVEN/PluginResolutionException

some other times for example we thought the jars were packed however they were not.

Please feel free to educate me in the way to go to make sure this errors are gone, how can I be of help debugging ?

/cheers

[mkristian]

@purbon today I saw mkristian/jar-dependencies#16 and then I clicking to the upstream issue and then I saw all those numerous problems you guys have. I do understand the point of improving user experience - so I understand the decision taken.

for example the above maven issue I never saw and I do an idea where it comes from. help debugging: tell me how to reproduce it :)

it is probably just tell jar_dependencies to use the latest ruby-maven.

set JARS_DEBUG=true and give me this extensive maven log. just the error is usually not enough.

about the jar being packed or not. if jar-dependencies is runtime dependency then gem install of this gem will not verndor the jars. if it is a depelopment dependency the jars a vendor inside the gem installation. can that be the reason you see inconsistent behaviour ?

[purbon]

Not sure, but worst to try it if this would have make an improvement, thanks for the clarification. I'm happy to provide you later with more intel on that, lets try not to hijack the thread in here :-) I will open issues in your repo.

/cheers

[mkristian]

yes, just open issues on jar-dependencies that is very much appreciated

[ph]

@mkristian Thank you for hoping in this thread, will try to team with you to fix theses issues.

[colinsurprenant]

quick update on this:

• in 1.5 we finally decided to embed the jars in the (few) plugin gems that required them and avoid install-time jar dependencies fetching.
• PR Manage plugins to be installed offline #3404 is proposed for offline installation leveraging the Bundle package command.

[mkristian]

in jruby you can pack a gem inside a jar file:

mkdir gem1
cd gem1
JARS_HOME=./jars GEM_HOME=. GEM_PATH=. gem install some.gem
jar -cvf ../gem1.jar specifications gems jars

this jar file you can just require 'gem1.jar' in jruby and the gem will be found by rubygems. there are probably some more little things to do on 1.7.x or 9k like setting JARS_HOME and/or adding uri:classloader://specifications to Gem::Specification.dirs.

the intention is to have a single archive containing the gem and its dependent gems and jars.

[purbon]

#3404 got merged with enhancements to how to handle offline plugins, feel free to close this issue if no longer valid.