Improve deployment of optional dependencies

[busbey]

We have DB bindings that need different optional dependencies. Currently we rely on users placing jars in a particular location on the filesystem which is error prone (see #232 ).

Get a better story together around this. Needs to account for the fact that different bindings might need conflicting versions of dependencies.

[busbey]

maybe the answer to this is classpath isolation of the drivers? then we could just build all of them as with-dependencies and only load the classpath of the module in use.

[busbey]

Several of the modules rely on jar - with - dependencies. We should clean this up as a part of this ticket, so that we can make it more apparent to an end user what dependencies are on their system. We cut hopefully also down on our binary distribution size by deduplicating.

[nitsanw]

Not sure you want to tackle the deployment model as part of this work, but I would think ideally we'd want a plugin model which allows:

• Separate development and versioning of core/plugins
• A compatibility API for the core(should be the DB API)
• Packaging of vendor specific deployable client (core + plugin)

This will result in faster turnaround for vendor bug fixes/contribution in their own plugin area and reduce pressure on core releases from vendor PRs.

[busbey]

• Separate development and versioning of core/plugins
• Packaging of vendor specific deployable client (core + plugin)

I think having convenience binaries that are just the deployable bits for particular vendors would be great at easing getting started. We get tons of questions that are essentially "I just want to use datastore X, what am I doing wrong with these thousand configuration knobs."

I have some strong reservations about versioning things independently though. It complicates providing a reference point for folks to reproduce results. As opposed to just 'use YCSB version foo', now folks have to track down particular bindings that might come from lord-knows-where. It also makes our testing way more complicated (because we'd need to test X versions of core with Y versions of Z plugins)

• A compatibility API for the core(should be the DB API)

We definitely need a defined API, for workloads and exporters as as well. Defining our API needs to be its own issue. I thought we had one. Please feel free to take a stab at defining one.

Personally, I've found the annotation-based approach of the Hadoop and HBase projects super easy to maintain. Essentially we make an annotation for "this is public" and "this is not public" and then label classes with them. There's even tooling to generate appropriately scoped javadocs that separate out the two.

[busbey]

Status update from this weekend: I got started on updating the distribution but ran into some troubles with the maven assembly plugin. Should still be able to finish by end of week, so the 0.2.0 branch date isn't at risk.

[busbey]

I have generation of a per - datastore binding distribution artifact working. Still getiing them all properly combined into an uber-distro like we currently have.

If it'll make reviewing the final version easier, I can push a WIP to my github branch.