chore(operations): Create feature flags for all components

[ghost]

Motivation

As the number of Vector's components steadily grows, the time cargo check takes to run grows too.

According to my measurements, "warm" runs of cargo check currently take at least 8 seconds and cannot be sped up by utilizing multiple CPU cores. This greatly impedes developers' productivity, as the waiting time for getting feedback from the compiler is long enough to make the whole experience non-interactive. For tests the feedback loop becomes even longer, so that cargo test --lib lua currently takes at least 21 seconds to run.

In order to alleviate this, I propose to put all components behind the corresponding feature flags in Cargo.toml to make it possible to build only Vector's core and a given component during development of this component. By default, as one might expect, all components are enabled, but it is possible to run cargo check/test/run with --no-default-features --features <component> to compile only the selected components.

With this approach cargo check --no-default-features --features transforms-lua takes 3 instead of 8 seconds and cargo test --lib --no-default-features --features --transforms-lua lua takes 5 instead of 21 seconds.

Implementation details

1. The features {sources,transforms,sinks}-* were added to Cargo.toml with the corresponding meta-features depending on all of these features.
2. Dependencies which are not required by Vector's core were made optional and added to the list of dependencies of the corresponding component features. This reduces the time of "cold" builds when some components are disabled.
3. In order to ensure that each component can be built in isolation, a corresponding CI check was added.

Further thoughts

I've noticed that some components depend on others, in particular

• sources-kubernetes depends on sources-file, transforms-json_parser, transforms-regex_parser;
• sources-syslog depends on sources-socket;
• sources-vector depends on sources-socket;
• sinks-humio_logs depends on sinks-splunk_hec;
• sinks-new_relic_logs dependes on sinks-http;
• sinks-sematext_logs depends on sinks-elasticsearc.

I think ideally we might want to factor out the common parts of code into separate crates in lib in order to avoid make components orthogonal to each other. I want to open an issue about it tomorrow.

In addition, I think some users might like the ability to build slim versions of Vector customized to their needs. It might be especially easy to do if rust-lang/cargo#3126 becomes implemented.

Closes #1514.

[lukesteensen]

Nice! Looks like this can make a real difference.

My main reservation is that it adds a bit of an annoying process anytime anyone adds a new component. It's definitely not the end of the world, but I can definitely see myself forgetting to do it. Is this something we're going to try to enforce, or are we ok with it falling behind slightly sometimes?

[ghost]

Is this something we're going to try to enforce, or are we ok with it falling behind slightly sometimes?

I've added a CI check which ensures that Vector built without default features has no available components. In case if we find out that it is too restricting, it can be removed later.

[lukesteensen]

Sounds good! I don't feel super strongly either way, but this seems like a good improvement to developer experience for those who use it.

[bruceg]

I wonder if this is the right way to go. It is certainly the least resistance way of improving the rebuild cycle, but complicates the developer experience by requiring additional pieces of configuration that have to be added and removed between builds (albeit on the command line).

Is there any way to break vector up into submodules (ex core, sinks, sources, transforms, bin) to reduce the rebuild time of the individual bits? Most of the testing is in the libs, so incremental builds and tests could skip the final main link step which sounds like a big win.

@jasonfried of @basecamp talks about feature flags

[ghost]

I wonder if this is the right way to go. It is certainly the least resistance way of improving the rebuild cycle, but complicates the developer experience by requiring additional pieces of configuration that have to be added and removed between builds (albeit on the command line).

Is there any way to break vector up into submodules (ex core, sinks, sources, transforms, bin) to reduce the rebuild time of the individual bits? Most of the testing is in the libs, so incremental builds and tests could skip the final main link step which sounds like a big win.

@bruceg I was thinking about putting each component into a separate library crate too, although I'm not sure how much effort would it take currently. But if we put each component into a separate library crate, I think we would still want to have ability to include/exclude these library crates from the resulting Vector binary. So these feature flags would still be in use.

See also #1941 and #1940.

[MOZGIII]

Extracting components into subcrates should yield even better results. When crate dependencies don’t change it’s not rebuilt, and this will make most of the work that doesn’t include global changes a lot quicker, while when we do actually work on global stuff the build system will make sure everything is rebuilt properly.

[MOZGIII]

Related to #1514

[ghost]

Related to #1514

I think it is even possible to say that this PR closes #1514 🙂

Extracting components into subcrates should yield even better results. When crate dependencies don’t change it’s not rebuilt, and this will make most of the work that doesn’t include global changes a lot quicker, while when we do actually work on global stuff the build system will make sure everything is rebuilt properly.

I agree. But it would be more work and requires some planning beforehand, while this PR solves the immediate problem.

[MOZGIII]

I think it is even possible to say that this PR closes #1514

I'm trying to imagine how I would harvest the benefits from feature flags to improve iteration times. I'll have to disable the features that I don't care about at a particular point in time manually, right? I think this is great, definitely an improvement!

[ghost]

I'm trying to imagine how I would harvest the benefits from feature flags to improve iteration times. I'll have to disable the features that I don't care about at a particular point in time manually, right? I think this is great, definitely an improvement!

I do it like this:

1. Install cargo-watch.
2. To constantly check or test, for example, lua transform on each change run

   cargo watch -s clear -s \
     'cargo check --lib --no-default-features --features transforms-lua'

   or

   cargo watch -s clear -s \
     'cargo test --lib --no-default-features --features transforms-lua transforms::lua'

3. I also have RUSTFLAGS='-Clinker=clang-9 -Clink-arg=-fuse-ld=lld' in my environment variables, which significantly improves the time for rebuilding tests (thanks to @LucioFranco for suggesting this).

As a side note: -s clear is used here instead of cargo-watch's -c option because it has slightly stronger effect, see watchexec/cargo-watch#141.

[MOZGIII]

I see. In my case, I'll have to also make the rust-analyzer aware of the features I'd like to enable, otherwise, it will do cargo-checks on everything. Currently, I'm still a bit annoyed with the time rust-analyzer needs to complete cargo-check, but at least it's bearable, while with rls is was just total nonsense.
This is definitely a step in the right direction, good job. 👍

[binarylogic]

@a-rodin should we document this in the build/compile docs? I could see users wanting this for custom builds, especially embedded devices.

[bruceg]

As a side note: -s clear is used here instead of cargo-watch's -c option because it has slightly stronger effect, see passcod/cargo-watch#141.

This may be terminal dependent. On both gnome-terminal and xfce4-terminal on my system, cargo-watch --clear has the effect you desire.

[lukesteensen]

Extracting components into subcrates should yield even better results.

We should be careful with these kinds of assumptions. While I agree in theory, I know there have been experience reports from other projects where aggressively splitting out crates has not ended up improving compile times to the degree they expected.

[LucioFranco]

We should be careful with these kinds of assumptions. While I agree in theory, I know there have been experience reports from other projects where aggressively splitting out crates has not ended up improving compile times to the degree they expected.

This is correct, we don't have enough code in our main crate to warrant an extraction imo. Linkerd did this and found nothing, now they have like 30 crates that are much harder to manager than a single crate.

[ghost]

@binarylogic

should we document this in the build/compile docs? I could see users wanting this for custom builds, especially embedded devices.
Yes, I've added it to the docs.

@bruceg

This may be terminal dependent. On both gnome-terminal and xfce4-terminal on my system, cargo-watch --clear has the effect you desire.

I agree. The difference I observed was in tmux.