Feature: Module Lifecycle and Experimental/Deprecated Features

[reneme]

This aims at implementing the suggestions in #3110 and #3853. Namely, it introduces a "lifecycle" to Botan's module system. In their info.txt file, individual modules may be flagged as "Experimental" or "Deprecated" ("Stable" is the default), like so:

<defines>
MY_MODULE -> 20240220
</defines>

<module_info>
name -> "My fancy new module"
brief -> "A new feature that isn't fully ready for prime time, yet."
lifecycle -> "Experimental"
</module_info>

Lifecycle

State	Meaning
"Stable"	The (automatic) default. Today, all modules are considered "Stable", essentially. Minor releases may not change the API contract.
"Deprecated"	Such a module behaves exactly the same as "Stable", though it is scheduled for removal in a future major release. While being built by default, ./configure.py will subtly warn about the deprecated modules that go into the build.
"Experimental"	Work in progress. API and functionality might not be stable; implementation quality may be "beta". These modules are not built by default and ./configure.py warns about any experimental modules that go into the build.

Configuration

This adds a few switches to ./configure.py to handle modules (and features) that are not "Stable":

• --enable-experimental-features defines BOTAN_ENABLE_EXPERIMENTAL_FEATURES in the code base and enables all experimental modules.
• --disable-experimental-features (the default)
• --enable-deprecated-features (the default)
• --disable-deprecated-features defines BOTAN_DISABLE_DEPRECATED_FEATURES in the code base and disables all deprecated modules

Additionally, the default build behaviour of experimental and deprecated modules may be explicitly changed using --enable-modules=MODS and --disable-modules=MODS.

Dependants of a disabled experimental/deprecated module won't be built (due to dependency failure), but they won't be transitively marked as experimental/deprecated. Library maintainers have to keep this in mind when changing modules' life cycle state.

Usage

This just introduces the feature but does not mark any modules/features as deprecated or experimental. A good first candidate for "experimental" might be ML-KEM-ipd (#3893). In retrospect, TLS with hybrid PQ/T key exchange (#3609) would have been a good candidate for an experimental module. Soon, Kyber 90s could become the first "deprecated" module: #3807 (comment).

Also, the macros BOTAN_ENABLE_EXPERIMENTAL_FEATURES and BOTAN_DISABLE_DEPRECATED_FEATURES are just introduced but not used, yet. For instance, GCM tags with just 64bits could become a deprecated feature (that is not a full module) at one point.

Documentation

The new switches and the life cycle states are mentioned in the relevant locations of the handbook. Also, the generated Doxygen pages for the modules contain information about the module's life cycle status, if they are not "Stable", like so:

[coveralls]

coverage: 92.082% (+0.005%) from 92.077%
when pulling 5269469 on Rohde-Schwarz:feature/module_lifecycle
into 850b267 on randombit:master.

[randombit]

One thing I’m not completely sure about - and this is a policy issue related to how we’ll use this, rather than the change itself - previously (or since 2.0.0, anyway) there was a clear distinction about when any functionality could be removed, namely in a new major version. Consider we introduce experimental module X, then decide it doesn’t work out. What is the procedure for making it clear this is something that “was” experimental, but is going to be removed soon? We can’t mark it as deprecated - that causes it to start building by default! I guess under this contract we can just yank it without notice. I suppose this can be decided on a case by case basis as we go forward.

Please update sem_ver.rst to make it clear experimental features are not covered by any SemVer guarantee. Otherwise lgtm.

[reneme]

What is the procedure for making it clear this is something that “was” experimental, but is going to be removed soon?

Mhm, fair point. But frankly, another state (e.g. "deprecated experimental") feels somewhat overkill for this use case. At least for adding it right now. In any case, if we'd really need such a lifecycle state for some module in the future, we could always just add it to the build system later on.

Regarding communication: We could make it a policy to open a discussion on GitHub about every experimental feature. This could be used for early feedback and also as a platform for the deprecation notice. To make this discoverable, we might even add another entry to the <module_info> (e.g. feedback_link, or perhaps something more general). Then, if someone builds with --enable-experimental-features we could print a list of all experimental modules with these feedback links in the configure.py-run.

I would consider that future-work, though.

[reneme]

Please update sem_ver.rst to make it clear experimental features are not covered by any SemVer guarantee.

Done. Ready for merge from my side.