[RFC] Expose public Event Stream varlink interface

[nyarly]

Context

As follow on to #320, stabilize and publish the interface for event notifications, so that other programs can consume them.

Proposal

Create a separate Varlink interface (called com.target.lorri.public).

Migrate the existing Event message and Monitor method from the com.target.lorri interface. Remove dependencies on other members of the interface to keep the interface small. For instance, nix_shell fields become string instead of NixShell.

Have the daemon continue to serve the Monitor method. Reduce ops/stream_events.rs to a simple pass through, reserializing events from the server. lorri stream_events exists in order to make events available on stdout. The message format and the method on the Lorri daemon socket is part of the Lorri public interface.

Credit is due to @curiousleo for the suggestion to use Varlink to describe the event messages.

[nyarly]

cc @Profpatsch

[curiousleo]

Create a separate Varlink interface (called com.target.lorri.public).

My suggestion would be to move the current stuff into com.target.lorri.internal and use com.target.lorri for the public interface.

Migrate the existing Event message and Monitor method from the com.target.lorri interface. Remove dependencies on other members of the interface to keep the interface small. For instance, nix_shell fields become string instead of NixShell.

👍

Have the daemon continue to serve the Monitor method. Reduce ops/stream_events.rs to a simple pass through, reserializing events from the server. lorri stream_events exists in order to make events available on stdout. The message format and the method on the Lorri daemon socket is part of the Lorri public interface.

This seems worth a re-think. Adding a public Varlink interface is a pretty big deal in itself and opens up tooling opportunities via language bindings as well as from Bash through the command line tool.

As such, I would

1. Rename this RFC to "Expose public API via Varlink".
2. Consider lorri stream_events separately: perhaps a good docs page about how to use Varlink from Bash is sufficient?

[nyarly]

A private internal and a public apex accepted and agreed.

A consequence is that com.target.lorri becomes a public interface, albeit with a much reduced surface. It's likely better to have a less awkward name for that interface going forward than to reduce the amount of change here and now.

Regardless, this change merits a minor version release.

[nyarly]

This seems worth a re-think. Adding a public Varlink interface is a pretty big deal in itself and opens up tooling opportunities via language bindings as well as from Bash through the command line tool.

The tooling opportunities were exactly what made this approach appeal to me. As an integrator, I always appreciate options.

Consider lorri stream_events separately: perhaps a good docs page about how to use Varlink from Bash is sufficient?

Perhaps? The stream_events command is already nearly trivial to understand and maintain. This change would simplify it further. The logic about snapshots vs. streams isn't prohibitive to implement outside of lorri, but it reduces friction to have it handled.

I'd propose, as a first step, to make this interface change, and update the still internal__stream_events and review from there.

[curiousleo]

I'd propose, as a first step, to make this interface change, and update the still internal__stream_events and review from there.

Sounds good to me. Thanks for addressing my comments!

[Profpatsch]

The tooling opportunities were exactly what made this approach appeal to me. As an integrator, I always appreciate options.

What does exposing the interface give us that a plain stream of json values doesn’t? i.e. do you have use-cases in mind that the stream_events output can’t satisfy?

I’m very skeptical about making varlink part of our external interface, because universal is anything but.

[nyarly]

@curiousleo makes the point that Varlink ships a command line tool that could be used to consume the stream, which I think would be mostly a personal preference.

However, a concrete use case might be a nicer libnotify gateway. libnotify allows for notifications to be updated (e.g. you could have a "building" notification that lingered until the build was complete, that then changed to "finished" and expired or "failed!" with a button to open the build log). If Varlink remains internal, that would have to be provided by Lorri, and the problem of adapting libnotify vs. MacOS notifications vs. can Windows display notifications? is teneable on its own, but really out of Lorri's scope. If we expose a limited Varlink interface, other programs can be written to manage that, and Varlink can generate their interface code.

More high-flown: I think it's incoherent and unaesthetic to use Varlink to describe a message outside of an interface (i.e. which includes at least one RPC method). As a user of Lorri, if I did want to consume the output of stream_events, having the documentation advise me to learn Varlink to understand how the messages would be formatted, but then caution that I should ignore Varlink as a message transport would strike me as... perverse.

It's arguable that it makes more sense to use Rust structs to serialize the Events out of stream_events (a la #346) and then document the JSON format with simple examples than it does to use Varlink to describe the message format but not a method to receive them.

[curiousleo]

What does exposing the interface give us that a plain stream of json values doesn’t?

Cross-language, cross-platform, type safe (as far as the language in question allows) APIs: https://varlink.org/Language-Bindings