Conversions for JSON DTO

[nyarly]

First task of issue #324

Overview

Creates a dedicated DTO format for JSON, so we can use that format as part of the public API.

Checklist

• Updated the documentation (code documentation, command help, ...)
• Tested the change (unit or integration tests)
• Amended the changelog in release.nix (see release.nix for instructions)

[curiousleo]

Hi @nyarly, thanks for getting back to this!

In #183 (comment), I wrote

We can use Varlink type definitions to describe the output format. Describing a JSON schema is something Varlink is pretty good at, in my opinion. People who want to build tooling for lorri using event_stream may find it easier to read the Varlink schema than Rust structs and enums. Finally, changes to this definition would be very obvious, this would pretty much rule out making output format changes by accident.

You went for Rust structs instead of Varlink definitions for the public API definition. I'd be curious about your perspective on the pros and cons of a Varlink vs. Rust representation of the public API.

[nyarly]

@curiousleo I think I missed that comment at the time. Let me look into using Varlink for this.

[nyarly]

One preliminary concern: is there a sensible way to make clear that there's the JSON Varlink (part of the public interface) and the internal Varlink (which is private and changeable)?

[curiousleo]

One preliminary concern: is there a sensible way to make clear that there's the JSON Varlink (part of the public interface) and the internal Varlink (which is private and changeable)?

There should be a new, separate Varlink file for the public interface. The existing Varlink file with the private interface should probably get a big banner at the top saying "this is a private interface; don't use it; may change at any time without notice".

[nyarly]

So, two varlink interfaces. New one called com.target.lorri.public (or .eventstream ?) I'd be tempted, in fact, to rename the current interface to com.target.lorri.private and use the apex interface name for public access.

I'm looking today at how varlink.rs provides serialization without wire protocol...

[nyarly]

...oh, the generated message structs are #[derive(Serialize)] Okay, this seems like it'll be straightforward to implement. The varlink messages just replace the handwritten structs, with the TryInto implementations applying to the message structs instead.

[nyarly]

Okay, so here's a line of thought @curiousleo @Profpatsch - I started working through a varlink Events message for serialization to JSON. It'll be straightforward to replace the Rust structs with that message, I think.

But, wouldn't it make sense to fully pull the Event interface out of the internal varlink definition and have the daemon use the same messaging? If we're exposing the message via varlink, it would be surprising (I would be surprised, at least) not to make direct varlink integration possible, and for that you need a method and a place to make the requests.

I think I'd be happy with stream_events being a very thin layer that just outputs the messages to stdout for pipeline construction since that was always the design. Having it simply reserialize the messages it receives would further simplify the idea.

But, it would fix the Event message in the interface... I'm not sure that's terrible, since it's effectively fixed already. No significant change can be useful without exposing it in the output, anyway.

In this case, I'd also argue against changing the structure of the message, since the "type safety" argument isn't as strong when the message is meant to be consumed by jq parsers.

[curiousleo]

But, wouldn't it make sense to fully pull the Event interface out of the internal varlink definition and have the daemon use the same messaging? If we're exposing the message via varlink, it would be surprising (I would be surprised, at least) not to make direct varlink integration possible, and for that you need a method and a place to make the requests.

I am not 100% sure what you're proposing concretely here. What I want to make sure is that (1) the public interface is well-defined, ideally as Varlink in a separate file; (2) no internal declarations (Varlink or Rust) leak into the public interface.

In this case, I'd also argue against changing the structure of the message, since the "type safety" argument isn't as strong when the message is meant to be consumed by jq parsers.

I don't think this argument is very strong. Every JSON interface has a "type" of some sort - when you call a JSON API, there are usually fields that you expect in the answer. The simpler the answer to the question "which fields can I expect?", the better. The proposed structure in #183 (comment) makes this answer very simple, which is useful no matter where the output is used.

[Profpatsch]

As far as I can see, all of this is included in the current lorri release.

We haven’t made the json format stable yet, but we could in the future.

[Profpatsch]

For reference: current lorri release: https://github.com/nix-community/lorri/releases/tag/1.4.0