Add support for option_payment_metadata

[t-bast]

Add support for option_payment_metadata as specified in lightning/bolts#912
This feature lets recipients generate "light" invoices without storing them in the local DB until they are paid.

The trick is to simply put the important data (mainly payment_secret and total_amount) in an encrypted blob inside the invoice, and verify its presence and integrity when receiving a payment that contains a payment_metadata field (and insert it in the DB on-the-fly).

This PR only implements support for the sender side, as recipients will only be able to leverage this features once all senders support it.

We do include a payment_metadata in all invoices we generate, which lets us know whether our payers support this feature: this is important data to have before we decide to make it mandatory and actually implement storage-less invoices.

We also take this opportunity to have our first invoice-only feature to implement feature bit filtering, as required by the spec (IN9 flags as specified in bolt 9): see the first commit of the PR.

The commits are somewhat independent and can be reviewed separately.

[joostjager]

So for interop testing, we can only do eclair -> lnd?

[t-bast]

So for interop testing, we can only do eclair -> lnd?

Right now yes, as the description mentions including a payment_metadata in invoices is not implemented yet. But it will when I have time to get back to it.

[joostjager]

Shall I get you a testnet invoice so that you can try to pay it?

[t-bast]

I'm not ready yet for compat tests, I should address my first three todos before that, I'll let you know when I'm ready.

[t-bast]

@joostjager this should now be ready for interop testing 🎉

[joostjager]

Tried, but running into this when opening a channel from lnd:

2021-12-07 13:38:31,151 WARN  fr.acinq.eclair.io.Peer n:028a52ef0c0d4fffe72239d723d3969f779d19d4645b774b9df2d463ee264fee45 c:18b9b4ef528d25374188358d068c3798c843f83c611472c8d999f69743209a6f - ignoring open_channel: option_channel_type was negotiated but channel_type is missing

I thought this thing was solved?

[t-bast]

I thought this thing was solved?

Looks like it's not! Are you sure you're running an up-to-date lnd?
This log means that we both have activated the option_channel_type feature, but lnd didn't send a channel_type in the tlv stream.
The spec clearly says that it must send the tlv.

[joostjager]

I also tried with vanilla v0.14.1-beta, same story.

Repro:

• Eclair initiates connection to Lnd
• Lnd attempts channel open to Eclair

[joostjager]

One finding is that eclair-cli parseinvoice doesn't show the metadata of my test invoice lnbcrt5u1ps67e6rpp53hg4urhja4t8xedl5xmvchru4gnu42ecte49p2ejrl9lqzt5crqqdqqmqgqypqxpq9cqzpgxqyz5vqsp59ywh2qdt07gkypxap72gwgalpnuafqgu7c5w4zpc3pc7p7a4ks4s9q2gqqqqqyssqfa22zca7m58e0xtrv4ny7p6fsqkkwsh2h8an3djl4n7mu3jd48qpvc5cye29e3jp4pu9ny0utfwzyrmlss6a9ahep03hc0xh0c0axncqy8afdn

[t-bast]

One finding is that eclair-cli parseinvoice doesn't show the metadata of my test invoice

Good catch, I need to add that!

[joostjager]

We now have this problem: lightning/bolts#912 (comment)

[codecov-commenter]

Codecov Report

Merging #2063 (bffbec2) into master (0b807d2) will increase coverage by 0.03%.
The diff coverage is 98.68%.

@@            Coverage Diff             @@
##           master    #2063      +/-   ##
==========================================
+ Coverage   87.58%   87.61%   +0.03%     
==========================================
  Files         165      166       +1     
  Lines       12788    12929     +141     
  Branches      563      550      -13     
==========================================
+ Hits        11200    11328     +128     
- Misses       1588     1601      +13     

Impacted Files	Coverage Δ
...re/src/main/scala/fr/acinq/eclair/NodeParams.scala	92.27% <ø> (+0.46%)	⬆️
...rc/main/scala/fr/acinq/eclair/io/Switchboard.scala	85.41% <ø> (ø)
...n/scala/fr/acinq/eclair/json/JsonSerializers.scala	87.87% <ø> (ø)
...cinq/eclair/payment/receive/MultiPartHandler.scala	93.61% <95.00%> (+0.43%)	⬆️
...core/src/main/scala/fr/acinq/eclair/Features.scala	99.14% <100.00%> (+0.04%)	⬆️
...ain/scala/fr/acinq/eclair/payment/Monitoring.scala	98.18% <100.00%> (+0.06%)	⬆️
.../scala/fr/acinq/eclair/payment/PaymentPacket.scala	91.54% <100.00%> (+0.12%)	⬆️
...scala/fr/acinq/eclair/payment/PaymentRequest.scala	92.51% <100.00%> (+0.16%)	⬆️
...cala/fr/acinq/eclair/payment/relay/NodeRelay.scala	95.93% <100.00%> (ø)
...clair/payment/send/MultiPartPaymentLifecycle.scala	94.44% <100.00%> (ø)
... and 18 more

[joostjager]

Spec update to use tlv field 16. If you update this PR, I can do another attempt at interop.

[t-bast]

Spec update to use tlv field 16. If you update this PR, I can do another attempt at interop.

Done.

[joostjager]

The interop test passed. I received payment metadata from eclair:

            "htlcs": [
                {
                    "chan_id": "20469607974371328",
                    "htlc_index": "4",
                    "amt_msat": "1000000",
                    "accept_height": 18636,
                    "accept_time": "1638951794",
                    "resolve_time": "1638951794",
                    "expiry_height": 18676,
                    "state": "SETTLED",
                    "custom_records": {
                    },
                    "mpp_total_amt_msat": "1000000",
                    "amp": null,
                    "metadata": "010203"
                }
            ],

[t-bast]

Thanks for testing this Joost!

[joostjager]

Tested LND->Eclair interop, seems to work too

2022-01-03 21:10:39,999 INFO f.a.e.p.r.PaymentHandler PAY h:1cfd29bfdf71bebc5b1b20ddb56dbc857db45c9f2db58551cc98deb6200a3e07 - received payment for amount=50000 msat totalAmount=50000 msat paymentMetadata=00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000

[t-bast]

Thanks for the tests @joostjager!

[t-bast]

@thomash-acinq I put you as reviewer as this touches invoice stuff, so it may be loosely related to your work on offers (at least it may be useful for you to track this change). I'm interested in your feedback on this, it's ready to review as the spec PR has been merged recently and we've verified interop with lnd.

[thomash-acinq]

Sorry for missing this PR earlier. It would be nice some integration tests, especially on how it interacts with trampoline.

[pm47]

LMostlyGTM, just a few nits or suggestions.

[thomash-acinq]

You can also add it to the PaymentReceived defined above instead of creating a new event just for the metadata.

[t-bast]

I tried, but it is much more invasive (in particular because we store PaymentReceived in the DB)...and I'm not sure in the long run we will want it in PaymentReceived (which happens after the payment has been accepted and fulfilled), but instead before we accept it, so I chose the simplest solution until we know more (but we can definitely change this later if it makes more sense in PaymentReceived!).