Encountering AmountBelowMinimum routing error in pathfinding is not gracefully handled

[mrfelton]

Background

When trying to pay an invoice to a new Phoenix wallet user that has no existing channels or inbound liquidity the ACINQ node will attempt to do on demand channel creation (similar to what the Breez folks are doing). If the invoice value is above 1000 sats then things work as expected. If the invoice value is below 1000 sats then the Phoenix interceptor node will return the error AmountBelowMinimum as they do not want to be opening channels to their users for such small payments.

When this happens, it appears that lnd’s mission control system marks the node pair as bad. This causes all subsequent attempts to pay to Phoenix to fail - even if they are over the 1000 sat threshold detailed in the amount_below_minimum error response.

When pathfinding, LND should handle AmountBelowMinimum errors by preventing payments through the problem node that are below the minimum amount as learned from the detail of the error response. But it should not affect payments above the threshold.

Your environment

• lnd v0.11.1-beta

Steps to reproduce

1. Attempt to pay an invoice through a path for which the penultimate node will return AmountBelowMinimum error.
2. Attempt to pay an invoice through a path that is above the limited learned from the previous attempt.

example:

1. reset mission control
2. try to pay invoice with low value to a Phoenix wallet that has no inbound channels, eg lnbc100n1p0m6qlrpp5xqrjmnyrdjxv254m0pl6037yusn5h39cdnxq7vm2p04kv252ttrsdqqxqyjw5q9qtzqqqqqq9qsqsp5vqpp6na0urpetme4he4529n8gpdrladyzhvz92scx6a0natdya6srzjqwryaup9lh50kkranzgcdnn2fgvx390wgj5jd07rwr3vxeje0glcllmm6h4a6ppvgsqqqqlgqqqqqeqqjq88ttyankaq99yzj3yg0re244ewp9fwy6pkvmwnp3pk2rwq8872x45923pyr33shvylfpxf596fmxptu4a3fq4ywc3yg2ac23az49znqp6sz5n3
3. This fails with the AmountBelowMinimum error from the ACINQ interceptor node - see full response here: https://paste.ubuntu.com/p/HkX264Tw8r/
4. now check mission control data which now looks like this - seems that the ACINQ interceptor node (03864ef025fde8fb587d989186ce6a4a186895ee44a926bfc370e2c366597a3f8f) has been blacklisted as a result:

From: 03b428ba4b48b524f1fa929203ddc2f0971c2077c2b89bb5b22fd83ed82ac2f7e1
To: 03864ef025fde8fb587d989186ce6a4a186895ee44a926bfc370e2c366597a3f8f
History: {
  "FailTime": 1606224878,
  "FailAmtSat": 0,
  "FailAmtMsat": 0,
  "SuccessTime": 0,
  "SuccessAmtSat": 0,
  "SuccessAmtMsat": 0
}
From: 03864ef025fde8fb587d989186ce6a4a186895ee44a926bfc370e2c366597a3f8f
To: 03b428ba4b48b524f1fa929203ddc2f0971c2077c2b89bb5b22fd83ed82ac2f7e1
History: {
  "FailTime": 1606224878,
  "FailAmtSat": 0,
  "FailAmtMsat": 0,
  "SuccessTime": 0,
  "SuccessAmtSat": 0,
  "SuccessAmtMsat": 0
}
From: 03864ef025fde8fb587d989186ce6a4a186895ee44a926bfc370e2c366597a3f8f
To: 02b931d20f2accec81af1ec26e608f4cad3f2359976b1cb31cf77bd5ebdd042c44
History: {
  "FailTime": 1606224878,
  "FailAmtSat": 0,
  "FailAmtMsat": 0,
  "SuccessTime": 0,
  "SuccessAmtSat": 0,
  "SuccessAmtMsat": 0
}
From: 02b931d20f2accec81af1ec26e608f4cad3f2359976b1cb31cf77bd5ebdd042c44
To: 03864ef025fde8fb587d989186ce6a4a186895ee44a926bfc370e2c366597a3f8f
History: {
  "FailTime": 1606224878,
  "FailAmtSat": 0,
  "FailAmtMsat": 0,
  "SuccessTime": 0,
  "SuccessAmtSat": 0,
  "SuccessAmtMsat": 0
}

5. Now, try to pay a large invoice to the same node lnbc100100n1p0m6p99pp5dtkt5e8aj0n5y442u6f458yxmpcanjmnc9gltj6uer36jjmsag7qdqqxqyjw5q9qtzqqqqqq9qsqsp5a4jzlhayvs4leal0nepkfp0ccc6mflfqg2vpz76fu2enkfz54v3qrzjqwryaup9lh50kkranzgcdnn2fgvx390wgj5jd07rwr3vxeje0glcllmm6h4a6ppvgsqqqqlgqqqqqeqqjq88ddtkt8z9h36nw5jy3v07myvvpmpac8gt6uwc7yryrpnhets7ghfzpmusm37v5gyvzdekkwhf88x7xfelzanfdr4n6gw9atalvj3wgqhg5tga
6. This fails immediately with FailureReasonNoRoute error:

{
  "PaymentHash":"76e1c02ec66f593b1d004bb64f84a1b78c643eb520c78ff459dc7523131afc83",
  "Value":10010,
  "CreationDate":1606224947,
  "Fee":0,
  "PaymentPreimage":"0000000000000000000000000000000000000000000000000000000000000000",
  "ValueSat":10010,
  "ValueMsat":10010000,
  "PaymentRequest":"",
  "Status":"Failed",
  "FeeSat":0,
  "FeeMsat":0,
  "CreationTimeNs":1606224947170531842,
  "Htlcs":[
  ],
  "PaymentIndex":25209,
  "FailureReason":"FailureReasonNoRoute"
}

Mission control remains unchanged.

So the net result is that if a single payment attempts fails with the AmountBelowMinimum error then this causes all subsequent attempts to fail, even if they are over the minimum value.

Problem seems to stem from code here:

lnd/routing/result_interpretation.go

Lines 343 to 364 in 1354a46

	// When an HTLC parameter is incorrect, the node sending the error may
	// be doing something wrong. But it could also be that its predecessor
	// is intentionally modifying the htlc parameters that we instructed it
	// via the hop payload. Therefore we penalize the incoming node pair. A
	// third cause of this error may be that we have an out of date channel
	// update. This is handled by the second chance logic up in mission
	// control.
	case *lnwire.FailAmountBelowMinimum,
	*lnwire.FailFeeInsufficient,
	*lnwire.FailIncorrectCltvExpiry,
	*lnwire.FailChannelDisabled:
	// Set the node pair for which a channel update may be out of
	// date. The second chance logic uses the policyFailure field.
	i.policyFailure = &DirectedNodePair{
	From: route.Hops[errorSourceIdx-1].PubKeyBytes,
	To: route.Hops[errorSourceIdx].PubKeyBytes,
	}
	// We report incoming channel. If a second pair is granted in
	// mission control, this report is ignored.
	reportIncoming()

See related discussion in slack here https://lightningcommunity.slack.com/archives/C6BDA6DGE/p1606225336336800

One suggestion was that htlc_minimum_msat should be integrated into mission control, similar to how it handles temporary channel failures due to lack of funds.

Expected behaviour

First payment attempt should fail since it's below the minimum. The second payment attempt should succeed since it's above the threshold.

Actual behaviour

Both payments fail. Essentially, a single case of AmountBelowMinimum error causes all future payments through the node to fail.

[Roasbeef]

If the invoice value is above 1000 sats then things work as expected. If the invoice value is below 1000 sats then the Phoenix interceptor node will return the error AmountBelowMinimum as they do not want to be opening channels to their users for such small payments.

Before things are resolved on our end (to factor in this new usage of the error code), Phoenix could simply start to advertise the correct min_htlc value in their channel update. lnd would simply then not attempt to route a payment below that value, which is the entire point of having the value be advertised on the channel graph level. Them advertising a value that's actually higher than their link level value is erroneous as this error should only happen if the sender has an out of date channel update for the given channel.

[Roasbeef]

Also one other detail here is that we'll actually try again after the initial failure, in order to give the node a chance and give us an up to date channel update for the channel. In this case since the channel doesn't actually exist (?) I guess they send the default set of channel update params which causes us to try again and fail, only then starting to blacklist the edge.

[mrfelton]

Right - the channel doesn't exist at this point (they intend to create the channel on demand) - The only thing that exists about the channel is a routing hint that's embedded in the invoice that is being paid, which looks like this:

{
  "type":"r",
  "length":82,
  "description":"routing_information",
  "value":{
    "public_key":"03864ef025fde8fb587d989186ce6a4a186895ee44a926bfc370e2c366597a3f8f",
    "short_channel_id":"ff7bd5ebdd042c44",
    "fee_base_msat":1000,
    "fee_proportional_millionths":100,
    "cltv_expiry_delta":144
  }
}

So no detail about min_htlc there.

[t-bast]

Before things are resolved on our end (to factor in this new usage of the error code), Phoenix could simply start to advertise the correct min_htlc value in their channel update.

That wouldn't be enough, because it seems like there's actually another bug where lnd doesn't take into account channel_updates returned in onion errors for private channels (taken from an invoice's routing hint): see #4957

[t-bast]

So no detail about min_htlc there.

That's because the invoice doesn't have a field for that information, that's why we have to rely on AmountBelowMinimum.
We're updating Phoenix to encourage users to set an amount in the invoice in such cases, which can work around that problem.

[joostjager]

Update routing hints upon failure: #2151

[Roasbeef]

@t-bast ah you're totally right re the bug related to private chans and chan updates! We'll take a look at re-prioritizing that for our next major release.

[t-bast]

Awesome thanks!

[Roasbeef]

This should be fixed by #5218

[Roasbeef]

Fixed by #5332