Feature request: Limitation of outgoing htlcs per channel

[M1ch43lV]

I am running regolancer simultaneously with 6 workers.
Sometimes a channel or the connection to the peer gets buggy and the different regolancer sessions do not limit the outgoing amount of htlcs so it is accumulating the payment htlcs on the buggy channel. See example: Nordicrails with 7 pending payment htlcs.

Especially in high-fee onchain environment a FC can get very expensive.
So it would be great to have a general check for stuck htlcs (e.g. allowing a configurable maximum of 2-3 stuck payments) on the outgoing channel before establishing new payments htlcs.

[rkfg]

This certainly does make sense, however, there are few serious caveats. Maybe @ziggie1984 can help with the API, I just took a quick look and couldn't find any effective ways to query unresolved HTLCs. The ways to obtain them:

• use ListChannels; extremely ineffective and slow, running this before every attempt would slow down the process to a crawl
• use ListPayments; can't show only unfinished payments, need to paginate and collect back, no certain way to make sure you've collected everything
• use SubscribeInvoices and track HTLCs internally; probably best speed out of all methods but you wouldn't know about HTLCs that were added before the program was even run

Perhaps since we do ListChannels anyway during the initial preparation, we can as well gather the existing HTLCs at that stage. Note, that it can still either be plagued with race conditions or contain false information, or the implementation would be pretty complex.

Race condition: we made sure there's space for more HTLCs in a particular channel, but before we attempt a payment another instance of regolancer did the same and added their HTLC (or maybe we have to forward someone else's). It's not bulletproof really.
False information: loading channels takes some time, if we subscribe to HTLC updates before or after that we might receive HTLC success/fail during ListChannels and since we haven't yet populated the data we'd miss HTLC cancellation and it will stay in our local tracking state until the program exits. I.e. regolancer would think there's a pending HTLC where there's none.
Complex implementation: store/replay the events that happened during initial ListChannels request. Maybe reload the full snapshot with ListChannels from time to time to make sure we haven't missed anything.

Or add a new API endpoint into upstream that returns all pending HTLCs, which shouldn't be hard technically I think but it'd require justification and this one problem is probably not enough.

In general, you can (and probably should) limit the number of in-flight HTLCs with a circuit breaker. Because even if you only limit regolancer you can still forward other payments through those channels and accumulate many stuck HTLCs.

In conclusion, not really sure this is in scope but if it's not hard to implement I welcome a PR. Otherwise increased complexity for a niche use case that can't even be reliably solved like this isn't worth it.

[M1ch43lV]

Very good explanation of the issues! Thanks!