Newsletters: add 186 (2022-02-09) #718
Conversation
There was a problem hiding this comment.
As usual a very nice overview of what has been happening. Thank you. I do believe that the PR 1227 of LDK was not summarized accurately though. To elaborate I put a longer comment within the text. Feel free to take over (parts of) my text (though I feel it might be too long for the newsletter)
If you agree with my assessment I could also offer to write a shorter summary in a separate PR for you to cherry pick from.
| historical payment failures/successes. These failures/successes are used to | ||
| determine the upper and lower bounds of channel balances, which gives the | ||
| route-finding logic a more accurate success probability when evaluating | ||
| routes. This is an implementation of the algorithm described in the paper by |
There was a problem hiding this comment.
Maybe the following comments are too long and expressive but I had just spend quite some time studying the PR and and the interesting research applied in this PR and it seems to me that the current summary of PR 1227 is a bit inaccurate. I will try to lay out my understanding (please correct me if I should misunderstand something)
If I understand correctly Pickhardt and Richter suggest to address the question of splitting multipart payments optimally by using minimum cost flows instead of searching for shortest paths of smaller amounts independently after splitting the full amount. This was was indeed addressed in news163 as indicated in the current draft of the newsletter.
However the PR 1227 of LDK seems not to add a minimum cost flow solver but focuses on adding Probabilistic channel scoring as a feature to the cost objective. This seems to be an idea that was previously introduced by Pickhardt et. al. and explained in news142. Besides adding probabilistic channel scoring as an additional feature (besides routing fees) as discussed in news142 (and as done by c-lighting and discussed in news172) the current PR 1227 also followed through the theoretical ideas explained in the probabilistic path finding paper to compute conditional probabilities if one previously observed successful or unsuccessful routing attempts. (I believe this is usually called Bayesian learning or Bayesian interference even though - and to my surprise - the authors of the two papers didn't use those terms)
Those ideas were also made more explicit in the work by Pickhardt and Richter by stating that nodes should maintain the so called Uncertainty Network. This is an additional view on the channel graph which encodes the success probabilities given the learnt knowledge about the liquidity in the channel from prior payment attempts with the help of evaluating conditional probabilities. This is indeed achieved by maintaining the upper and lower bounds of the liquidity in the channels. This was summarized with explicit formulas on how to update the probabilities to learn from previous failed and successful attempts by Pickhardt in the LDK issue tracker at lightningdevkit/rust-lightning#1170 (comment) before PR 1227 was opened
The question of multi part splitting seems not to be addressed or impoved by LDK yet as indicated in issue lightningdevkit/rust-lightning#1276.
There was a problem hiding this comment.
Ah my bad I linked the wrong paper! 🤦
I think my general description is still somewhat accurate, is there an important detail that you think I've missed that our readers would like summarized?
There was a problem hiding this comment.
I think it's no surprise that you confused the two paper. The author of PR 1227 also links to the second paper on minimum cost flows - where (to be fair) the uncertainty network was explicitly introduced by applying the section of information gain and conditional probabilities from the first paper on probabilistic path finding.
With respect to your question: I think it might make sense for the readers of the newsletter to recommend the summary of the two papers (and mention both?) which is given in lightningdevkit/rust-lightning#1170 (comment). I think that this text and the following discussion extracts the theoretical knowledge from both papers to a very practical hands on guide for engineers and developers. It certainly has helped me to understand what is going on there and how to
- maintain the Uncertainty Network by keeping the minimum and maximum liquidity (as stressed in your text) and learning those boundaries properly from failed and successful attempts.
- Translate the liquidity boundaries to conditional probabilities
The later seems to be (besides adding probabilistic channel scoring to the cost function in the first step) the core contribution of PR 1227. Adding the conditional probabilities also goes far beyond the c-lightning implementation of probabilistic scoring. The previous c-lightning change has - according to the blog post by Christian Decker - demonstrated substantial improvements by decreasing failure rate, overall time to complete and number of attempts by roughly 30 to 50%. Unfortunately there was no evaluation or benchmark conducted for the additional improvements that are introduced by PR 1227.
Other than that (but probably too detailed for the newsletter):
- PR 1227 introduces a half life of 3600 seconds to forget the learnt boundaries and this choice was somewhat arbitrary.
- Similar to the c-lightning implementation there was quite some discussion on how to combine the probabilistic channel scoring with the fee based scoring (weighted arithmetic mean vs harmonic or geometric mean). It seems no satisfying solution was found.
|
|
||
| - **Discussion about RBF policy:** Gloria Zhao started a | ||
| [discussion][zhao rbf] on the Bitcoin-Dev mailing list about | ||
| Replace-by-Fee ([RBF][topic rbf]) policy. Her initial email provides |
There was a problem hiding this comment.
maybe replace "initial" with "well received" as it seems a lot of people have been very delighted about the mail?
| determine for certain, without the use of heuristics, whether or not | ||
| it will earn the miner of that next block more fee income. Several | ||
| developers replied with comments on Zhao's summary and her proposals, | ||
| including with additional or alternative proposals for changes that |
There was a problem hiding this comment.
| including with additional or alternative proposals for changes that | |
| including additional or alternative proposals for changes that |
| [output script descriptor][topic descriptors] for any scriptPubKeys | ||
| that are decoded. | ||
|
|
||
| - [LND #6626][] sets 5% as the default fee for payments routed through |
There was a problem hiding this comment.
I believe this should be lightningnetwork/lnd#6226 instead. Pushing a small fix for this.
|
Made edits. Thanks @lightning-developer for the detailed post---I made @dongcarl's text a bit more generic and added some of the links you suggested, which I hope satisfies your concerns. Many thanks also to @glozow @bitschmidty @moneyball for your reviews. Reviewed contributions by @glozow @dongcarl and @adamjonas --- they all look great, thanks! Updated releases and added topic links. Thanks everyone! |
Fixes an issue reported by Steve Lee (thanks!)
bitschmidty
force-pushed
the
2022-02-09-newsletter
branch
from
f517230
to
66a6822
Compare
|
Super friends unite w/ contributions from @harding @dongcarl @xekyo @glozow @adamjonas and reviews from @lightning-developer @glozow @moneyball |
Sorry for opening the PR late!