Fix the multi-part payments

[cdecker]

Fixes #3852
Fixes #3851
Fixes #3855

Removes the scary warning about bcli not being able to auto-connect to the JSON-RPC:
Fixes #3553
Fixes #3859

[cdecker]

Cherry-picked the contents of #3856 as suggested by @ZmnSCPxj but maintained authorship, since credit goes to @vincenzopalazzo and @ZmnSCPxj for finding this and fixing it before I could even look at it 👍

[cdecker]

Rebased and squashed after merding #3860

[rustyrussell]

OK, this passed the Twitter test!

Minor issue, it splits 2x too aggressively, proposed fix here:
rustyrussell@49cb5ba

Actually, I've just pushed it on the end, along with another fix I lost somehow :(

[rustyrussell]

Ack d6bb191

[ZmnSCPxj]

Perhaps late to bring this up, but: do we need to actually arrange payments in a tree as opposed to a flat array of sub-payments?

The reason I bring this up is because we might want to merge sub-payments. For example, consider the case where the only viable paths for two sub-payments cost too much (i.e. fee budget is violated), but if we merge them (and their fee budgets) we could succeed.

But if the sub-payments we want to merge are cousins instead of siblings in the tree of payments, how do we viably merge them? For example, consider the case below:

        0
       / \
      /   \
     1     2
    / \   / \
   3   4 5   6

Suppose 4 and 6 are currently ongoing (possibly because they reached the destination, so the destination is waiting on the other parts). And 3 and 5 are failing getroute because the viable routes are too expensive (e.g. 4 and 6 have locked up funds on cheaper routes). However, if we merge 3 and 5, it may be feasible (because they would pool their fee budgets together).

As each subpayment needs a unique numeric id, we could use a monotonically-increasing counter and add an explicit field. Merged payments can get any of the ids of the merged payments.

And of course modules have to handle merges as well, if their data structures are not void.

[rustyrussell]

Trivial rebase for typo fix.

[cdecker]

Perhaps late to bring this up, but: do we need to actually arrange payments in a tree as opposed to a flat array of sub-payments?

The reason I bring this up is because we might want to merge sub-payments. For example, consider the case where the only viable paths for two sub-payments cost too much (i.e. fee budget is violated), but if we merge them (and their fee budgets) we could succeed.

But if the sub-payments we want to merge are cousins instead of siblings in the tree of payments, how do we viably merge them? For example, consider the case below:

        0
       / \
      /   \
     1     2
    / \   / \
   3   4 5   6

Suppose 4 and 6 are currently ongoing (possibly because they reached the destination, so the destination is waiting on the other parts). And 3 and 5 are failing getroute because the viable routes are too expensive (e.g. 4 and 6 have locked up funds on cheaper routes). However, if we merge 3 and 5, it may be feasible (because they would pool their fee budgets together).

As each subpayment needs a unique numeric id, we could use a monotonically-increasing counter and add an explicit field. Merged payments can get any of the ids of the merged payments.

And of course modules have to handle merges as well, if their data structures are not void.

Discussions are always welcome, though I don't think that this should be discussed in a PR where it is likely to get lost, so I opened a new issue for this.

The TL;DR is that I think the merging or not of payments is independent of how we link the payments internally. We can represent the tree in an array just as well, and as you point out the partid are consecutive already, so we could just also stash them in an additional array.

I'm wondering whether we'd want to merge at all, or whether other techniques such as fee stealing from other subpayments that we consider to "have worked" (TM) are more promising.

[rustyrussell]

Ack aad62fc