lnd c-lightning incompatibility

[Crypt-iQ]

When restoring HTLC's, c-lightning will send the HTLC's out of order (i.e. not monotonically increasing) which lnd doesn't expect. Example log output from lnd:

[alice] 2020-08-10 19:21:10.499 [DBG] PEER: Received UpdateAddHTLC(chan_id=cde2d57f1167394d462596f96a776159f92f2b26b3ecfa6bfc39bf5bbabf52cb, id=2938, amt=74788 mSAT, expiry=597, hash=bef6cbdcd234f054169c3e9e1b8ebd50ee8af7d35e70f2c405ade505c8268f19) from 0276086490698c35dd4bb59139f2c4f7fb362e4a28896ad1ab1152fb0c4830a6d6@127.0.0.1:53056
[alice] 2020-08-10 19:21:10.499 [DBG] PEER: Received UpdateAddHTLC(chan_id=cde2d57f1167394d462596f96a776159f92f2b26b3ecfa6bfc39bf5bbabf52cb, id=2914, amt=58650 mSAT, expiry=597, hash=bef6cbdcd234f054169c3e9e1b8ebd50ee8af7d35e70f2c405ade505c8268f19) from 0276086490698c35dd4bb59139f2c4f7fb362e4a28896ad1ab1152fb0c4830a6d6@127.0.0.1:53056
[alice] 2020-08-10 19:21:10.499 [DBG] PEER: Received UpdateAddHTLC(chan_id=cde2d57f1167394d462596f96a776159f92f2b26b3ecfa6bfc39bf5bbabf52cb, id=2801, amt=88584 mSAT, expiry=597, hash=bef6cbdcd234f054169c3e9e1b8ebd50ee8af7d35e70f2c405ade505c8268f19) from 0276086490698c35dd4bb59139f2c4f7fb362e4a28896ad1ab1152fb0c4830a6d6@127.0.0.1:53056
[alice] 2020-08-10 19:21:10.499 [DBG] PEER: Received UpdateAddHTLC(chan_id=cde2d57f1167394d462596f96a776159f92f2b26b3ecfa6bfc39bf5bbabf52cb, id=2943, amt=69839 mSAT, expiry=597, hash=bef6cbdcd234f054169c3e9e1b8ebd50ee8af7d35e70f2c405ade505c8268f19) from 0276086490698c35dd4bb59139f2c4f7fb362e4a28896ad1ab1152fb0c4830a6d6@127.0.0.1:53056
[alice] 2020-08-10 19:21:10.499 [DBG] PEER: Received UpdateAddHTLC(chan_id=cde2d57f1167394d462596f96a776159f92f2b26b3ecfa6bfc39bf5bbabf52cb, id=2959, amt=134633 mSAT, expiry=597, hash=cc2c6ce50b9b50ee4a482f5493e0e8e00dd5c9f470b8e410960dfef1ee668bf7) from 0276086490698c35dd4bb59139f2c4f7fb362e4a28896ad1ab1152fb0c4830a6d6@127.0.0.1:53056
[alice] 2020-08-10 19:21:10.499 [DBG] HSWC: ChannelLink(551:1:0): loaded 0 fwd pks
[alice] 2020-08-10 19:21:10.499 [DBG] PEER: Received UpdateAddHTLC(chan_id=cde2d57f1167394d462596f96a776159f92f2b26b3ecfa6bfc39bf5bbabf52cb, id=2777, amt=81882 mSAT, expiry=597, hash=bef6cbdcd234f054169c3e9e1b8ebd50ee8af7d35e70f2c405ade505c8268f19) from 0276086490698c35dd4bb59139f2c4f7fb362e4a28896ad1ab1152fb0c4830a6d6@127.0.0.1:53056
[alice] 2020-08-10 19:21:10.499 [ERR] HSWC: ChannelLink(551:1:0): failing link: unable to handle upstream add HTLC: ID 2787 on HTLC add does not match expected next ID 2776 with error: invalid update
[alice] 2020-08-10 19:21:10.499 [ERR] HSWC: ChannelLink(551:1:0): link failed, exiting htlcManager
[alice] 2020-08-10 19:21:10.499 [INF] HSWC: ChannelLink(551:1:0): exited

If c-lightning logs are needed, I can send those over. Thanks.

[cdecker]

I assume this is after the c-lightning node was restarted, i.e., it loaded the pending HTLCs from the database. Could it be that you are using the postgresql backend?

If that's the case it could be that a simple ORDER BY in the loading SQL query is sufficient to fix this.

For context: sqlite3 maintains insertion order when retrieving rows, which we might implicitly be relying on. postgres on the other hand relaxes this behavior for performance reasons, and could be causing this out of order loading.

[Crypt-iQ]

This was on a Debian box and the dependencies were installed by following the installation docs. So I installed libsqlite3-dev. I don't believe postgres was used since I didn't install it.

[cdecker]

Interesting, if you didn't specify a postgres wallet, then you'll have used the default sqlite3 backend. There are still a couple of potential causes for this issue:

• The c-lightning side didn't load / apply the HTLC updates in the correct order (unlikely since sqlite3 will maintain insertion order if not told to use a different order).
• The two sides are indeed out of sync:
  • c-lightning removed an HTLC prematurely
  • lnd didn't remove an HTLC that it should have removed

To me it seems like the latter at this point. Do you have logs for the c-lightning side, and specifically do you have logs regarding the HTLC in question?

[Crypt-iQ]

I have complete logs for this, where would you like them?

[rustyrussell]

Hmm, please try: sqlite3 ~/.lightning/bitcoin/lightningd.sqlite3 'select hex(last_sent_commit) from channels '

That will tell us what htlcs it's trying to send, to be sure.