Activate dual funding by default (#2825)

And update the release notes.

docs/release-notes/eclair-vnext.md

@@ -4,6 +4,22 @@
 
 ## Major changes
 
+### Dual funding
+
+After many years of work and refining the protocol, [dual funding](https://github.com/lightning/bolts/pull/851) has been added to the BOLTs.
+
+This release of eclair activates dual funding, and it will automatically be used with [cln](https://github.com/ElementsProject/lightning/) nodes. When opening channels to nodes that don't support dual funding, the older funding protocol will be used automatically.
+
+One of the immediate benefits of dual funding is that the funding transaction can now be RBF-ed, using the `rbfopen` RPC.
+
+There is currently no way to automatically add funds to channels that are being opened to your node, as deciding whether to do so or not really depends on each node operator's peering strategy.
+We have however created a [sample plugin](https://github.com/ACINQ/eclair-plugins/tree/master/channel-funding) that node operators can fork to implement their own strategy for contributing to inbound channels.
+
+### Update minimal version of Bitcoin Core
+
+With this release, eclair requires using Bitcoin Core 24.1.
+Newer versions of Bitcoin Core may be used, but haven't been extensively tested.
+
 ### Use priority instead of block target for feerates
 
 Eclair now uses a `slow`/`medium`/`fast` notation for feerates (in the style of mempool.space),
@@ -75,7 +91,45 @@ This feature leaks a bit of information about the balance when the channel is al
 
 ### Miscellaneous improvements and bug fixes
 
-<insert changes>
+#### Use bitcoinheaders.net v2
+
+Eclair uses <https://bitcoinheaders.net/> as one of its sources of blockchain data to detect when our node is being eclipsed.
+The format of this service is changing, and the older format will be deprecated soon.
+We thus encourage eclair nodes to update to ensure that they still have access to this blockchain watchdog.
+
+#### Force-closing anchor channels fee management
+
+Various improvements have been made to force-closing channels that need fees to be attached using CPFP or RBF.
+Those changes ensure that eclair nodes don't end up paying unnecessarily high fees to force-close channels, even when the mempool is full.
+
+#### Improve DB usage when closing channels
+
+When channels that have relayed a lot of HTLCs are closed, we can forget the revocation data for all of those HTLCs and free up space in our DB. We previously did that synchronously, which meant deleting potentially millions of rows synchronously. This isn't a high priority task, so we're now asynchronously deleting that data in smaller batches.
+
+Node operators can control the rate at which that data is deleted by updating the following values in `eclair.conf`:
+
+```conf
+// During normal channel operation, we need to store information about past HTLCs to be able to punish our peer if
+// they publish a revoked commitment. Once a channel closes or a splice transaction confirms, we can clean up past
+// data (which reduces the size of our DB). Since there may be millions of rows to delete and we don't want to slow
+// down the node, we delete those rows in batches at regular intervals.
+eclair.db.revoked-htlc-info-cleaner {
+  // Number of rows to delete per batch: a higher value will clean up the DB faster, but may have a higher impact on performance.
+  batch-size = 50000
+  // Frequency at which batches of rows are deleted: a lower value will clean up the DB faster, but may have a higher impact on performance.
+  interval = 15 minutes
+}
+```
+
+See <https://github.com/ACINQ/eclair/pull/2705> for more details.
+
+#### Correctly unlock wallet inputs during transaction eviction
+
+When the mempool is full and transactions are evicted, and potentially double-spent, the Bitcoin Core wallet cannot always safely unlock inputs.
+
+Eclair is now automatically detecting such cases and telling Bitcoin Core to unlock inputs that are safe to use. This ensures that node operators don't end up with unavailable liquidity.
+
+See <https://github.com/ACINQ/eclair/pull/2817> and <https://github.com/ACINQ/eclair/pull/2818> for more details.
 
 ## Verifying signatures
 

eclair-core/src/main/resources/reference.conf

@@ -70,7 +70,7 @@ eclair {
     option_anchors_zero_fee_htlc_tx = optional
     option_route_blinding = disabled
     option_shutdown_anysegwit = optional
-    option_dual_fund = disabled
+    option_dual_fund = optional
     option_quiesce = disabled
     option_onion_messages = optional
     option_channel_type = optional

eclair-core/src/test/scala/fr/acinq/eclair/blockchain/DummyOnChainWallet.scala

@@ -137,6 +137,7 @@ class SingleKeyOnChainWallet extends OnChainWallet with OnchainPubkeyCache {
   val pubkey = privkey.publicKey
   // We create a new dummy input transaction for every funding request.
   var inputs = Seq.empty[Transaction]
+  val published = collection.concurrent.TrieMap.empty[TxId, Transaction]
   var rolledback = Seq.empty[Transaction]
   var doubleSpent = Set.empty[TxId]
   var abandoned = Set.empty[TxId]
@@ -187,6 +188,7 @@ class SingleKeyOnChainWallet extends OnChainWallet with OnchainPubkeyCache {
 
   override def publishTransaction(tx: Transaction)(implicit ec: ExecutionContext): Future[TxId] = {
     inputs = inputs :+ tx
+    published += (tx.txid -> tx)
     Future.successful(tx.txid)
   }
 

eclair-core/src/test/scala/fr/acinq/eclair/integration/IntegrationSpec.scala

@@ -103,6 +103,8 @@ abstract class IntegrationSpec extends TestKitBaseClass with BitcoindService wit
     s"eclair.features.${ShutdownAnySegwit.rfcName}" -> "optional",
     s"eclair.features.${ChannelType.rfcName}" -> "optional",
     s"eclair.features.${RouteBlinding.rfcName}" -> "optional",
+    // We keep dual-funding disabled in tests, unless explicitly requested, as most of the network doesn't support it yet.
+    s"eclair.features.${DualFunding.rfcName}" -> "disabled",
   ).asJava)
 
   val withStaticRemoteKey = commonFeatures.withFallback(ConfigFactory.parseMap(Map(

eclair-core/src/test/scala/fr/acinq/eclair/integration/basic/TwoNodesIntegrationSpec.scala

@@ -1,7 +1,7 @@
 package fr.acinq.eclair.integration.basic
 
 import fr.acinq.bitcoin.scalacompat.{ByteVector32, SatoshiLong}
-import fr.acinq.eclair.channel.{DATA_NORMAL, NORMAL, RealScidStatus, WAIT_FOR_FUNDING_CONFIRMED}
+import fr.acinq.eclair.channel._
 import fr.acinq.eclair.integration.basic.fixtures.MinimalNodeFixture.getPeerChannels
 import fr.acinq.eclair.integration.basic.fixtures.composite.TwoNodesFixture
 import fr.acinq.eclair.testutils.FixtureSpec
@@ -51,7 +51,7 @@ class TwoNodesIntegrationSpec extends FixtureSpec with IntegrationPatience {
     val channelId2 = openChannel(bob, alice, 110_000 sat).channelId
     val channels = getPeerChannels(alice, bob.nodeId)
     assert(channels.map(_.data.channelId).toSet == Set(channelId1, channelId2))
-    channels.foreach(c => assert(c.state == WAIT_FOR_FUNDING_CONFIRMED))
+    channels.foreach(c => assert(c.state == WAIT_FOR_DUAL_FUNDING_SIGNED || c.state == WAIT_FOR_DUAL_FUNDING_CONFIRMED))
     confirmChannel(alice, bob, channelId1, BlockHeight(420_000), 21)
     confirmChannel(bob, alice, channelId2, BlockHeight(420_000), 22)
     getPeerChannels(bob, alice.nodeId).foreach(c => assert(c.data.isInstanceOf[DATA_NORMAL]))

eclair-core/src/test/scala/fr/acinq/eclair/integration/basic/fixtures/MinimalNodeFixture.scala

@@ -10,7 +10,7 @@ import com.typesafe.config.ConfigFactory
 import fr.acinq.bitcoin.scalacompat.Crypto.PublicKey
 import fr.acinq.bitcoin.scalacompat.{Block, ByteVector32, Satoshi, SatoshiLong, Transaction, TxId}
 import fr.acinq.eclair.ShortChannelId.txIndex
-import fr.acinq.eclair.blockchain.DummyOnChainWallet
+import fr.acinq.eclair.blockchain.SingleKeyOnChainWallet
 import fr.acinq.eclair.blockchain.bitcoind.ZmqWatcher
 import fr.acinq.eclair.blockchain.bitcoind.ZmqWatcher.{WatchFundingConfirmed, WatchFundingConfirmedTriggered, WatchFundingDeeplyBuried, WatchFundingDeeplyBuriedTriggered}
 import fr.acinq.eclair.blockchain.fee.{FeeratePerKw, FeeratesPerKw}
@@ -30,7 +30,7 @@ import fr.acinq.eclair.payment.relay.{ChannelRelayer, PostRestartHtlcCleaner, Re
 import fr.acinq.eclair.payment.send.PaymentInitiator
 import fr.acinq.eclair.router.Router
 import fr.acinq.eclair.wire.protocol.IPAddress
-import fr.acinq.eclair.{BlockHeight, MilliSatoshi, NodeParams, RealShortChannelId, SubscriptionsComplete, TestBitcoinCoreClient, TestDatabases}
+import fr.acinq.eclair.{BlockHeight, MilliSatoshi, NodeParams, SubscriptionsComplete, TestBitcoinCoreClient, TestDatabases}
 import org.scalatest.concurrent.{Eventually, IntegrationPatience}
 import org.scalatest.{Assertions, EitherValues}
 
@@ -56,7 +56,7 @@ case class MinimalNodeFixture private(nodeParams: NodeParams,
                                       offerManager: typed.ActorRef[OfferManager.Command],
                                       postman: typed.ActorRef[Postman.Command],
                                       watcher: TestProbe,
-                                      wallet: DummyOnChainWallet,
+                                      wallet: SingleKeyOnChainWallet,
                                       bitcoinClient: TestBitcoinCoreClient) {
   val nodeId = nodeParams.nodeId
   val routeParams = nodeParams.routerConf.pathFindingExperimentConf.experiments.values.head.getDefaultRouteParams
@@ -88,7 +88,7 @@ object MinimalNodeFixture extends Assertions with Eventually with IntegrationPat
     val readyListener = TestProbe("ready-listener")
     system.eventStream.subscribe(readyListener.ref, classOf[SubscriptionsComplete])
     val bitcoinClient = new TestBitcoinCoreClient()
-    val wallet = new DummyOnChainWallet()
+    val wallet = new SingleKeyOnChainWallet()
     val watcher = TestProbe("watcher")
     val triggerer = TestProbe("payment-triggerer")
     val watcherTyped = watcher.ref.toTyped[ZmqWatcher.Command]
@@ -187,14 +187,15 @@ object MinimalNodeFixture extends Assertions with Eventually with IntegrationPat
   }
 
   def fundingTx(node: MinimalNodeFixture, channelId: ByteVector32)(implicit system: ActorSystem): Transaction = {
-    val fundingTxid = getChannelData(node, channelId).asInstanceOf[ChannelDataWithCommitments].commitments.latest.fundingTxId
-    node.wallet.funded(fundingTxid)
+    getChannelData(node, channelId).asInstanceOf[ChannelDataWithCommitments].commitments.latest.localFundingStatus.signedTx_opt.get
   }
 
   def confirmChannel(node1: MinimalNodeFixture, node2: MinimalNodeFixture, channelId: ByteVector32, blockHeight: BlockHeight, txIndex: Int)(implicit system: ActorSystem): Option[RealScidStatus.Temporary] = {
-    assert(getChannelState(node1, channelId) == WAIT_FOR_FUNDING_CONFIRMED)
-    val data1Before = getChannelData(node1, channelId).asInstanceOf[DATA_WAIT_FOR_FUNDING_CONFIRMED]
-    val fundingTx = data1Before.fundingTx_opt.get
+    val fundingTx = getChannelData(node1, channelId) match {
+      case d: DATA_WAIT_FOR_DUAL_FUNDING_SIGNED => d.signingSession.fundingTx.tx.buildUnsignedTx()
+      case d: DATA_WAIT_FOR_DUAL_FUNDING_CONFIRMED => d.latestFundingTx.sharedTx.tx.buildUnsignedTx()
+      case d => fail(s"unexpected channel=$d")
+    }
 
     val watch1 = node1.watcher.fishForMessage() { case w: WatchFundingConfirmed if w.txId == fundingTx.txid => true; case _ => false }.asInstanceOf[WatchFundingConfirmed]
     val watch2 = node2.watcher.fishForMessage() { case w: WatchFundingConfirmed if w.txId == fundingTx.txid => true; case _ => false }.asInstanceOf[WatchFundingConfirmed]
@@ -218,8 +219,7 @@ object MinimalNodeFixture extends Assertions with Eventually with IntegrationPat
   def confirmChannelDeep(node1: MinimalNodeFixture, node2: MinimalNodeFixture, channelId: ByteVector32, blockHeight: BlockHeight, txIndex: Int)(implicit system: ActorSystem): RealScidStatus.Final = {
     assert(getChannelState(node1, channelId) == NORMAL)
     val data1Before = getChannelData(node1, channelId).asInstanceOf[DATA_NORMAL]
-    val fundingTxid = data1Before.commitments.latest.fundingTxId
-    val fundingTx = node1.wallet.funded(fundingTxid)
+    val fundingTx = data1Before.commitments.latest.localFundingStatus.signedTx_opt.get
 
     val watch1 = node1.watcher.fishForMessage() { case w: WatchFundingDeeplyBuried if w.txId == fundingTx.txid => true; case _ => false }.asInstanceOf[WatchFundingDeeplyBuried]
     val watch2 = node2.watcher.fishForMessage() { case w: WatchFundingDeeplyBuried if w.txId == fundingTx.txid => true; case _ => false }.asInstanceOf[WatchFundingDeeplyBuried]
@@ -269,17 +269,6 @@ object MinimalNodeFixture extends Assertions with Eventually with IntegrationPat
     sender.expectMsgType[Router.Data]
   }
 
-  /**
-   * Computes a deterministic [[RealShortChannelId]] based on a txid. We need this so that watchers can verify
-   * transactions in a independent and stateless fashion, since there is no actual blockchain in those tests.
-   */
-  def deterministicShortId(txId: TxId): RealShortChannelId = {
-    val blockHeight = txId.value.take(3).toInt(signed = false)
-    val txIndex = txId.value.takeRight(2).toInt(signed = false)
-    val outputIndex = 0 // funding txs created by the dummy wallet used in tests only have one output
-    RealShortChannelId(BlockHeight(blockHeight), txIndex, outputIndex)
-  }
-
   /** All known funding txs (we don't evaluate immediately because new ones could be created) */
   def knownFundingTxs(nodes: MinimalNodeFixture*): () => Iterable[Transaction] = () => nodes.flatMap(_.wallet.published.values)
 
@@ -306,9 +295,9 @@ object MinimalNodeFixture extends Assertions with Eventually with IntegrationPat
         Behaviors.withTimers { timers =>
           Behaviors.receiveMessagePartial {
             case vr: ZmqWatcher.ValidateRequest =>
-              val res = knownFundingTxs().find(tx => deterministicShortId(tx.txid) == vr.ann.shortChannelId) match {
+              val res = knownFundingTxs().find(tx => deterministicTxCoordinates(tx.txid) == (vr.ann.shortChannelId.blockHeight, txIndex(vr.ann.shortChannelId))) match {
                 case Some(fundingTx) => Right(fundingTx, ZmqWatcher.UtxoStatus.Unspent)
-                case None => Left(new RuntimeException(s"unknown realScid=${vr.ann.shortChannelId}, known=${knownFundingTxs().map(tx => deterministicShortId(tx.txid)).mkString(",")}"))
+                case None => Left(new RuntimeException(s"unknown realScid=${vr.ann.shortChannelId}, known=${knownFundingTxs().map(tx => deterministicTxCoordinates(tx.txid)).mkString(",")}"))
               }
               vr.replyTo ! ZmqWatcher.ValidateResult(vr.ann, res)
               Behaviors.same
@@ -319,16 +308,16 @@ object MinimalNodeFixture extends Assertions with Eventually with IntegrationPat
               }
               Behaviors.same
             case watch: ZmqWatcher.WatchFundingConfirmed if confirm =>
-              val realScid = deterministicShortId(watch.txId)
+              val (blockHeight, txIndex) = deterministicTxCoordinates(watch.txId)
               knownFundingTxs().find(_.txid == watch.txId) match {
-                case Some(fundingTx) => watch.replyTo ! ZmqWatcher.WatchFundingConfirmedTriggered(realScid.blockHeight, txIndex(realScid), fundingTx)
+                case Some(fundingTx) => watch.replyTo ! ZmqWatcher.WatchFundingConfirmedTriggered(blockHeight, txIndex, fundingTx)
                 case None => timers.startSingleTimer(watch, 10 millis)
               }
               Behaviors.same
             case watch: ZmqWatcher.WatchFundingDeeplyBuried if deepConfirm =>
-              val realScid = deterministicShortId(watch.txId)
+              val (blockHeight, txIndex) = deterministicTxCoordinates(watch.txId)
               knownFundingTxs().find(_.txid == watch.txId) match {
-                case Some(fundingTx) => watch.replyTo ! ZmqWatcher.WatchFundingDeeplyBuriedTriggered(realScid.blockHeight, txIndex(realScid), fundingTx)
+                case Some(fundingTx) => watch.replyTo ! ZmqWatcher.WatchFundingDeeplyBuriedTriggered(blockHeight, txIndex, fundingTx)
                 case None => timers.startSingleTimer(watch, 10 millis)
               }
               Behaviors.same
@@ -338,6 +327,16 @@ object MinimalNodeFixture extends Assertions with Eventually with IntegrationPat
         }
       }
     }
+
+    /**
+     * We don't use a blockchain in this test setup, but we want to simulate channels being confirmed.
+     * We choose a block height and transaction index at which the channel confirms deterministically from its txid.
+     */
+    def deterministicTxCoordinates(txId: TxId): (BlockHeight, Int) = {
+      val blockHeight = txId.value.take(3).toInt(signed = false)
+      val txIndex = txId.value.takeRight(2).toInt(signed = false)
+      (BlockHeight(blockHeight), txIndex)
+    }
   }
 
   def sendPayment(node1: MinimalNodeFixture, amount: MilliSatoshi, invoice: Bolt11Invoice)(implicit system: ActorSystem): Either[PaymentFailed, PaymentSent] = {

eclair-core/src/test/scala/fr/acinq/eclair/integration/basic/payment/OfferPaymentSpec.scala

@@ -34,14 +34,11 @@ import fr.acinq.eclair.payment._
 import fr.acinq.eclair.payment.offer.OfferManager
 import fr.acinq.eclair.payment.receive.MultiPartHandler.{DummyBlindedHop, ReceivingRoute}
 import fr.acinq.eclair.payment.send.PaymentInitiator.{SendPaymentToNode, SendSpontaneousPayment}
-import fr.acinq.eclair.payment.send.{ClearRecipient, OfferPayment, PaymentLifecycle}
-import fr.acinq.eclair.router.Router
+import fr.acinq.eclair.payment.send.{OfferPayment, PaymentLifecycle}
 import fr.acinq.eclair.testutils.FixtureSpec
 import fr.acinq.eclair.wire.protocol.OfferTypes.{Offer, OfferPaths}
-import fr.acinq.eclair.wire.protocol.{IncorrectOrUnknownPaymentDetails, InvalidOnionBlinding}
-import fr.acinq.eclair.{CltvExpiry, CltvExpiryDelta, Features, MilliSatoshi, MilliSatoshiLong, ShortChannelId, randomBytes32, randomKey}
 import fr.acinq.eclair.wire.protocol.{IncorrectOrUnknownPaymentDetails, InvalidOnionBlinding, OfferTypes}
-import fr.acinq.eclair.{CltvExpiryDelta, Features, MilliSatoshi, MilliSatoshiLong, randomBytes32, randomKey}
+import fr.acinq.eclair.{CltvExpiryDelta, Features, MilliSatoshi, MilliSatoshiLong, ShortChannelId, randomBytes32, randomKey}
 import org.scalatest.concurrent.IntegrationPatience
 import org.scalatest.{Tag, TestData}
 import scodec.bits.HexStringSyntax