Merge pull request #189 from greatest-ape/udp-overhaul-workers

udp: rewrite to use shared state instead of socket/swarm workers

CHANGELOG.md

@@ -18,28 +18,28 @@
 
 #### Changed
 
+* Switch from socket worker/swarm worker division to a single type of worker,
+  for performance reasons. Several config file keys were removed since they
+  are no longer needed.
 * Index peers by packet source IP and provided port, instead of by peer_id.
   This prevents users from impersonating others and is likely also slightly
   faster for IPv4 peers.
-* Remove support for unbounded worker channels
-* Add backpressure in socket workers. They will postpone reading from the
-  socket if sending a request to a swarm worker failed
 * Avoid a heap allocation for torrents with two or less peers. This can save
   a lot of memory if many torrents are tracked
 * Improve announce performance by avoiding having to filter response peers
 * In announce response statistics, don't include announcing peer
-* Distribute announce responses from swarm workers over socket workers to
-  decrease performance loss due to underutilized threads
 * Harden ConnectionValidator to make IP spoofing even more costly
 * Remove config key `network.poll_event_capacity` (always use 1)
 * Speed up parsing and serialization of requests and responses by using
   [zerocopy](https://crates.io/crates/zerocopy)
 * Report socket worker related prometheus stats per worker
+* Remove CPU pinning support
 
 #### Fixed
 
 * Quit whole application if any worker thread quits
 * Disallow announce requests with port value of 0
+* Fix io_uring UB issues
 
 ### aquatic_http
 

README.md

@@ -31,19 +31,15 @@ Known users:
 
 ## Performance of the UDP implementation
 
-![UDP BitTorrent tracker throughput comparison](./documents/aquatic-udp-load-test-illustration-2023-01-11.png)
+![UDP BitTorrent tracker throughput](./documents/aquatic-udp-load-test-2024-02-10.png)
 
-More benchmark details are available [here](./documents/aquatic-udp-load-test-2023-01-11.pdf).
+More benchmark details are available [here](./documents/aquatic-udp-load-test-2024-02-10.md).
 
 ## Usage
 
 Please refer to the README pages for the respective implementations listed in
 the table above.
 
-## Architectural overview
-
-![Architectural overview of aquatic](./documents/aquatic-architecture-2024.svg)
-
 ## Copyright and license
 
 Copyright (c) Joakim Frostegård

crates/bencher/src/protocols/udp.rs

@@ -58,6 +58,12 @@ impl UdpCommand {
         indexmap::indexmap! {
             1 => SetConfig {
                 implementations: indexmap! {
+                    UdpTracker::Aquatic => vec![
+                        AquaticUdpRunner::with_mio(1, Priority::High),
+                    ],
+                    UdpTracker::AquaticIoUring => vec![
+                        AquaticUdpRunner::with_io_uring(1, Priority::High),
+                    ],
                     UdpTracker::OpenTracker => vec![
                         OpenTrackerUdpRunner::new(0, Priority::Medium), // Handle requests within event loop
                         OpenTrackerUdpRunner::new(1, Priority::High),
@@ -74,16 +80,13 @@ impl UdpCommand {
             2 => SetConfig {
                 implementations: indexmap! {
                     UdpTracker::Aquatic => vec![
-                        AquaticUdpRunner::with_mio(1, 1, Priority::Medium),
-                        AquaticUdpRunner::with_mio(2, 1, Priority::High),
+                        AquaticUdpRunner::with_mio(2, Priority::High),
                     ],
                     UdpTracker::AquaticIoUring => vec![
-                        AquaticUdpRunner::with_io_uring(1, 1, Priority::Medium),
-                        AquaticUdpRunner::with_io_uring(2, 1, Priority::High),
+                        AquaticUdpRunner::with_io_uring(2, Priority::High),
                     ],
                     UdpTracker::OpenTracker => vec![
                         OpenTrackerUdpRunner::new(2, Priority::High),
-                        OpenTrackerUdpRunner::new(4, Priority::Medium),
                     ],
                     UdpTracker::Chihaya => vec![
                         ChihayaUdpRunner::new(),
@@ -97,12 +100,10 @@ impl UdpCommand {
             4 => SetConfig {
                 implementations: indexmap! {
                     UdpTracker::Aquatic => vec![
-                        AquaticUdpRunner::with_mio(3, 1, Priority::High),
-                        AquaticUdpRunner::with_mio(4, 1, Priority::Medium),
+                        AquaticUdpRunner::with_mio(4, Priority::High),
                     ],
                     UdpTracker::AquaticIoUring => vec![
-                        AquaticUdpRunner::with_io_uring(3, 1, Priority::High),
-                        AquaticUdpRunner::with_io_uring(4, 1, Priority::Medium),
+                        AquaticUdpRunner::with_io_uring(4, Priority::High),
                     ],
                     UdpTracker::OpenTracker => vec![
                         OpenTrackerUdpRunner::new(4, Priority::High),
@@ -119,10 +120,10 @@ impl UdpCommand {
             6 => SetConfig {
                 implementations: indexmap! {
                     UdpTracker::Aquatic => vec![
-                        AquaticUdpRunner::with_mio(5, 1, Priority::High),
+                        AquaticUdpRunner::with_mio(6, Priority::High),
                     ],
                     UdpTracker::AquaticIoUring => vec![
-                        AquaticUdpRunner::with_io_uring(5, 1, Priority::High),
+                        AquaticUdpRunner::with_io_uring(6, Priority::High),
                     ],
                     UdpTracker::OpenTracker => vec![
                         OpenTrackerUdpRunner::new(6, Priority::High),
@@ -139,10 +140,10 @@ impl UdpCommand {
             8 => SetConfig {
                 implementations: indexmap! {
                     UdpTracker::Aquatic => vec![
-                        AquaticUdpRunner::with_mio(7, 1, Priority::High),
+                        AquaticUdpRunner::with_mio(8, Priority::High),
                     ],
                     UdpTracker::AquaticIoUring => vec![
-                        AquaticUdpRunner::with_io_uring(7, 1, Priority::High),
+                        AquaticUdpRunner::with_io_uring(8, Priority::High),
                     ],
                     UdpTracker::OpenTracker => vec![
                         OpenTrackerUdpRunner::new(8, Priority::High),
@@ -159,12 +160,10 @@ impl UdpCommand {
             12 => SetConfig {
                 implementations: indexmap! {
                     UdpTracker::Aquatic => vec![
-                        AquaticUdpRunner::with_mio(10, 2, Priority::High),
-                        AquaticUdpRunner::with_mio(9, 3, Priority::Medium),
+                        AquaticUdpRunner::with_mio(12, Priority::High),
                     ],
                     UdpTracker::AquaticIoUring => vec![
-                        AquaticUdpRunner::with_io_uring(10, 2, Priority::High),
-                        AquaticUdpRunner::with_io_uring(9, 3, Priority::Medium),
+                        AquaticUdpRunner::with_io_uring(12, Priority::High),
                     ],
                     UdpTracker::OpenTracker => vec![
                         OpenTrackerUdpRunner::new(12, Priority::High),
@@ -181,10 +180,10 @@ impl UdpCommand {
             16 => SetConfig {
                 implementations: indexmap! {
                     UdpTracker::Aquatic => vec![
-                        AquaticUdpRunner::with_mio(13, 3, Priority::High),
+                        AquaticUdpRunner::with_mio(16, Priority::High),
                     ],
                     UdpTracker::AquaticIoUring => vec![
-                        AquaticUdpRunner::with_io_uring(13, 3, Priority::High),
+                        AquaticUdpRunner::with_io_uring(16, Priority::High),
                     ],
                     UdpTracker::OpenTracker => vec![
                         OpenTrackerUdpRunner::new(16, Priority::High),
@@ -211,32 +210,27 @@ impl UdpCommand {
 #[derive(Debug, Clone)]
 struct AquaticUdpRunner {
     socket_workers: usize,
-    swarm_workers: usize,
     use_io_uring: bool,
     priority: Priority,
 }
 
 impl AquaticUdpRunner {
     fn with_mio(
         socket_workers: usize,
-        swarm_workers: usize,
         priority: Priority,
     ) -> Rc<dyn ProcessRunner<Command = UdpCommand>> {
         Rc::new(Self {
             socket_workers,
-            swarm_workers,
             use_io_uring: false,
             priority,
         })
     }
     fn with_io_uring(
         socket_workers: usize,
-        swarm_workers: usize,
         priority: Priority,
     ) -> Rc<dyn ProcessRunner<Command = UdpCommand>> {
         Rc::new(Self {
             socket_workers,
-            swarm_workers,
             use_io_uring: true,
             priority,
         })
@@ -256,7 +250,6 @@ impl ProcessRunner for AquaticUdpRunner {
         let mut c = aquatic_udp::config::Config::default();
 
         c.socket_workers = self.socket_workers;
-        c.swarm_workers = self.swarm_workers;
         c.network.address = SocketAddr::V4(SocketAddrV4::new(Ipv4Addr::LOCALHOST, 3000));
         c.network.use_io_uring = self.use_io_uring;
         c.protocol.max_response_peers = 30;
@@ -283,7 +276,6 @@ impl ProcessRunner for AquaticUdpRunner {
     fn keys(&self) -> IndexMap<String, String> {
         indexmap! {
             "socket workers".to_string() => self.socket_workers.to_string(),
-            "swarm workers".to_string() => self.swarm_workers.to_string(),
         }
     }
 }

crates/common/src/lib.rs

@@ -163,6 +163,7 @@ pub enum WorkerType {
     Socket(usize),
     Statistics,
     Signals,
+    Cleaning,
     #[cfg(feature = "prometheus")]
     Prometheus,
 }
@@ -174,6 +175,7 @@ impl Display for WorkerType {
             Self::Socket(index) => f.write_fmt(format_args!("Socket worker {}", index + 1)),
             Self::Statistics => f.write_str("Statistics worker"),
             Self::Signals => f.write_str("Signals worker"),
+            Self::Cleaning => f.write_str("Cleaning worker"),
             #[cfg(feature = "prometheus")]
             Self::Prometheus => f.write_str("Prometheus worker"),
         }

crates/http/README.md

@@ -109,6 +109,10 @@ Implements:
 `aquatic_http` has not been tested as much as `aquatic_udp`, but likely works
 fine in production.
 
+## Architectural overview
+
+![Architectural overview of aquatic](../../documents/aquatic-architecture-2024.svg)
+
 ## Copyright and license
 
 Copyright (c) Joakim Frostegård

crates/udp/Cargo.toml

@@ -48,6 +48,7 @@ log = "0.4"
 mimalloc = { version = "0.1", default-features = false }
 mio = { version = "0.8", features = ["net", "os-poll"] }
 num-format = "0.4"
+parking_lot = "0.12"
 rand = { version = "0.8", features = ["small_rng"] }
 serde = { version = "1", features = ["derive"] }
 signal-hook = { version = "0.3" }

crates/udp/README.md

@@ -21,9 +21,9 @@ This is the most mature implementation in the aquatic family. I consider it full
 
 ## Performance
 
-![UDP BitTorrent tracker throughput comparison](../../documents/aquatic-udp-load-test-illustration-2023-01-11.png)
+![UDP BitTorrent tracker throughput](../../documents/aquatic-udp-load-test-2024-02-10.png)
 
-More benchmark details are available [here](../../documents/aquatic-udp-load-test-2023-01-11.pdf).
+More benchmark details are available [here](../../documents/aquatic-udp-load-test-2024-02-10.md).
 
 ## Usage