Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

statistics #130

Merged
merged 9 commits into from
Nov 19, 2021
Merged

statistics #130

merged 9 commits into from
Nov 19, 2021

Conversation

robertream
Copy link
Collaborator

@russelltg this is mostly to address #70, but it also includes yet another major redesign of the srt_protocol module. The redesign made it easier for me to both test and implement statistics, but I also fixed a few bugs on the way as I added more to the unit test coverage. I also fixed #106 and #104. Sorry, I know it might be an overwhelming PR to review, but it would be great to get it merged so we can iteratively fill out all the desired statistics with much smaller commits. :)

@codecov
Copy link

codecov bot commented Nov 18, 2021

Codecov Report

Merging #130 (d9feafb) into master (dde23fa) will increase coverage by 1.35%.
The diff coverage is 95.08%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master     #130      +/-   ##
==========================================
+ Coverage   87.50%   88.85%   +1.35%     
==========================================
  Files          43       52       +9     
  Lines        8181     8527     +346     
==========================================
+ Hits         7159     7577     +418     
+ Misses       1022      950      -72     
Impacted Files Coverage Δ
srt-protocol/src/lib.rs 100.00% <ø> (ø)
...rt-protocol/src/packet/control/loss_compression.rs 100.00% <ø> (ø)
srt-protocol/src/packet/control/srt.rs 90.07% <ø> (ø)
srt-protocol/src/packet/data.rs 86.95% <ø> (ø)
srt-protocol/src/packet/mod.rs 69.84% <ø> (ø)
srt-protocol/src/packet/msg_number.rs 100.00% <ø> (ø)
srt-protocol/src/packet/socket_id.rs 100.00% <ø> (ø)
srt-protocol/src/packet/srt_version.rs 77.77% <ø> (ø)
srt-protocol/src/protocol/crypto/mod.rs 95.88% <ø> (ø)
srt-protocol/src/protocol/crypto/wrap.rs 95.34% <ø> (ø)
... and 52 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update dde23fa...d9feafb. Read the comment docs.

Copy link
Owner

@russelltg russelltg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be great to see statistics added to srt-transmit. For now it would be fine to get print the Debug implemenation

Also--remind me why we have both run_handler_loop and run_input_loop?

I'm a bit unsure about the new package structure of "export everything"--i'm much more generally in favor of not exporting private types (I noticed this with ReceiverContext, which seems clearly private, but it is exported from the crate...). I'm okay with it for now, at some point I'll make sure our exports are minimal.

Overall looks okay though, some of it I didn't fully look over because it's impossible to see what actually changed in these files...but we have a great testsuite which makes me much more confident in general.

srt-protocol/src/statistics.rs Show resolved Hide resolved
srt-tokio/src/socket.rs Show resolved Hide resolved
srt-tokio/src/socket.rs Outdated Show resolved Hide resolved
srt-protocol/src/connection/mod.rs Show resolved Hide resolved
srt-protocol/src/connection/mod.rs Outdated Show resolved Hide resolved
}
}

#[allow(clippy::trivially_copy_pass_by_ref)]
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why suppress this? Can't we just pass by self?

Copy link
Collaborator Author

@robertream robertream Nov 18, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was copied from an original implementation, basically untouched, so this is not net new code. I am unsure why this was here, I imagine just to appease clippy at some point.

@@ -158,6 +164,19 @@ impl MessagePacketCount {
}
}

#[derive(Debug, Default)]
pub struct ReceiveBufferStatistics {
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't see this used anywhere??

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is dead code, left over from an early iteration on the design, I'll delete.

Some(usize::try_from(s - self.seqno0).unwrap())
let index = usize::try_from(seq_number - self.seqno0).unwrap();
// assert!(index < self.buffer.len(), "{:?} !< {:?}", index, self.buffer.len());
Some(index)
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This can probably be rolled back--I assume you found that that assert was not true?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, and I revisited the code, in a type system as powerful as Rust's is, I find defensive runtime assertions in production code to be a code smell, often times a result of Primitive Obsession.

@russelltg
Copy link
Owner

Also, I'm sure you saw but

warning: field is never read: `adjustments`
  --> srt-protocol/src/protocol/receiver/time.rs:16:5
   |
16 |     adjustments: u64,
   |     ^^^^^^^^^^^^^^^^
   |
   = note: `#[warn(dead_code)]` on by default

warning: field is never read: `drift`
  --> srt-protocol/src/protocol/receiver/time.rs:17:5
   |
17 |     drift: (TimeSpan, TimeSpan),
   |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^

warning: `srt-protocol` (lib) generated 2 warnings

@robertream
Copy link
Collaborator Author

robertream commented Nov 18, 2021

It would be great to see statistics added to srt-transmit. For now it would be fine to get print the Debug implemenation

Yes, I was going to do this next.

Also--remind me why we have both run_handler_loop and run_input_loop?

I wasn't sure which API I liked best yet. We could also keep both, but extract the run_input_loop into a separate struct that wraps the DuplexConnection. This would reduce the confusion.

I'm a bit unsure about the new package structure of "export everything"--i'm much more generally in favor of not exporting private types (I noticed this with ReceiverContext, which seems clearly private, but it is exported from the crate...). I'm okay with it for now, at some point I'll make sure our exports are minimal.

I agree, and all of the protocol module is really just private. In this process of redesign/refactoring it was helpful to keep most everything as public. If we organize the modules so the public surface area and the private surface area are split apart into separate modules as close to the root of the library as possible, the visibility of the "internal" private modules is not a big concern. I think packet, connection, and settings will likely be the public surface area, the rest will be fully private, including sender and receiver.

ReceiverContext and SenderContext are used to invoke/implement behavior (i.e. mutations) for the Receiver and Sender "objects", yet implementing these behaviors in a way that is intimately familiar and responsible with the data of the Receiver and Sender. The design of ReceiverContext and SenderContext is influenced by (or at least my best attempt at) the DCI design pattern.

I chose this approach because dependency injection is problematic in general in most languages, but especially in Rust. Having used Go quite a bit at work the last few years, I also dislike passing dependencies into functions everywhere, it makes the call site messy. We already do this with "now", but ultimately I was seeing the need for multiple objects that would be participating in the interaction that implements these behaviors (packet output, metrics, timers, errors/events, logging, etc.). So it started to look like DCI could help.

Overall looks okay though, some of it I didn't fully look over because it's impossible to see what actually changed in these files...but we have a great testsuite which makes me much more confident in general.

I LOVE this test suite!

@robertream
Copy link
Collaborator Author

It would be great to see statistics added to srt-transmit. For now it would be fine to get print the Debug implementation.

@russelltg I added statistics to the high_bandwidth test. I made an attempt at srt-transmit but it really is quite a rat's nest of code. I don't have the energy to deal with something like that right now, because it would be too tempting for me to dramatically clean up the code which would end up taking much longer than needed to just get it working. I'll defer this exercise to someone else for now, but I imagine I'll eventually circle back and clean up srt-transmit too.

@russelltg
Copy link
Owner

Yeah...srt-transmit does a bunch of stuff and it's pretty complex. Let's add it when we feel more confident about the interface.

@russelltg russelltg closed this Nov 19, 2021
@russelltg russelltg reopened this Nov 19, 2021
@russelltg
Copy link
Owner

Oops

@russelltg russelltg merged commit aa61177 into russelltg:master Nov 19, 2021
@robertream robertream deleted the statistics branch December 19, 2021 18:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
2 participants