-
Notifications
You must be signed in to change notification settings - Fork 41
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Refactor: move shared test functionality to a dev-dependency #170
Refactor: move shared test functionality to a dev-dependency #170
Conversation
Hi @josecelano , Do you know why my latest commit fails the |
Hi @WarmBeer, you can check in on the workflow output. I have a problem I have not been able to fix. For some reason, when the Visual Studio Code changes a Try to run the linter locally for the file that has the error in the pipeline and using exactly the same configuration. We have to find out what this is happening. It's a pretty annoying error, especially when you realize there was an error in the pipeline some commits later :-(. |
Thank you @josecelano ! I was using |
Hey @josecelano , I think the issue lies here: https://github.com/torrust/torrust-tracker/actions/runs/4233786546/workflow#L16 . The format job in the pipeline runs on the When I run I do prefer the nightly fmt though, so I guess we should keep it like this. |
Upss, sorry, I thought I told you I was also using the nightly:
I think I started using it when @da2ce7 was working on fixing all clipply warnings. |
Hi @josecelano , @da2ce7 , This PR is ready for review :). |
ffc1307
to
ce856bd
Compare
src/http/tracker_interface.rs
Outdated
|
||
/// Trait to be implemented by a http interface for the tracker. | ||
#[allow(clippy::module_name_repetitions)] | ||
pub trait TrackerInterfaceTrait: Sync + Send { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
hi @WarmBeer, I usually try to avoid using words like "Interface", "Trait", "Base" in class/interface/struct names. Besides, with that name, it seems you are implementing a Tracker. What do you think about HttpTrackerStarter
or HttpTrackerLauncher
?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I see you have the other struct:
pub struct TrackerInterface<S> {
cfg: torrust_tracker_configuration::HttpTracker,
state: S,
}
Maybe we could call this one just HttpTrackerHandler
or HttpServerHandler
, depending on whether we want to use HttpServer
or HttpTrackerServer
for the thread running the service.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In fact, you are already using those names in:
pub type StoppedHttpServer<I> = TrackerInterface<Stopped<I>>;
pub type RunningHttpServer<I> = TrackerInterface<Running<I>>;
Why not this?:
pub type StoppedHttpServer<I> = HttpServer<Stopped<I>>;
pub type RunningHttpServer<I> = HttpServer<Running<I>>;
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In summary, you would have:
pub trait HttpServerStarter: Sync + Send {
fn start_with_graceful_shutdown<F>(...) -> ...
}
pub struct HttpServer<S> {
cfg: torrust_tracker_configuration::HttpTracker,
state: S,
}
pub type StoppedHttpServer<I> = HttpServer<Stopped<I>>;
pub type RunningHttpServer<I> = HttpServer<Running<I>>;
impl<I: HttpServerStarter + 'static> StoppedHttpServer<I> { ... }
impl<I: HttpServerStarter> RunningHttpServer<I> { ... }
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hi @josecelano ,
To provide a bit of background to my commits:
We have talked about making the tracker agnostic of the server
or interface
that is being used to access it, so that we can run it using any http
or udp
server. As of right now, we already have two http
server implementations: Axum
and Warp
.
To not have to deal with a separate server struct for every implementation when starting them, I have introduced the TrackerInterface
wrapper struct. This TrackerInterface
struct wraps any http server implementation that implements the trait TrackerInterfaceTrait
. Like axum and warp.
The reason that I chose the word interface
is because to me the Tracker
struct is the core of our tracker. And udp
and http
are just interfaces to interact with the public functions of our core Tracker
. But I see that the name might be too confusing.
I think that naming the types like this would avoid any confusion:
struct
http::TrackerInterface
-> http::HttpServerWrapper
trait
http::TrackerInterfaceTrait
-> http::HttpServer
What do you think?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Actually after thinking about it I like the name “HttpServerStarter” :)
As the struct that implements the trait is only a launcher of the a http server and not the entire server itself.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
hi, @WarmBeer thank you for clarifying it!
First at all, We are going to remove the Warp implementation soon, but I guess we could have different tracker versions in the future: V1, V2, etcetera. But in that case, I think it would not work because we would break that "interface".
Secondly, I agree that the Tracker
is the "core" tracker, and the UDP and HTTP trackers are wrappers, but I think they are not only thin wrappers. From my perspective, the Tracker
is a domain service and the UDPTracker
, and
HTTPTracker
are application services.
I think UDP and HTTP tracker services have these main responsibilities:
- A different "delivery" layer.
- Authentication
- Collect specific statistics
https://enterprisecraftsmanship.com/posts/domain-vs-application-services/
I think the API would be another App service as well.
Finally, I usually call "server" a running service. In our case, the thread which is running the app service (UDP, HTTP or API).
What do you think?
On the other hand, I find it strange that the core Tracker
handles HTTP authentication for the HTTP tracker.
In the end, I suppose it depends on how you see it, but I think the current core Tracker
already has too many responsibilities. I would even more functions from the tracker (I think you have some comments about it, for example for persistence).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Actually after thinking about it I like the name “HttpServerStarter” :)
As the struct that implements the trait is only a launcher of the a http server and not the entire server itself.
I think its only responsibilities are:
- Launching the actual server
- Knowing its state
- Knowing the configuration needed to launch it. Secondary responsibility derived from the first one.
tests/common/tracker.rs
Outdated
} | ||
|
||
// TODO: Move to test-helpers crate once `Tracker` is isolated. | ||
pub fn tracker_instance(configuration: &Arc<torrust_tracker_configuration::Configuration>) -> Arc<Tracker> { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
hi @WarmBeer tracker_isntance
makes me think that's a singleton and not a factory. Maybe tracker_factory
or new_tracker_instance
. I would choose the first one because it's not a constructor; it's a dependency "resolver". In fact, I think this is the app container and bootstrapping at the same time.
src/apis/server.rs
Outdated
|
||
use crate::apis::server::ApiServer; | ||
use crate::tracker; | ||
use crate::tracker::statistics; | ||
|
||
fn tracker_configuration() -> Arc<Configuration> { | ||
Arc::new(ephemeral()) | ||
Arc::new(torrust_tracker_test_helpers::configuration::ephemeral()) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think it would be enougth with:
Arc::new(configuration::ephemeral())
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Cool!
Hi @WarmBeer I think we have a similar pattern here when we run the stats event listener: impl Keeper {
#[must_use]
pub fn new() -> Self {
Self { repository: Repo::new() }
}
#[must_use]
pub fn new_active_instance() -> (Box<dyn EventSender>, Repo) {
let mut stats_tracker = Self::new();
let stats_event_sender = stats_tracker.run_event_listener();
(stats_event_sender, stats_tracker.repository)
}
pub fn run_event_listener(&mut self) -> Box<dyn EventSender> {
let (sender, receiver) = mpsc::channel::<Event>(CHANNEL_BUFFER_SIZE);
let stats_repository = self.repository.clone();
tokio::spawn(async move { event_listener(receiver, stats_repository).await });
Box::new(Sender { sender })
}
} I've added a new test for the tracker and I'm getting this error: thread 'tracker::tests::the_tracker::configured_as_whitelisted::handling_an_scrape_request::it_should_be_able_to_build_a_zeroed_scrape_data_for_a_list_of_info_hashes' panicked at 'there is no reactor running, must be called from the context of a Tokio 1.x runtime', src/tracker/statistics.rs:74:9
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace It's seems it needs that listeners to be ready but it's strange because I'm testing a simple method (which is not async) and does not have any dependency. Anyway, it's only related to this PR but I suppose we should apply the same pattern (state machine) you have applied here. We can do it in a new PR. |
Hi @josecelano , You can only call |
Hi @josecelano , I have fixed most tests that I broke with my latest commit. There are however still 18 tests that fail and I don't really know why. It seems that most of these fails have something to do with this error I get from the tracker:
Do you know what causes this? |
Hi @WarmBeer I'm using this crate https://docs.rs/axum-client-ip/latest/axum_client_ip/ To resolve the remote client IP and it requires to start the server like this: .serve(
// Don't forget to add `ConnetInfo` if you aren't behind a proxy
app.into_make_service_with_connect_info::<SocketAddr>() ) I think you probably remove that part, but I haven't checked it. |
Thank you @josecelano , that fixed it! |
src/http/tracker_interface.rs
Outdated
@@ -0,0 +1,100 @@ | |||
use std::future::Future; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hi @WarmBeer should we also rename the mod tracker_interface
?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, I forgot :)
src/tracker/services/torrent.rs
Outdated
use crate::protocol::info_hash::InfoHash; | ||
use crate::tracker::services::common::tracker_factory; | ||
use crate::tracker::services::torrent::tests::sample_peer; | ||
use crate::tracker::services::torrent::{get_torrent_info, Info}; | ||
|
||
pub fn tracker_configuration() -> Arc<Configuration> { | ||
Arc::new(ephemeral_configuration()) | ||
Arc::new(torrust_tracker_test_helpers::configuration::ephemeral()) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I've commented on another line like this:
I think this should be enough:
Arc::new(configuration::ephemeral())
src/apis/server.rs
Outdated
let job = tokio::spawn(async move { | ||
if let (true, Some(ssl_cert_path), Some(ssl_key_path)) = (cfg.ssl_enabled, cfg.ssl_cert_path, cfg.ssl_key_path) { | ||
let tls_config = RustlsConfig::from_pem_file(ssl_cert_path, ssl_key_path) | ||
.await | ||
.expect("Could not read ssl cert and/or key."); | ||
|
||
start_tls_from_tcp_listener_with_graceful_shutdown(listener, tls_config, &tracker, receiver) | ||
.await | ||
.expect("Could not start from tcp listener with tls."); | ||
} else { | ||
start_from_tcp_listener_with_graceful_shutdown(listener, &tracker, receiver) | ||
.await | ||
.expect("Could not start from tcp listener."); | ||
} | ||
}); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hi @josecelano ,
Your question has kept me occupied for an hour xd. So the problem with these tokio spawns is that they will not abort the parent program when a panic occurs. So we won't actually know if there was a panic in a task.
You solved this partially here by using a channel to report whether the api server has started. The problem with our current code however is that it will only check whether the RusttlsConfig
was created successfully and not if the actual server has started successfully.
We could add another channel that will report something when the await on the server start function has yielded an error. This way we can wait a little bit after starting the server start job/tokio task to see whether an error has been sent through this channel.
So to answer your questions:
Has the behaviour changed when you cannot start the server with TSL?
Yes and I need to fix it :).
Does the expect means the whole application is going to halt when the TLS configuration is missing? If that's true, should not we just warn the user and try to start other services?
Since the expect
will yield a panic on a tokio task, only the task will abort, but the rest of the app will continue like nothing happened.
In my opinion, the entire app should abort when a panic occurs in any of the tasks.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hi @WarmBeer, thank you for the clarification!
I think we should start all or nothing too. People don't read logs :-). If something fails and you don't care, you should explicitly disable it.
In summary, we need to change even the previous behaviour if it only shows a warning.
This is getting more and more complex. I do not know if we should add a second one-shot channel or maybe a normal channel and define a list of messages between the launcher and the "launched" thing. Alternatively, we could try to check if the service is alive and ready just using the public contract. For example, for the API, we could check if the status
endpoint is returning a 200
, but I do not like this solution because the launcher uses a "polling" approach. I think a proper communication channel between the laucher and the child is the right solution.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I will restore the current "develop" behaviour in this PR and will open a new issue for how we are going to deal with this.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
OK @WarmBeer. I think we could replace the warning with an error, at least.
impl<S> TestEnvironment<S> { | ||
/// Add a torrent to the tracker | ||
#[allow(dead_code)] | ||
pub async fn add_torrent(&self, info_hash: &InfoHash, peer: &Peer) { | ||
self.tracker.update_torrent_with_peer_and_get_stats(info_hash, peer).await; | ||
} | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hi @WarmBeer, this is OK but if you read this discussion I think we should stop using that method. In fact, it should be private.
Besides, now we have a high-level public method to announce a new peer: Tracker::announce
. Maybe in this case we could use:
impl<S> TestEnvironment<S> {
/// Add a torrent to the tracker
#[allow(dead_code)]
pub async fn announce_torrent(&self, info_hash: &InfoHash, peer: &mut Peer, remote_client_ip: &IpAddr) {
self.tracker.announce(info_hash, peer: &mut Peer, remote_client_ip).await;
}
}
Althougth that could have side effects we do not want in these tests.
efe78c0
to
46d545d
Compare
Hi @josecelano , @da2ce7 , This PR has been rebased and is ready for review/merging (finally). |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hey @WarmBeer I think there are a lot of regressions. I think you reverted some changes that were already merged.
src/apis/handlers.rs
Outdated
@@ -17,7 +17,7 @@ use crate::apis::resources::auth_key::AuthKey; | |||
use crate::apis::resources::stats::Stats; | |||
use crate::apis::resources::torrent::ListItem; | |||
use crate::protocol::info_hash::InfoHash; | |||
use crate::tracker::auth::Key; | |||
use crate::tracker::auth::KeyId; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hi @WarmBeer I renamed the KeyId
to Key
.
src/apis/handlers.rs
Outdated
@@ -86,7 +86,7 @@ pub async fn remove_torrent_from_whitelist_handler( | |||
} | |||
|
|||
pub async fn reload_whitelist_handler(State(tracker): State<Arc<Tracker>>) -> Response { | |||
match tracker.load_whitelist_from_database().await { | |||
match tracker.load_whitelist().await { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hi @WarmBeer I renamed the load_whitelist
to load_whitelist_from_database
because some of them had the suffix from_database
and others don't:
load_keys_from_database
load_whitelist_from_database
load_torrents_from_database
src/apis/resources/auth_key.rs
Outdated
} | ||
|
||
impl From<AuthKey> for auth::ExpiringKey { | ||
impl From<AuthKey> for auth::Key { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hi @WarmBeer Key
was reanamed to ExpiringKey
src/apis/resources/auth_key.rs
Outdated
auth::Key::from(auth_key_resource), | ||
auth::Key { | ||
key: "IaWDneuFNZi8IB4MPA3qW1CD0M30EZSM".to_string(), // cspell:disable-line | ||
valid_until: Some(Current::add(&Duration::new(duration_in_secs, 0)).unwrap()) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@WarmBeer , the valid_until
attribute is not optional anymore.
Hi @josecelano , Oops, that was not my intention. I did the same rebase steps as last time: git clean -fx
git fetch —all
git switch develop
git merge --ff-only
git switch 156-fix-random-test-failure-due-to-port-collision
git reset develop
git add -A
git commit -m "isolated configuration, test-helpers and primitives crates"
git push --force Should I do a merge commit instead? |
Hi @WarmBeer I think that only works if I would reset git clean -fx
git fetch —all
git switch develop
git merge --ff-only
git switch 156-fix-random-test-failure-due-to-port-collision
git rebase develop
# Resolve conflicts
# Rewrite history
git reset develop
git add -A
git commit -m "isolated configuration, test-helpers and primitives crates"
git push --force |
46d545d
to
efe78c0
Compare
…nd isolated crates
51d858b
to
7e550c6
Compare
Hi @josecelano , @da2ce7 , This should be ready for merging now :) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hi @WarmBeer I think all the Axum integration tests are missing. They were mostly the same but there are some new:
https://github.com/torrust/torrust-tracker/blob/develop/tests/http_tracker.rs
Hey @josecelano , Whenever I try to merge origin/develop into this branch, I get wildly different files compared to origin/develop. It merges some functions where I have no idea where they came from. Some functions don't even come from either branch. For example:
mod axum_http_tracker_server {
// WIP: migration HTTP from Warp to Axum
use local_ip_address::local_ip;
use torrust_tracker::http::axum_implementation::extractors::remote_client_ip::RemoteClientIp;
use torrust_tracker::http::axum_implementation::resources::ok::Ok;
use torrust_tracker::http::Version;
use crate::http::client::Client;
use crate::http::server::start_default_http_tracker;
#[tokio::test]
async fn should_return_the_status() {
// This is a temporary test to test the new Axum HTTP tracker server scaffolding
let http_tracker_server = start_default_http_tracker(Version::Axum).await;
let client_ip = local_ip().unwrap();
let response = Client::bind(http_tracker_server.get_connection_info(), client_ip)
.get("status")
.await;
let ok: Ok = serde_json::from_str(&response.text().await.unwrap()).unwrap();
assert_eq!(
ok,
Ok {
remote_client_ip: RemoteClientIp {
right_most_x_forwarded_for: None,
connection_info_ip: Some(client_ip)
}
}
);
}
...
} This function does not even exist in |
That was removed here. It was added, but them removed later. |
4ad9815 refactor: remove duplicate code (Jose Celano) 0097c85 fix: clippy warnings (Jose Celano) cf9e9a9 fix: merge conflicts (Jose Celano) a611fbd chore: fix clippy errors (Warm Beer) d020c5a refactor: `tracker_api` launching and testing (Warm Beer) 5b95b5d refactor: renamed `tracker_interface` to `server` and shortened `configuration::ephemeral` calls (Warm Beer) fac2be8 fix: all http tracker tests (Warm Beer) d914e5c refactor: replaced test http servers with the new test environments (Warm Beer) 4f2b035 refactor: renamed `TrackerInterface` struct and relevant trait (Warm Beer) f40e43f refactor: prepend paths to ephemeral calls (Warm Beer) 191fbac refactor: moved signals mod to own file (Warm Beer) 504bad3 refactor: abstract away http server implementations (Warm Beer) a72c123 refactor: removed duplicate ephemeral configuration fn (Warm Beer) 164f29a isolated configuration, test-helpers and primitives crates (Warm Beer) Pull request description: Hi @WarmBeer, I've rebased your [PR](#170), fixing all merge conflicts. Do you want to rewrite the history of your changes or can we merge it as it's is? Now you can rewrite your commits if you want to, with the following: ```s git reset develop git add -A git commit -m "isolated configuration, test-helpers and primitives crates" git push --force ``` As you mentioned [here](#170 (comment)). ACKs for top commit: josecelano: ACK 4ad9815 Tree-SHA512: 64083f697bb24d2c0037c332555f6eaad46adf675f937f473ce0e2bd874a49e1d3581f4aecc73a131f5db30456248b369a0e1e1ef1f35b4024b131adfe557994
You can't share test functionality between unit tests or unit tests <-> integration tests, without defining this functionality in production code. Creating a "test-helpers" dev-dependency will solve that.
This PR solves:
Prerequisites
Sometimes we want to call production code from the test-helpers dependency. This will however create a cyclic dependency issue with our current code (test-helpers <-> torrust-tracker). Instead of these two crates depending on each other, we should instead make them dependent on the same sub-crates. So we must isolate code in their own crates as much as possible.
For example the tracker configuration code. This can be easily moved to its own crate. Then we would be able to import it using the test-helpers crate and the torrust-tracker crate without creating a cyclic dependency issue.