RFC: Introduce setup, teardown, and fixture attributes for cargo test

[wiseaidev]

Problem

The current testing framework in Rust, as implemented by cargo test, lacks a convenient way to define and manage setup teardown, and fixtures objects for tests. While it is possible to implement such functionality using structs, implementing the Drop trait in case of teardown, and such, this approach can be cumbersome and lacks the expressiveness and convenience of more declarative fixture management.

The absence of a dedicated mechanism for defining fixtures and their lifecycles results in the following issues:

1. Test authors often need to implement repetitive and error-prone setup teardown code and common structs instance for fixtures, making tests less maintainable.
2. Complex test setups, teardowns, and resource management require a non-trivial amount of boilerplate code.
3. There is no standard way to share setup/teardown, and fixtures across multiple tests or test modules, potentially leading to duplication of setup/teardown code and fixtures.

Proposed Solution

This RFC proposes the introduction of three new attributes #[setup], #[teardown], and #[fixture] which can be used in Rust test functions to indicate functions that should serve as setup/teardown functions, and fixture objects respectively. Test authors can define setup functions/teardown functions, and common objects with these attributes, and the testing framework will ensure that they are executed appropriately before and after test functions that use them. For instance:

use std::fs;
use tempfile::TempDir;

#[fixture]
fn temp_dir() -> TempDir {
    let temp_dir = TempDir::new().expect("Failed to create temporary directory");
    temp_dir
}

#[setup]
fn setup() {
   // Setup before tests
}

#[teardown]
fn teardown() {
   // Cleanup after tests 
}

#[test(setup, teardown)] // The usage of these functions is debatable
fn test_with_temp_dir(temp_dir) { // fixtures are passed to the test function as args
    // Test logic with temp_dir fixture arg
    // ...
}

With these attributes, test authors can set up necessary fixtures or resources in a clear and standardized manner before the test execution and ensure proper cleanup afterward. This feature will streamline the development of tests, particularly in cases where complex setup and teardown operations are required.

Notes

Introducing setup, teardown, and fixture attributes in Cargo Test will improve the test development experience by simplifying resource management, enhancing test predictability, and enabling developers to write more complex tests with ease. This RFC aligns with the goal of making Rust testing more ergonomic and developer-friendly.

P.S. Just stumbled upon rstest which provides fixtures, but lacks a setup/teardown mechanism.

[epage]

FYI this is owned by libtest which is in the rust repo, and not cargo. We've also recently formed a testing-devex team to focus on improving the workflows around tests in a cross-team way.

As this is a fairly fundamental change, I would recommend experimenting in a library first for these ideas (pytest-rs is where my experiments live) and then bringing that proposal forward on internals.

However, this is unlikely to happen, directly. Last I talked to the libs team, we walked away with the idea that we'd work to make harness=false harnesses first class citizens and focus on feature development there, rather than in libtest. pytest-rs is my experiment towards that goal.

For myself, I see setup and teardown to be redundant with fixture and anti-patterns as they tend towards "god fixtuures" that encompass too much and slow down tests. The proposal given also doesn't provide a way to pass state from setup to the test to teardown and doesn't cover process-wide vs test-wide setup/teardown.

[wiseaidev]

Thank you for your response, @epage. I appreciate your input, and I'll definitely take a closer look at the repository you mentioned.

Regarding the use of these attributes, I left a comment in the code, specifically, "// The usage of these functions is debatable." I haven't fully explored the usage and implementation details yet, but I thought it was essential to introduce this idea in case it hasn't been discussed before, especially concerning the simplification of unit tests.

Here's a more detailed suggestion I'd like to propose:

1. If the setup and teardown functions are not utilized by any function within a module, the default assumption would be that they are meant to be applied across the entire module. In practical terms, this implies that every unit test within the module would execute these functions both before and after running their respective code.

// module-based setup teardown.

#[setup]
fn setup_fn() {
   // Setup before tests
}

#[teardown]
fn teardown_fn() {
   // Cleanup after tests 
}

#[test]
fn test_one() {
    // Test logic
}

#[test]
fn test_two() {
    // Test logic
}

#[test]
fn test_three() {
    // Test logic
}

2. If the setup and teardown functions are specifically called by one or more functions designated with the #[test] attribute, it signifies that only these particular tests should trigger these functions. This adjustment could potentially introduce a breaking change to the test attribute since it now accepts a parameter, which specifies the setup and teardown functions. Furthermore, this change may pave the way for additional attributes to be included in the future.

// test-based setup teardown.

#[setup]
fn setup_fn() {
   // Setup before tests
}

#[teardown]
fn teardown_fn() {
   // Cleanup after tests 
}

#[test(setup_fn, teardown_fn)] // run setup/teardown for this test only
fn test_one() {
    // Test logic
}

#[test]
fn test_two() {
    // Test logic
}

#[test]
fn test_three() {
    // Test logic
}

In essence, this proposal aims to enhance the flexibility and clarity of how setup and teardown functions are employed in unit tests within your codebase. It encourages more precise control over when these functions are invoked, making the testing process more adaptable and robust.

[nyabinary]

Thank you for your response, @epage. I appreciate your input, and I'll definitely take a closer look at the repository you mentioned.

Regarding the use of these attributes, I left a comment in the code, specifically, "// The usage of these functions is debatable." I haven't fully explored the usage and implementation details yet, but I thought it was essential to introduce this idea in case it hasn't been discussed before, especially concerning the simplification of unit tests.

Here's a more detailed suggestion I'd like to propose:

1. If the `setup` and `teardown` functions are not utilized by any function within a module, the default assumption would be that they are meant to be applied across the entire module. In practical terms, this implies that every unit test within the module would execute these functions both before and after running their respective code.

// module-based setup teardown.

#[setup]
fn setup_fn() {
   // Setup before tests
}

#[teardown]
fn teardown_fn() {
   // Cleanup after tests 
}

#[test]
fn test_one() {
    // Test logic
}

#[test]
fn test_two() {
    // Test logic
}

#[test]
fn test_three() {
    // Test logic
}

2. If the `setup` and `teardown` functions are specifically called by one or more functions designated with the `#[test]` attribute, it signifies that only these particular tests should trigger these functions. This adjustment could potentially introduce a breaking change to the `test` attribute since it now accepts a parameter, which specifies the setup and teardown functions. Furthermore, this change may pave the way for additional attributes to be included in the future.

// test-based setup teardown.

#[setup]
fn setup_fn() {
   // Setup before tests
}

#[teardown]
fn teardown_fn() {
   // Cleanup after tests 
}

#[test(setup_fn, teardown_fn)] // run setup/teardown for this test only
fn test_one() {
    // Test logic
}

#[test]
fn test_two() {
    // Test logic
}

#[test]
fn test_three() {
    // Test logic
}

In essence, this proposal aims to enhance the flexibility and clarity of how setup and teardown functions are employed in unit tests within your codebase. It encourages more precise control over when these functions are invoked, making the testing process more adaptable and robust.

goofy ass ai written response 😭

[zaddok]

We definitely need something like this for testing, so I really like where you are going with this, but I am not sure about this specifically.

Different tests might have different setup requirements, but more importantly often the point of setup is to create some variables, such as a database connection, so I am not sure how useful a setup function is if it won't be able to return anything. I feel like a setup function that cannot return a variable would be annoying, it'd leave me wondering what the point of it is if I cant use it for my normal setup functionality.

[dead-claudia]

Related:

• Need way to run some code before any of unit tests #35762
• Allow crate to attach uniform initialization behavior to every #[test] rfcs#1664
• https://rust-lang.github.io/rfcs/2318-custom-test-frameworks.html (PR: eRFC: Custom test frameworks rfcs#2318, tracking issue: Tracking issue for eRFC 2318, Custom test frameworks #50297)