Clean app state after running an E2E test?

[josecelano]

Hi, we have been discussing whether it is a good idea or not to clean the stage changes made by a test.

For example, if you have an E2E test for user registration, you can leave a newly created user or delete it after running the test.

We are using random values for the tests, so they are isolated, and you can run them many times. For example, the test to sign up a new user always uses new random data for the user. If the test fails and you rerun it, you will not have a false negative because the user already exists.

I was discussing this topic with @Wolfremium13 on different PRs. We agreed that cleaning the state after running the test was a good practice. If you need to debug why a test is failing, you can comment the code cleaning the state after running the test.

@mario-nt mentioned that cleaning the state after running a test is actually considered a bad practice on the Cypress documentation: https://docs.cypress.io/guides/references/best-practices#Using-after-Or-afterEach-Hooks

I would differentiate the two environments we usually use for running tests:

• Totally isolated: we create an independent clean database (SQLite) for each test. We prefer this one when it's possible because we can mock dependencies and create all the dependencies.
• Shared env. We run the real app with docker compose shared by the test suite. All tests are being executed against the same env.

I think this discussion only applies to the second one, which is the one we use for E2E testing.

I just want to open the discussion to track pros and cons. I personally think it's better to clean the state (as much as you can). It does not matter if sometimes a test fails and the state is not clean in that case. In fact, in that case, you would prefer to rerun the test without cleaning the state to check what happened.

Links

• An example of cleaning after running the test: E2E test: torrent download #222

cc @da2ce7 @mario-nt

[mario-nt]

Following Cypress's best practices, app state clean up should be done before running any tests, as the after() and afterEach() hooks are not executed if any step of the tests fails.

I would create a resetDatabase() or cleanDatabaseData() and a create/loginAdminUser()/loginGuestUser() commands before every test, that way, we reset the app state between tests.

In case we want to debug the tests when they fail, we can run it again with those commands commented.

I see two ways of implementing this:

1. Writing that logic in the before() or beforeEach() hooks in every test file
2. Doing the same but in the E2E support file

I like the second option, as it removes the need to write the same code in every single test, but, at the same time, it hinders readability.

Overall I think cleaning the app state after every test is the right thing to do regardless of the way it is implemented.

[josecelano]

Hi @mario-nt I suppose the resetDatabase() function resets the whole database state, not only the state changed by the test you want to execute.

What about writing a resetDatabase() specific for each test in the before() hook?

I like the idea of resetting the whole database because we ensure the test knows its dependencies. But I do not like that we cannot run tests in parallel if we are using a shared testing environment. Besides, resetting the whole database could be slow.

On the other hand, I do like coupling tests to the DB implementation either. In order to clean only changes made by the test, the test must know what changes are produced in the database; therefore the test must know the DB implementation.

Regarding running test in parallel it seem with CYpress you need to add more environments: https://docs.cypress.io/guides/cloud/smart-orchestration/parallelization

Conclusion

• I would use the "support file" to reset the DB. It should not be slow at this point. Since we are using SQLite we could just delete the file and create a new one instead of running queries. WE could even have a template SQLite file with the DB migrations executed, so we don't have to run them before each test.
• This would decouple tests from DB implementation.
• We have to remove the code to clean the database after running the test.

[mario-nt]

I agree that if we run tests in parallel deleting all the data from the database would lead to errors.

As we are currently only using SQLite for tests and running them sequentially I think for the time being, we can just delete the database file in the support file as you said. Even though this might not be a valid solution in the future.

In the future, we can think about implementing commands for deleting the database data related to each test (in all the different implementations SQLite, MySQL,...) or running tests in parallel.