-
-
Notifications
You must be signed in to change notification settings - Fork 515
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
tox4: plugin migration use cases #1974
Comments
We have https://pypi.org/project/tox-current-env/ and I'd happily discuss the use case with you. If new tox could be used as a library, we would be able to drop the plugin entirely and just ask tox for the data we need. |
Please do describe what functionality you would like to be exposed to, and why would it be better than a plugin? Thanks! |
Will do tomorrow. EDIT: Or the day after. |
I had to do some juggling for my At that time, @gaborbernat had some insights into the improvements needed in a future tox 4. What is happening now. |
Can you clarify what do you mean by defer here? |
Any chance unify the interface of install deps and install pkgs so that we can change the behavior in one hook. |
Definitely not, these two sections have totally different formats and contracts. It just happens pip can handle both using the same interface, but that's not generally true. If you're using an installer that can handle it the same way you can still define two hooks that calls the same function. |
tox-current-envWhat do we wantWe want a way to run upstream tests when we build RPM packages for Fedora. We want a standardized way to get a list of test dependencies (by we install them ourselves) and a standardized way to get commands to execute (or a standardized way to execute them in a Python environment we choose). Why do we use toxUnfortunately, there is no standard. We've been told that tox is the de-facto standard and that any future standard (if any) is likely to evolve from tox. I don't recall where is this information coming from, but it makes sense to us. What does our plugin doWhen certain command line switch is used, it redefines the virtualenv creation (and fakes it with symbolic links to Why would a library be betterIf there was a library, we could "ask" tox: What are the dependencies for this toxenv? What are the commands for this toxenv? (Or alternatively: Execute the commands via subprocess.) I think that this approach is better, because it:
Ideally this library would not even require virtuelnv, but I understand that this would require packaging it separately and that is a lot to ask. |
@hroncok for your use case my recommended solution would be to replace tox environment creation with a no op that just serves to tox the host python, translate pypi dependencies to RPM dependencies, and use apt to install them. All this you could do in a plugin. This would be the recommended way, and safer IMHO. Now if you'd insist on rolling your own pipeline to get the raw dependencies and commands you can use |
I thought that we need an actual environment, so we create a fake one. I can have look if this can be simplified.
Due to technical limitations, we cannot install stuff directly, we need to echo it on stdout and exit. That is what we do, more or less, via the plugin (except that we do the translation via a wrapper).
OK. We'll keep doing it that way then.
I'll check it out. |
In a tox 4 world, you'd just need to implement the abstract parts of https://github.com/tox-dev/tox/blob/rewrite/src/tox/tox_env/python/api.py#L73, by default we use a virtualenv+pip based https://github.com/tox-dev/tox/blob/rewrite/src/tox/tox_env/python/virtual_env/api.py#L28, but you're free to replace those parts with host python + dnf. Note the API is not stable yet 👍🏻 so don't jump on this bandwagon just yet, but this is the midterm plan. |
Hi, haven't looked at tox-current-env in detail yet, but it sounds a lot like the plugin, I wrote to facilitate running tests and tasks via tox inside a container that already contains a prepared venv or needs packages installed in that prepared venv: https://pypi.org/project/tox-direct/ circumvents the venv creation and uses the env that tox was started from. I am just getting acquainted with the rewrite after a long time off from the whole project, but if I understand correctly one reason for the rewrite is to be able to accommodate different run time environments directly, including docker containers, so it would be interesting to find out what and how much of plugins like tox-direct and tox-current-env is actually still needed in tox4. |
They're still needed. You want to be able to plug in the system python as the tox python env. The subject is not how to skip X, but more like how do we integrate X to work with tox. The expectation here is that users will implement abstract base classes, such as https://github.com/tox-dev/tox/blob/rewrite/src/tox/tox_env/python/runner.py#L22; and use that as the runner class. |
A POC of tox-conda has been now done under https://github.com/tox-dev/tox-conda/tree/main |
Hi, I've updated tox-pdm to support both tox 3 and 4. It turns out not easier than v3(232 lines v.s. 157 lines). I don't know if I am missing any best practices. |
Thanks. I don't think it would be easier (well definitely not code line wise), but in tox 4 you can do more with fewer hacks. Would have been nice if you opened a PR on the new plugin so I can add inline comments to the relevant parts. |
Yes, the v3 plugin is really HACKY. There are two main points that make me feel uncomfortable:
If you are interested, here is the PR that you can leave comments on: https://github.com/pdm-project/tox-pdm/pull/11/files |
Can't be helped sadly 🤔
Yeah, this is a trade-off for allowing easy override of things, while making explicit what needs to be implemented (via abstractmethods). IDEs are generally really good at saying what's missing, they likely use the same attribute though. |
Clarification for users, tox-gh-actions supports tox v4: https://github.com/ymyzk/tox-gh-actions/tree/tox4. @gaborbernat made a new implementation at https://github.com/tox-dev/tox-gh. |
Happy to report, tox-docker now works with 4.0.0a9. I'm happy to cut a release once you feel the APIs are stable and unlikely to change any more @gaborbernat, just let me know. |
I'm trying to rewrite
I think that the second option is better and something like that might be useful also for other plugins working in the way "do nothing unless explicitly activated by an CLI option". What do you think @gaborbernat ? One more question: Do I understand it correctly that the |
I'm not sure I understand all your questions, but few major points:
Yes, but I don't think you want to do changes there in any shape or form. Can discuss more interactively over https://discord.gg/tox if you want. |
Thanks for that but the main question remains open. It is possible to do something like this: @impl
def tox_register_tox_env(register):
if option.current_env or option.print_deps_to or option.print_extras_to:
register.add_run_env(CurrentEnvRunner) Are options available in any form during the registration of environments? |
Not at the moment, there was no use case yet needing the CLI arguments for env registration 🤔 Why would you want to keep the registration optional? |
Because our plugin can be enabled via CLI arguments only (no configuration options), which means that if you run just |
Registering the environment runner does not imply activating it. It should be safe to register it and only active it based on CLI flags within https://tox.wiki/en/rewrite/plugins.html#tox.plugin.spec.tox_add_core_config. If registering the environment has side-effect you might need to rethink your design of the plugin. Can you share the code you have, I'm happy to chime in via a review on it, to guide the best path ahead. |
Thanks to your help, @gaborbernat , I was able to create this: fedora-python/tox-current-env#42 Could you please take a look and tell me if I'm doing something completely wrong in the implementation? It seems to work even some parts look kinda hacky but the whole plugin seems to be a corner case. |
The functionality got moved into tox core starting with tox 4.0. tox-dev/tox#1974
Hello 👋 I have been trying to update the pypi-filter plugin for tox 4. The plugin runs a PyPI proxy server as a subprocess and then used to set the indexserver option so that requests to PyPI went through the proxy. (The proxy can filter out versions of any packages being installed, including transitive deps etc). I have mostly managed to convert it to tox 4 but the issue I have is there doesn't seem to be a hook at the right point in the cycle for me to use. Using I looked around at overriding the virtualenv builder to achieve what I wanted but my head started spinning pretty quick, so I thought I would ask for input on how best to approach the goals of this plugin in the tox 4 land. Thanks! |
Ideally, when do you think this plugin should start/stop? |
I would like it to start before the 99.9% of the time it wouldn't matter if it was running for the package build step, it's goal isn't really to fix issues in building the package. |
Those phases are env phases, so do you want to run one instance per env then? |
Yes, that's what we were doing in tox3 land. |
So my recommendation is to:
|
@Cadair https://tox.wiki/en/latest/changelog.html#v4-0-9-2022-12-13 has now all you need |
Good morning everyone I was unhappy with the fact that tox-backticks has not been ported to tox4 yet, and also unhappy by some bugs (curly braces escaping does not work) and/or the behaviour of this plugin (trailing newlines not being stripped...), so I decided to start my own plugin. I've been learning the tox internals for the past few days, and produced a very early-stage tox-backtocks plugin. This is a work in progress but it currently works with some basic cases, and, if you can believe me, I intend to do something robust, useful and user-friendly. I'm gonna start testing it and using it on the 100+ projects we have at work that currently use tox-backticks, in order to finally being able to upgrade to tox>=4. If it's not too much to ask, I would appreciate some feedbacks on the code (I'm using some protected attributes of some tox objects... and did not find any other way to achieve my goal) and also if any tox-backticks user is reading me, feel free to report any failure and/or misbehaviour. Thank you. |
Would be easier if you'd have a PR we can open. You are accessing here a private attribute that's probably not a good idea https://github.com/Anvil/tox-backtocks/blob/main/tox_backtocks/__init__.py#L54. I'll close this issue, feel free to open a new discussion (rather than issue for feedback). |
This issue is used to mark the migration plans for various tox plugins. Shows which plugins and use cases we considered when defining the plugin interface for tox 4.
Need to fork
POC done
Done
Moved to core
The text was updated successfully, but these errors were encountered: