Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Do wheel installs from VCS/directories using ephemeral caching #4764

Merged
merged 36 commits into from
Jan 23, 2018

Conversation

pradyunsg
Copy link
Member

@pradyunsg pradyunsg commented Oct 4, 2017

Closes #4501
Closes #4562
Closes #4714
Closes #536

Quoting from #4501:

Follow up from gh-4144

To allow build system abstractions, we want installation to go through wheels in more cases. In particular, installing packages from a local directory or a VCS URL currently runs 'setup.py install'. The aim of this PR is to have it build a wheel, which is stored in an ephemeral cache directory, used for installation, and then discarded.

We can't cache it permanently based on the path/URL, because the code there might change, but our cache wouldn't be invalidated.

This is implemented by reusing the current WheelCache and wrapping what is basically 2 instances of it in a wrapper that implements everything a Cache should, with an extra helper for direct access to the ephemeral directory when caching wheels.


I'm building on top of some awesome work by @takluyver, @alex and @xoviat.

takluyver and others added 23 commits July 5, 2017 20:04
Follow up from pypagh-4144

To allow build system abstractions, we want installation to go through
wheels in more cases. In particular, installing packages from a local
directory or a VCS URL currently runs 'setup.py install'. The aim of
this PR is to have it build a wheel, which is stored in an ephemeral
cache directory, used for installation, and then discarded.

We can't cache it permanently based on the path/URL, because the code
there might change, but our cache wouldn't be invalidated.
When there's a previous build directory, no_clean is set
@pradyunsg pradyunsg self-assigned this Oct 4, 2017
@pradyunsg pradyunsg requested a review from a team October 4, 2017 13:03
@pradyunsg pradyunsg added type: enhancement Improvements to functionality C: cache Dealing with cache and files in it labels Oct 4, 2017
@@ -102,13 +103,18 @@ def _link_for_candidate(self, link, candidate):

return index.Link(path_to_url(path))

def cleanup(self):
Copy link
Member Author

@pradyunsg pradyunsg Oct 4, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I should note that @xoviat had proposed in #4714 to make Caches into context managers, to avoid using this cleanup() pattern.

I'm still confused about how I feel about that. FWIW, we can do that in a later PR. I have some other ideas too but I'd prefer to experiment with that after this PR is done.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@rhettinger What do you think about using context managers for cleanup? You're the person that I got this from.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just to be clear, I definitely think using a context manager is a good idea. But I agree with @pradyunsg that it's something we should defer until a followup PR. Let's get the functionality landed then address the cleanup.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree with that; I was just wondering what others thought in principle.

@pradyunsg
Copy link
Member Author

For some reason, my newer commits aren't showing up on GitHub's UI. :/

@pradyunsg pradyunsg mentioned this pull request Oct 24, 2017
@pradyunsg
Copy link
Member Author

@pypa/pip-committers This PR does not depend on the current PEP 518 implementation; it's something we should probably be doing anyway.

Can we merge this?

@pfmoore
Copy link
Member

pfmoore commented Nov 14, 2017

+1 from me

@BrownTruck
Copy link
Contributor

Hello!

I am an automated bot and I have noticed that this pull request is not currently able to be merged. If you are able to either merge the master branch into this pull request or rebase this pull request against master then it will eligible for code review and hopefully merging!

@BrownTruck BrownTruck added the needs rebase or merge PR has conflicts with current master label Nov 16, 2017
@pypa-bot pypa-bot removed the needs rebase or merge PR has conflicts with current master label Nov 17, 2017
@pradyunsg pradyunsg mentioned this pull request Dec 1, 2017
@pradyunsg
Copy link
Member Author

Okay. This should make this PR mergeable.

@pypa/pip-committers Do we want to go ahead with this, regardless of PEP 517/518?

@pfmoore
Copy link
Member

pfmoore commented Dec 1, 2017

+1 from me. At this point, I'm not sure if we'll need to do a pip 10 release without PEP 517/518, but I'd rather this be merged just in case we decide to.

@pradyunsg
Copy link
Member Author

pip 10 release without PEP 517/518

@pfmoore You think it's a good idea to discuss this on #4802?

@pradyunsg
Copy link
Member Author

@pypa/pip-committers Ping! :)

I still think this change can go in on it's own. Is there something we want to do before merging this?


Aside: When merging this PR, please use plain merge instead of squash/rebase merges since I have other branches sitting on top of this one.

@pfmoore pfmoore merged commit 81fb515 into pypa:master Jan 23, 2018
@pfmoore
Copy link
Member

pfmoore commented Jan 23, 2018

OK, merged - hope I got the right type of merge for you.

@pradyunsg
Copy link
Member Author

Yeps. Thanks! ^>^

@pradyunsg pradyunsg deleted the cache/ephem-wheel-cache branch January 24, 2018 11:37
@pradyunsg pradyunsg removed the !release blocker Hold a release until this is resolved label Jul 15, 2018
@lock
Copy link

lock bot commented Jun 2, 2019

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

@lock lock bot added the auto-locked Outdated issues that have been locked by automation label Jun 2, 2019
@lock lock bot locked as resolved and limited conversation to collaborators Jun 2, 2019
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
auto-locked Outdated issues that have been locked by automation C: cache Dealing with cache and files in it type: enhancement Improvements to functionality
Projects
None yet
Development

Successfully merging this pull request may close these issues.

pip should reinstall .
8 participants