Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

use git subtree to graft Pkg3, etc. into stdlib #25942

Closed
wants to merge 974 commits into from
Closed

use git subtree to graft Pkg3, etc. into stdlib #25942

wants to merge 974 commits into from

Conversation

StefanKarpinski
Copy link
Sponsor Member

This was astonishingly easy to do with git subtree. It really might be a great solution for developing stdlib packages outside of the main repo but having the code available without any additional futzing upon checkout of julia. On this branch, you can do using Pkg3 from the julia prompt and it works:

julia> using Pkg3
[ Info: Precompiling module Pkg3

pkg> init
[ Info: Initialized environment in /Users/stefan/projects/julia by creating the file Project.toml

pkg> add JSON
[ Info: Resolving package versions
[ Info: Updating "Project.toml"
 [682c06a0] + JSON v0.16.4
[ Info: Updating "Manifest.toml"
 [34da2185] + Compat v0.49.0
 [682c06a0] + JSON v0.16.4
 [4d1e1d77] + Nullables v0.0.3

Resulting project and manifest files.

Project.toml:

[deps]
JSON = "682c06a0-de6a-54ab-a142-c8b1cf79cde6"

Manifest.toml

[[Compat]]
git-tree-sha1 = "e74e175d5b9c7411f17d7db90ccdfdf492bde9e9"
uuid = "34da2185-b29b-5c13-b0c7-acf172513d20"
version = "0.49.0"

[[JSON]]
deps = ["Compat", "Nullables"]
git-tree-sha1 = "e4b57d438f523a2ad5f9f3510598f00961d4bc08"
uuid = "682c06a0-de6a-54ab-a142-c8b1cf79cde6"
version = "0.16.4"

[[Nullables]]
deps = ["Compat"]
git-tree-sha1 = "05176cebbb49ab979006b0bd12008c3f9a54f60c"
uuid = "4d1e1d77-625e-5b40-9113-a560ec7a8ecd"
version = "0.0.3"

Upshot

julia> using JSON
[ Info: Precompiling module JSON
ERROR: LoadError: LoadError: ArgumentError: Module Unicode not found in current path.
Run `Pkg.add("Unicode")` to install the Unicode package.

Ok, so there are some rough edges to smooth out, but this finally basically works.

StefanKarpinski and others added 30 commits October 17, 2017 15:01
@StefanKarpinski
toposort builds in dependency order (fails if not a DAG)
ae5a17b
character and string concatenation
55c211c
@StefanKarpinski
fix bug in using project file path as env
a2f800b
@StefanKarpinski
simplfy build processing code
f8a8248
@StefanKarpinski
only build after writing out new environment
28c803e 
otherwise the build process doesn't pick up correct versions
@StefanKarpinski
fix the PyCall build issue, use 16-byte slugs for version dirs
d9f70b8 
The miniconda installer creates the correct shebang line no problem
but conda update encounters long shebang lines and decides that they
are dangerous, so it silently replaces a "long" (≥128 bytes) path to
`python` with `/usr/bin/env python`. What could go wrong?!? This works
around that behavior by using a 16-byte "slug" in package/version
paths instead of using the full "$uuid/$sha1" string, which, to be
fair, is an excessive 77 characters in the middle of paths.

The issue was that Conda looks for long shebang lines (>127) and
silently replaces the or
@StefanKarpinski
qualify call to Base.info since info is a local variable
6e32ff7
@TotalVerb
Merge pull request #406 from chakravala/pull-request/55c211cb
a75cf4f 
character and string concatenation
@Keno @staticfloat
Drop Compat dependency (#42)
aba4a66
@StefanKarpinski
use mod 62 arithmetic instead of SHA2-224 for slug generation
718182c 
otherwise the SHA package would have to be internalized
@StefanKarpinski
always print log file path when building
e229047 
when the build takes a while, you can tail the file to see what's up
@StefanKarpinski
up in manifest mode, default to all manifest packages, not just pro…
853ca08 
…ject packages
@StefanKarpinski
implement a basic help system
1c96bcb
@StefanKarpinski
tweak log file printing; help text
468bfbe
@StefanKarpinski
nicer printing of multiple help topics; allow ?add up w/o space
d6aef97
@StefanKarpinski
Empty LOAD_PATH on load and add Pkg3 dir
f6d6de5 
This allows Pkg3 loading to work even if it's loaded with `include`
instead of by being on the `LOAD_PATH` and running `using Pkg3`.
@StefanKarpinski
actually fix loading issues (#9)
4596b54
@StefanKarpinski
avoid creating Project.toml without added dependencies
81291ae
@staticfloat
Much smarter file wrapper that loads in files one chunk at a time
79bcc3f
@ararslan
Add BitSet (#407)
5bd5e8c
@StefanKarpinski
bin/update.jl: run other scripts with correct LOAD_PATH
5b1f812
@ylxdzsw @staticfloat
add HMAC functionalities (#45)
43b2bad
@maleadt
Support more forms of at-nospecialize.
775d9ee
@maleadt
Avoid method override warnings.
c0edbd7
@maleadt
Merge pull request #409 from JuliaLang/tb/nospecialize
fbd9cb2 
Support more forms of at-nospecialize.
@KristofferC
remove precompile from internalized toml
7d4badf
@StefanKarpinski
Merge pull request #20 from KristofferC/patch-1
f210028 
remove precompile from internalized toml
@KristofferC
internalize BinaryProvider
2550b81
@SimonDanisch
no precompilation remark, add pkg3 clone, pkg dir
5483b6a 
add information I was missing when installing and remove remark about precompilation, which doesn't seem to be true anymore.
@StefanKarpinski
Merge pull request #26 from SimonDanisch/patch-1
13ebc13 
no precompilation remark, add pkg3 clone, pkg dir
ararslan and others added 9 commits February 5, 2018 19:13
@ararslan
Merge pull request #484 from JuliaLang/nl/search
b2aac2e 
Add occursin and find* methods replacing search and findin
@nalimilan @ararslan
Add size(::LinearIndices) (#455)
8cca897 
* Add size(::LinearIndices)

Fixes indexing LinearIndices with an array. This is useful in particular
to convert cartesian indices that find() now returns to linear indices.

* Add size(::LinearIndices) on older 0.7 versions
@StefanKarpinski
changes to work with new loading stuff + more deprecation fixes
3c0c9e2
@StefanKarpinski
add Pkg3 Project.toml file
2958883
@StefanKarpinski
add Pkg3 Project.toml file
1507917
@StefanKarpinski
Merge pull request #488 from JuliaLang/sk/project-file
95db156 
add Pkg3 Project.toml file
@StefanKarpinski
Add 'stdlib/Compat/' from commit '95db156e9adf11adf400caf7c3db29ea025…
331bf19 
…a389a'

git-subtree-dir: stdlib/Compat
git-subtree-mainline: 2017524
git-subtree-split: 95db156
@StefanKarpinski
Add 'stdlib/SHA/' from commit '2958883111f7ddc37b485a7f91af18be4ab84434'
278ede2 
git-subtree-dir: stdlib/SHA
git-subtree-mainline: 331bf19
git-subtree-split: 2958883
@StefanKarpinski
Add 'stdlib/Pkg3/' from commit '3c0c9e2834b5a7a1c0f4fa04ae25f61cc4ba8…
588e9a4 
…483'

git-subtree-dir: stdlib/Pkg3
git-subtree-mainline: 278ede2
git-subtree-split: 3c0c9e2
@vchuravy
Copy link
Member

vchuravy commented Feb 8, 2018 via email

@StefanKarpinski
Copy link
Sponsor Member Author

Yes, the things in Pkg3/ext should become proper stdlib modules, but I wasn't quite ready to add those right now – they are at the moment, implementation details of Pkg3. I'm not 100% sold on git subtree, but so far it's been a pleasure to use. The main advantages seem to be:

  1. See history of stdlib packages in the same repo as Base.
  2. Develop stdlib packages together with Base, upstream the changes when they're ready.

@StefanKarpinski StefanKarpinski changed the title use git subtree to graft Pkg3, SHA and Compat into stdlib use git subtree to graft Pkg3, etc. into stdlib Feb 8, 2018
@StefanKarpinski StefanKarpinski added this to the 1.0 milestone Feb 8, 2018
@StefanKarpinski StefanKarpinski added the status:triage This should be discussed on a triage call label Feb 8, 2018
@StefanKarpinski
Copy link
Sponsor Member Author

Milestoning, not because we have to do it this way but because we have to do something like this.

@o314
Copy link
Contributor

o314 commented Feb 8, 2018

Drupal followed a similar way a few month ago

Subtree splits are now being automated by drupal infrastructure. We have a jenkins job that is polling git.drupal.org/project/drupal.git every minute and runs a script that traverses a git checkout of drupal, looking for composer.json files. If it discovers a composer.json (except in test directories), it will:

  • create a subtree split of that directory
  • parse composer.json
  • If it doesnt exist, it will create a new repo at github.com/drupal/{composer-project-name}
  • It pushes the subtree split to github.com/drupal/{composer-project-name}
  • If it doesnt exist, it will create a new entry at packagist.org
  • If the packagist entry does exist, it is instructed to update its entry from github.

So new components or new composer.json parts that we wish to expose outside the drupal ecosystem should automatically appear on packagist.

@ https://www.drupal.org/project/infrastructure/issues/2352091#comment-12056883

The symfony framework (~ new drupal kernel, and more) guys gave a talk about this at drupalcon 2016 : a monorepo vs manyrepos (slides), (video), and (code)

@StefanKarpinski
Copy link
Sponsor Member Author

Triage dixit: no Compat in stdlib, so excise that dependency. Pkg3 + SHA in base is fine, but favors @vchuravy's approach to use @vtjnash's "git external" mechanism. The TerminalMenus is a different story since it should be merged into REPL; for that, using subtree to import and do a temporary bridge between internal and external is fine.

@StefanKarpinski StefanKarpinski removed the status:triage This should be discussed on a triage call label Feb 8, 2018
@JeffBezanson
Copy link
Sponsor Member

In triage several of us prefer @vchuravy 's approach (#25714). I think that will make it clearer that the packages are separate, and owned by different repos. There's something to be said for being able to update Base and stdlibs all at once, but we're already used to having many packages in external repos that might need to be updated in sync with Base.

@KristofferC
Copy link
Sponsor Member

KristofferC commented Feb 8, 2018 •
edited
Loading

That seems very(!!!) unfortunate in my view. Having stuff in the same repo tremendously ease development and coherence. All other stdlibs live in the repo. Why should Pkg3 be the only one getting punished!?

I think that will make it clearer that the packages are separate, and owned by different repos.

I am all for getting rid of the Pkg3 repo and just have it here if it would mean not having to do this. Just develop it here then.

but we're already used to having many packages in external repos that might need to be updated in sync with Base.

And this is typically very annoying and time-consuming. Run make docs now and see how well it works. Presumably, we want to run the Pkg3 test as part of Base CI (to make sure we don't break the whole ecosystem on master at once) which would make this even worse than the Documenter situation.

Also, multiple stdlib packages depend on Pkg right now. Are they going to be in this repo and depend on a repo located somewhere else?

@KristofferC
Copy link
Sponsor Member

KristofferC commented Feb 8, 2018 •
edited
Loading

I didn't manage to get to the triage call in time for this, but if the discussion was only between keeping the Pkg3 repo separate and using subtree to sync changes vs using git-external then my suggestion here is just to move Pkg3 to a stdlib and delete the old Pkg3 repo. Just like any other stdlib we currently have. Whenever we decide to use some other method for dealing with stdlibs, we do it for all at once. Using Pkg3 as the guinea pig for this is not a good choice regarding getting to 1.0 as fast as possible, in my opinion.

@JeffBezanson
Copy link
Sponsor Member

If Compat is out of the picture, then I'm much more ok with using git subtrees. I also wouldn't object to entirely moving the Pkg3 code here.

@KristofferC
Copy link
Sponsor Member

KristofferC commented Feb 9, 2018 •
edited
Loading

Yes, Compat is out of the picture.

Great, I can update this PR, with #25956 and #25953 in the picture.

@StefanKarpinski StefanKarpinski removed this from the 1.0 milestone Feb 9, 2018
@StefanKarpinski
Copy link
Sponsor Member Author

I'll close this then since @KristofferC's PRs supersede it.

@KristofferC
Copy link
Sponsor Member

I was going to update this one but can make a new one.

@StefanKarpinski
Copy link
Sponsor Member Author

Eh, just make a new one. There's too much junk in here.

@vtjnash
Copy link
Sponsor Member

vtjnash commented Feb 9, 2018 •
edited
Loading

Why should Pkg3 be the only one getting punished!?

Being frozen in stdlib is a punishment, being free to develop separately should be a positive.

Develop stdlib packages together with Base, upstream the changes when they're ready.

Regardless of where this code lives, I think we should encourage development to happen upstream, with separate release cycles, issue tracker, CI, etc., then just import the stable releases here.

@DilumAluthge DilumAluthge deleted the sk/pkg3 branch March 25, 2021 22:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.