Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Yet more spkg_configure for standard python packages #36276

Merged
merged 43 commits into from
Sep 27, 2023

Conversation

dimpase
Copy link
Member

@dimpase dimpase commented Sep 16, 2023

This is a continuation of #36256

📝 Checklist

  • The title is concise, informative, and self-explanatory.
  • The description explains in detail what this PR is about.
  • I have linked a relevant issue or discussion.
  • I have created tests covering the changes.
  • I have updated the documentation accordingly.

⌛ Dependencies

Also, a Jupyter/Python issue was uncovered there, which might need work.

@dimpase dimpase marked this pull request as draft September 16, 2023 13:30
@dimpase dimpase force-pushed the yet_more_spkg_conf_for_python branch from 83ea32e to 97ca5cb Compare September 16, 2023 15:40
@dimpase
Copy link
Member Author

dimpase commented Sep 16, 2023

PYDEVD_DISABLE_FILE_VALIDATION=1 is needed for ./sage -t src/sage/repl/ipython_kernel/kernel.py to pass, for some reason.

@dimpase
Copy link
Member Author

dimpase commented Sep 16, 2023

Tested on Debian oldstable with pyenv-installed Python 3.11.5. Configured as

./configure --enable-system-site-packages --with-python=$(pyenv root)/shims/python3

I had to deviate from version constraint on sphinx by pip-installing version 7.0.0,
and pip-installing setuptools version 63.4.3.

Output of pip freeze is attached.
pip_freeze_py3.11.5_with_pyenv.txt

orlitzky and others added 16 commits September 16, 2023 14:17
The pytz package is a dependency of three others: babel, rpy2, and
tzlocal. And tzlocal is itself a dependency of rpy2. Babel is already
disabled with --disable-doc, and rpy2 and tzlocal are disabled with
--disable-r. It therefore makes sense to disable pytz when both
--disable-doc and --disable-r are given.
The pycparser package is a dependency of two others, cffi and rpy2,
which can be disabled with --disable-notebook and --disable-r
respectively. As a result we can disable pycparser when both flags are
present.
The pytz_deprecation_shim package is a dependency only of tzlocal,
which is already disabled with --disable-r.
Passing --disable-notebook to ./configure already disables the
argon2_cffi package, so it should be safe to disable its bindings
as well.
The soupsieve package is a dependency of beautifulsoup4, which is
already disabled with --disable-notebook. It should therefore be safe
to disable soupsieve under the same circumstances.
The sphinx_basic_ng package is solely a dependency of furo, which is
already disabled with --disable-doc. We should therefore disable
sphinx_basic_ng with --disable-doc as well.
The fastjsonschema package is solely a dependency of nbformat, which
is already disabled with --disable-notebook. We should therefore
disable fastjsonschema with --disable-notebook as well.
@dimpase
Copy link
Member Author

dimpase commented Sep 16, 2023

@mkoeppe - I already mentioned this issue in #36256 - nbconvert could not find
templates. Judging from several /doesnotexist in the log here, are we poisoning the correct places? In this case it is $(pyenv root)/versions/3.11.5/share/jupyter/nbconvert/, but it does not come up on the list of what nbconvert calls root_dirs.

[sagemath_doc_html-none] 0.00s - Debugger warning: It seems that frozen modules are being used, which may
[sagemath_doc_html-none] 0.00s - make the debugger miss breakpoints. Please pass -Xfrozen_modules=off
[sagemath_doc_html-none] 0.00s - to python to disable frozen modules.
[sagemath_doc_html-none] 0.00s - Note: Debugging will proceed. Set PYDEVD_DISABLE_FILE_VALIDATION=1 to disable this validation.
[sagemath_doc_html-none] [plot3d   ] Exception occurred:
[sagemath_doc_html-none] [plot3d   ]   File "/home/scratch/scratch2/dimpase/sage/pyenv/versions/3.11.5/lib/python3.11/site-packages/nbconvert/exporters/templateexporter.py", line 653, in get_template_names
[sagemath_doc_html-none] [plot3d   ]     raise ValueError(msg)
[sagemath_doc_html-none] [plot3d   ] ValueError: No template sub-directory with name 'script' found in the following paths:
[sagemath_doc_html-none] [plot3d   ]    /doesnotexist
[sagemath_doc_html-none] [plot3d   ]    /home/scratch/scratch2/dimpase/sage/sage/local/var/lib/sage/venv-python3.11/share/jupyter
[sagemath_doc_html-none] [plot3d   ]    /doesnotexist
[sagemath_doc_html-none] [plot3d   ]    /home/scratch/dimpase/.sage///local/share/jupyter
[sagemath_doc_html-none] [plot3d   ]    /usr/local/share/jupyter
[sagemath_doc_html-none] [plot3d   ]    /usr/share/jupyter
[sagemath_doc_html-none] [plot3d   ] The full traceback has been saved in /tmp/sphinx-err-15y95ur8.log, if you want to report the issue to the developers.
[sagemath_doc_html-none] [plot3d   ] Please also report this if it was a user error, so that a better error message can be provided next time.
[sagemath_doc_html-none] [plot3d   ] A bug report can be filed in the tracker at <https://github.com/sphinx-doc/sphinx/issues>. Thanks!
[sagemath_doc_html-none] Error building the documentation.
[sagemath_doc_html-none] Traceback (most recent call last):
[sagemath_doc_html-none]   File "<frozen runpy>", line 198, in _run_module_as_main
[sagemath_doc_html-none]   File "<frozen runpy>", line 88, in _run_code
[sagemath_doc_html-none]   File "/home/scratch/scratch2/dimpase/sage/sage/pkgs/sage-docbuild/sage_docbuild/__main__.py", line 503, in <module>
[sagemath_doc_html-none]     sys.exit(main())
[sagemath_doc_html-none]              ^^^^^^
[sagemath_doc_html-none]   File "/home/scratch/scratch2/dimpase/sage/sage/pkgs/sage-docbuild/sage_docbuild/__main__.py", line 499, in main
[sagemath_doc_html-none]     builder()
[sagemath_doc_html-none]   File "/home/scratch/scratch2/dimpase/sage/sage/pkgs/sage-docbuild/sage_docbuild/builders.py", line 818, in _wrapper
[sagemath_doc_html-none]     getattr(DocBuilder, build_type)(self, *args, **kwds)
[sagemath_doc_html-none]   File "/home/scratch/scratch2/dimpase/sage/sage/pkgs/sage-docbuild/sage_docbuild/builders.py", line 162, in f
[sagemath_doc_html-none]     runsphinx()
[sagemath_doc_html-none]   File "/home/scratch/scratch2/dimpase/sage/sage/pkgs/sage-docbuild/sage_docbuild/sphinxbuild.py", line 327, in runsphinx
[sagemath_doc_html-none]     sys.stderr.raise_errors()
[sagemath_doc_html-none]   File "/home/scratch/scratch2/dimpase/sage/sage/pkgs/sage-docbuild/sage_docbuild/sphinxbuild.py", line 263, in raise_errors
[sagemath_doc_html-none]     raise OSError(self._error)
[sagemath_doc_html-none] OSError: Exception occurred:
[sagemath_doc_html-none]

@dimpase
Copy link
Member Author

dimpase commented Sep 22, 2023

Is there anything to do here, still? I'm taking the liberty to turn this to positive review.

@mkoeppe
Copy link
Member

mkoeppe commented Sep 22, 2023

It's in good enough shape for this experimental feature, so I agree with the positive review.

On gentoo-python-3.11-standard-sitepackages https://github.com/sagemath/sage/actions/runs/6264830191/job/17012722151, I see

  [sagemath_doc_html-none]     File "/sage/pkgs/sage-docbuild/sage_docbuild/builders.py", line 980, in get_new_and_updated_modules
  [sagemath_doc_html-none]       env = self.get_sphinx_environment()
  [sagemath_doc_html-none]             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  [sagemath_doc_html-none]     File "/sage/pkgs/sage-docbuild/sage_docbuild/builders.py", line 876, in get_sphinx_environment
  [sagemath_doc_html-none]       env = pickle.load(f)
  [sagemath_doc_html-none]             ^^^^^^^^^^^^^^
  [sagemath_doc_html-none]   AttributeError: Can't get attribute '_stable_repr_object' on <module 'sphinx.builders.html' from '/usr/lib/python3.11/site-packages/sphinx/builders/html/__init__.py'>
  [sagemath_doc_html-none]   
  [sagemath_doc_html-none]       Note: incremental documentation builds sometimes cause spurious

The other tested platforms look OK (despite the scary red)

@orlitzky
Copy link
Contributor

Is that spkg-configure.m4 for jupyter_packaging still being created?

@vbraun
Copy link
Member

vbraun commented Sep 22, 2023

merge conflict

vbraun pushed a commit to vbraun/sage that referenced this pull request Sep 23, 2023
    
This is needed for Sphinx 7.1+, as outlined in sagemath#36295 and discovered in
sagemath#36276

This will resolve sagemath#36295

It also fixes a bug in `region_plot`, where in some cases `plot_points`
were not passed to the backend.
    
URL: sagemath#36296
Reported by: Dima Pasechnik
Reviewer(s): Dima Pasechnik, Kwankyu Lee, Michael Orlitzky
@mkoeppe
Copy link
Member

mkoeppe commented Sep 23, 2023

Is that spkg-configure.m4 for jupyter_packaging still being created?

removed in 180e814

@github-actions
Copy link

Documentation preview for this PR (built with commit 180e814; changes) is ready! 🎉

mkoeppe added a commit to mkoeppe/sympy that referenced this pull request Sep 24, 2023
vbraun pushed a commit to vbraun/sage that referenced this pull request Sep 25, 2023
    
This is a continuation of sagemath#36256

- Fixes sagemath#36301
- Fixes https://groups.google.com/g/sage-
release/c/1wOBmhvNJqc/m/Jk14VAbjBAAJ (hence marked critical)

### 📝 Checklist

<!-- Put an `x` in all the boxes that apply. -->
<!-- If your change requires a documentation PR, please link it
appropriately -->
<!-- If you're unsure about any of these, don't hesitate to ask. We're
here to help! -->
<!-- Feel free to remove irrelevant items. -->

- [x] The title is concise, informative, and self-explanatory.
- [x] The description explains in detail what this PR is about.
- [x] I have linked a relevant issue or discussion.
- [ ] I have created tests covering the changes.
- [ ] I have updated the documentation accordingly.

### ⌛ Dependencies


- sagemath#36296: to use an up to date Sphinx
- sagemath#36267: updated ipympl, etc

Also, a Jupyter/Python issue was uncovered there, which might need work.
    
URL: sagemath#36276
Reported by: Dima Pasechnik
Reviewer(s): Matthias Köppe, Michael Orlitzky
vbraun pushed a commit to vbraun/sage that referenced this pull request Sep 27, 2023
…her improvements

    
<!-- ^^^^^
Please provide a concise, informative and self-explanatory title.
Don't put issue numbers in there, do this in the PR body below.
For example, instead of "Fixes sagemath#1234" use "Introduce new method to
calculate 1+1"
-->
<!-- Describe your changes here in detail -->

The new script `.ci/merge-fixes.sh` uses the GitHub CLI to retrieve open
PRs with the "blocker" label from sagemath/sage and tries to merge them
into the tested branch, ignoring any PRs that give merge failures.

We call this script in all CI workflows. Thus,
- hotfixes for the linter (such as sagemath#36327) or
- fixes for build problems (such as sagemath#36276)

will take effect in the CI workflows of all PRs.

This can be seen in the runs of the workflows of the present PR (section
"Merge CI fixes from sagemath/sage").

Additionally,
- we fix `debian-trixie`, which failed because a system package is no
longer available
- we fix `macos-conda-maximal` by removing some fictitious
distros/conda.txt files
- we refactor ci-macos using a new reusable workflow; example runs:
   - https://github.com/mkoeppe/sage/actions/runs/6308124398
   -
https://github.com/sagemath/cysignals/actions/runs/6316714387?pr=185
- we update the ci-macos platforms, including a configuration for the
new and problematic Xcode 15.0

<!-- Why is this change required? What problem does it solve? -->
<!-- If this PR resolves an open issue, please link to it here. For
example "Fixes sagemath#12345". -->
<!-- If your change requires a documentation PR, please link it
appropriately. -->

### 📝 Checklist

<!-- Put an `x` in all the boxes that apply. -->
<!-- If your change requires a documentation PR, please link it
appropriately -->
<!-- If you're unsure about any of these, don't hesitate to ask. We're
here to help! -->
<!-- Feel free to remove irrelevant items. -->

- [x] The title is concise, informative, and self-explanatory.
- [x] The description explains in detail what this PR is about.
- [ ] I have linked a relevant issue or discussion.
- [ ] I have created tests covering the changes.
- [ ] I have updated the documentation accordingly.

### ⌛ Dependencies

<!-- List all open PRs that this PR logically depends on
- sagemath#12345: short description why this is a dependency
- sagemath#34567: ...
-->

<!-- If you're unsure about any of these, don't hesitate to ask. We're
here to help! -->
    
URL: sagemath#36338
Reported by: Matthias Köppe
Reviewer(s): Dima Pasechnik, Jonathan Kliem, Kwankyu Lee, Matthias Köppe, Tobias Diez
@vbraun vbraun merged commit 0b5fc8e into sagemath:develop Sep 27, 2023
30 of 49 checks passed
@mkoeppe mkoeppe added this to the sage-10.2 milestone Sep 27, 2023
vbraun pushed a commit to vbraun/sage that referenced this pull request Oct 17, 2023
    
a continuation of sagemath#36276  (more system Python packages support via
`--enable-system-site-packages`)

Also, added a Cython 3.0.2 patch to fix the usage of `--enable-system-
site-packages` option. Without it, and sufficiently many Python system
packages, sagelib won't build. (Therefore, blocker label added)
    
URL: sagemath#36332
Reported by: Dima Pasechnik
Reviewer(s): Michael Orlitzky
vbraun pushed a commit to vbraun/sage that referenced this pull request Oct 19, 2023
    
a continuation of sagemath#36276  (more system Python packages support via
`--enable-system-site-packages`)

Also, added a Cython 3.0.2 patch to fix the usage of `--enable-system-
site-packages` option. Without it, and sufficiently many Python system
packages, sagelib won't build. (Therefore, blocker label added)
    
URL: sagemath#36332
Reported by: Dima Pasechnik
Reviewer(s): Michael Orlitzky
oscarbenjamin pushed a commit to oscarbenjamin/sympy that referenced this pull request Nov 3, 2023
clrpackages pushed a commit to clearlinux-pkgs/sympy that referenced this pull request Jun 6, 2024
Jerry James (1):
      Remove unused keyword arguments to ast types

Matthias Köppe (4):
      CI: Update ci-sage.yml
      CI: Update ci-sage.yml to use debian-bullseye (Python 3.9)
      CI: Update ci-sage.yml to use sagemath/sage#36276
      .github/workflows/ci-sage.yml: Back to using sagemath/sage @ develop

Oscar Benjamin (55):
      maint: bump version to 1.12.1
      Add noqa for flake8
      Add noqa
      doc: fix dangling hyphens per sphinx-lint
      Update pyversion check in optdep tests
      Fix .mailmap
      Update versions for optional dependency jobs in CI
      Update AUTHORS file
      fix(docs): fix broken sphinx references
      maint: update for new asv
      Install mpmath in optdep tests
      fix(core): various fixes for latest mpmath changes
      Don't install libclang in optdep tests
      Add more Python versions to test install
      Use 3.13-dev in release.yml
      Pin asv==0.5.1
      Temporarily run sage ci in PRs
      Only run SAGE CI on merge
      Chage version to 1.12.1a1
      Update release in CI
      maint(ci): add a numpy prerelease job to CI
      rename jobs
      Fix parsing and printing of numpy types for numpy 2.0
      Add requirements-dev.txt
      Add requiremennts-dev.txt to whitelist
      Fix version check for SciPy prereleases
      Fix scipy version check
      maint(ci): add CI tests for mpmath master branch
      fix python version
      Fix yml syntax
      maint(ci): make mpmath CI job run after main tests
      fix(core): don't use mpmath's to_pickable for pickling
      Fix GA workflow
      Configure graphviz
      release: push to PyPI on tag push
      Fix actions versions
      Add job to push to PYPI on tag push
      Fix duplicate jo ame
      Add comment
      Restrict workflow to sympy-1.12.* tags
      release: Automate GitHub release and docs update
      Fix workflow
      Bump version in release.py
      release: use GH_TOKENN for GitHub release
      release: use ssh key instead of token
      maint(deps): require mpmath >= 1.1.0, < 1.4.0
      maint(release): bump version to 1.12.1
      remove unneeded f-string
      release: reenable pusing to PyPI
      release: don't make a draft github release
      release: bump version to 1.12.1
      release: Use ${{}} in release.yml
      Use github.env...
      maint: update .mailmap and AUTHORS
      maint: update actions versions in CI

Pablo Galindo Salgado (1):
      Fix factorial parsing for Python 3.12

Raj Sapale (2):
      Changed the signature methods for _array_ methods in Sympy to :
      author: add Raj Sapale to .mailmap

S.Y. Lee (6):
      Fix deprecation issues with python 3.12 ast lib
      Replace ast.Num
      Fix additional deprecation warnings
      Use pypy 2.7
      Fix deprecated bitwise not on boolean and fix pytest return warning
      Remove semantically wrong test

Sam Lubelsky (6):
      Added test cases for the new __array__ methods(Copy=true/false/None)
      fixed trailing whitespace
      actually fixed trailing whitespace
      now imports skip properly.
      added numpy version checks to determine if copy=False should be tested
      fixed trailing whitespace
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

build/pkgs/toml doesn't have install info - remove?
5 participants