gh-89727: Fix pathlib.Path.walk RecursionError on deep trees #100282

zmievsa · 2022-12-15T22:46:49Z

Use a stack to implement pathlib.Path.walk iteratively instead of recursively to avoid hitting recursion limits on deeply nested trees.

Issue: Change shutil.rmtree and os.walk to support very deep hierarchies #89727

netlify · 2022-12-15T22:46:53Z

✅ Deploy Preview for python-cpython-preview canceled.

Name	Link
🔨 Latest commit	`2bfc47d`
🔍 Latest deploy log	https://app.netlify.com/sites/python-cpython-preview/deploys/639c50140be628000884140d

zmievsa · 2022-12-15T22:53:08Z

TODO:

Add a news blip
Add a test that checks for deeply nested directories
Figure out a better name for is_result

Update:
All done!

…on-depth

…f github.com:ovsyanka83/cpython into pythongh-89727/fix-pathlib.Path.walk-recursion-depth

brettcannon · 2022-12-16T19:23:49Z

/cc @barneygale

Lib/pathlib.py

Lib/test/test_pathlib.py

…on-depth

carljm

LGTM. I assume because of the initial underscore we don't have to worry about people who may have subclassed Path and overridden the _walk method (now to no effect.)

zmievsa · 2022-12-20T22:47:56Z

@carljm remember that Path.walk has only been created in 3.12 which means that 99% of libraries/projects do not and cannot depend on it yet.

zmievsa · 2022-12-24T10:38:09Z

@barneygale any estimates of when you will have time to take a look at it? No pressure or rush though.

barneygale · 2022-12-24T15:10:14Z

I'll look at this within the next few days!

Lib/pathlib.py

Co-authored-by: Barney Gale <barney.gale@gmail.com>

barneygale

Looks good to me!

zmievsa · 2023-01-11T21:57:42Z

@AlexWaygood what would be the next steps for this PR? I am ready to continue improving it

…on-depth

AlexWaygood · 2023-01-11T21:59:58Z

Sorry @Ovsyanka83, I'll try to take a proper look soon!

Lib/pathlib.py

Lib/test/test_pathlib.py

Co-authored-by: Brett Cannon <brett@python.org>

barneygale · 2023-02-01T15:37:37Z

@AlexWaygood gentle nudge to review this when you have the time! Thanks :)

CuriousLearner

LGTM.

Thanks for your contribution.

barneygale · 2023-02-28T16:33:45Z

@AlexWaygood https://youtu.be/CdqoNKCCt7A

Lib/pathlib.py

zmievsa · 2023-03-22T18:13:46Z

@barneygale thanks!

Wow. Have you received merge rights? Congrats! Did they make you a core developer or is that a special rule just for you and pathlib?

AlexWaygood · 2023-03-22T18:31:57Z

We made him a core dev! https://discuss.python.org/t/vote-to-promote-barney-gale/24801

barneygale · 2023-03-22T20:04:51Z

@Ovsyanka83 thanks for your patience and for working on this, it's a neat patch :). I'm hoping to build an iterative version of glob() atop!

…ythonGH-100282) Use a stack to implement `pathlib.Path.walk()` iteratively instead of recursively to avoid hitting recursion limits on deeply nested trees. Co-authored-by: Barney Gale <barney.gale@gmail.com> Co-authored-by: Brett Cannon <brett@python.org>

Use `Path.walk()` to implement the recursive wildcard `**`. This method uses an iterative (rather than recursive) walk - see pythonGH-100282.

Use `Path.walk()` to implement the recursive wildcard `**`. This method uses an iterative (rather than recursive) walk - see GH-100282.

Replace recursion with iteration

037a717

bedevere-bot mentioned this pull request Dec 15, 2022

Change shutil.rmtree and os.walk to support very deep hierarchies #89727

Open

bedevere-bot added the awaiting review label Dec 15, 2022

Unify walk and _walk methods

f8968b6

zmievsa and others added 5 commits December 16, 2022 03:10

Remove empty typehint

ecabf8c

Fix all tests

6ec3c3e

Add test for recursion limit

fb5a4c7

📜🤖 Added by blurb_it.

55beb6e

Merge branch 'main' into pythongh-89727/fix-pathlib.Path.walk-recursi…

7f36140

…on-depth

zmievsa marked this pull request as ready for review December 16, 2022 10:32

zmievsa requested a review from brettcannon as a code owner December 16, 2022 10:32

zmievsa added 2 commits December 16, 2022 15:01

Fix recursion limit issue in different envs

508bd4d

Merge branch 'pythongh-89727/fix-pathlib.Path.walk-recursion-depth' o…

2bfc47d

…f github.com:ovsyanka83/cpython into pythongh-89727/fix-pathlib.Path.walk-recursion-depth

pitrou requested a review from serhiy-storchaka December 16, 2022 14:27

brettcannon added the topic-pathlib label Dec 16, 2022

Make everything consistent with os.walk

20476ff

kalvdans reviewed Dec 20, 2022

View reviewed changes

Lib/pathlib.py Outdated Show resolved Hide resolved

Remove unused import

4f2de62

carljm reviewed Dec 20, 2022

View reviewed changes

Lib/test/test_pathlib.py Outdated Show resolved Hide resolved

zmievsa added 2 commits December 21, 2022 01:47

Merge branch 'main' into pythongh-89727/fix-pathlib.Path.walk-recursi…

6313b6a

…on-depth

Replace the usage of infinite_recursion with set_recursion_limit

17497f0

carljm approved these changes Dec 20, 2022

View reviewed changes

bedevere-bot added awaiting core review and removed awaiting review labels Dec 20, 2022

barneygale reviewed Jan 2, 2023

View reviewed changes

Lib/pathlib.py Outdated Show resolved Hide resolved

Update Lib/pathlib.py

b7d36de

Co-authored-by: Barney Gale <barney.gale@gmail.com>

barneygale approved these changes Jan 2, 2023

View reviewed changes

Code review fixes

d428a89

Merge branch 'main' into pythongh-89727/fix-pathlib.Path.walk-recursi…

4ce5291

…on-depth

brettcannon approved these changes Jan 21, 2023

View reviewed changes

Lib/pathlib.py Outdated Show resolved Hide resolved

Lib/test/test_pathlib.py Outdated Show resolved Hide resolved

bedevere-bot added awaiting merge and removed awaiting core review labels Jan 21, 2023

Apply suggestions from code review

49d4237

Co-authored-by: Brett Cannon <brett@python.org>

CuriousLearner approved these changes Feb 18, 2023

View reviewed changes

barneygale reviewed Mar 11, 2023

View reviewed changes

Lib/pathlib.py Outdated Show resolved Hide resolved

barneygale mentioned this pull request Mar 12, 2023

Path.rglob performance issues in deeply nested directories compared to glob.glob(recursive=True) #102613

Closed

ionite34 mentioned this pull request Mar 12, 2023

BOT-3EN: snekbox hits a recusion exceded exception in pathlib when parsing lots of nested directories and 500s python-discord/snekbox#172

Closed

Rename stack to paths

9bc2500

barneygale approved these changes Mar 19, 2023

View reviewed changes

barneygale merged commit 713df2c into python:main Mar 22, 2023

bedevere-bot removed the awaiting merge label Mar 22, 2023

zmievsa deleted the gh-89727/fix-pathlib.Path.walk-recursion-depth branch March 22, 2023 18:20

barneygale mentioned this pull request May 11, 2023

GH-102613: Fix recursion error from pathlib.Path.glob() #104373

Merged

barneygale added a commit that referenced this pull request May 15, 2023

GH-102613: Fix recursion error from pathlib.Path.glob() (GH-104373)

cb88ae6

Use `Path.walk()` to implement the recursive wildcard `**`. This method uses an iterative (rather than recursive) walk - see GH-100282.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

gh-89727: Fix pathlib.Path.walk RecursionError on deep trees #100282

gh-89727: Fix pathlib.Path.walk RecursionError on deep trees #100282

zmievsa commented Dec 15, 2022 •

edited

Loading

netlify bot commented Dec 15, 2022 •

edited

Loading

zmievsa commented Dec 15, 2022 •

edited

Loading

brettcannon commented Dec 16, 2022

carljm left a comment

zmievsa commented Dec 20, 2022

zmievsa commented Dec 24, 2022

barneygale commented Dec 24, 2022

barneygale left a comment

zmievsa commented Jan 11, 2023

AlexWaygood commented Jan 11, 2023

barneygale commented Feb 1, 2023 •

edited

Loading

CuriousLearner left a comment

barneygale commented Feb 28, 2023

zmievsa commented Mar 22, 2023

AlexWaygood commented Mar 22, 2023

barneygale commented Mar 22, 2023

gh-89727: Fix pathlib.Path.walk RecursionError on deep trees #100282

gh-89727: Fix pathlib.Path.walk RecursionError on deep trees #100282

Conversation

zmievsa commented Dec 15, 2022 • edited Loading

netlify bot commented Dec 15, 2022 • edited Loading

✅ Deploy Preview for python-cpython-preview canceled.

zmievsa commented Dec 15, 2022 • edited Loading

brettcannon commented Dec 16, 2022

carljm left a comment

Choose a reason for hiding this comment

zmievsa commented Dec 20, 2022

zmievsa commented Dec 24, 2022

barneygale commented Dec 24, 2022

barneygale left a comment

Choose a reason for hiding this comment

zmievsa commented Jan 11, 2023

AlexWaygood commented Jan 11, 2023

barneygale commented Feb 1, 2023 • edited Loading

CuriousLearner left a comment

Choose a reason for hiding this comment

barneygale commented Feb 28, 2023

zmievsa commented Mar 22, 2023

AlexWaygood commented Mar 22, 2023

barneygale commented Mar 22, 2023

zmievsa commented Dec 15, 2022 •

edited

Loading

netlify bot commented Dec 15, 2022 •

edited

Loading

zmievsa commented Dec 15, 2022 •

edited

Loading

barneygale commented Feb 1, 2023 •

edited

Loading